Skip to content

Add script for calculating DB memory settings #3541

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
denniskuijs wants to merge 11 commits into
base: main
Choose a base branch
from
+38 −1
Draft

Add script for calculating DB memory settings #3541

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
e5e80f2
Add script for calculating DB memory settings
denniskuijs Nov 3, 2026
6990605
wip
denniskuijs Nov 4, 2026
3547a22
wip
denniskuijs Nov 4, 2026
675902e
Merge branch 'main' into update/deploy.sh
denniskuijs Nov 4, 2026
f4c77ed
wip
denniskuijs Nov 4, 2026
992aa94
wip
denniskuijs Nov 4, 2026
eb3f359
wip
denniskuijs Nov 4, 2026
b04ba10
wip
denniskuijs Nov 4, 2026
2992485
wip
denniskuijs Nov 4, 2026
2362ba5
wip
denniskuijs Nov 4, 2026
6826cbb
wip
denniskuijs Nov 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .ci_cd/deploy.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,14 @@ if [ -f "temp/deployment.tar.gz" ]; then
docker load < temp/deployment.tar.gz
fi

# Calculate DB memory settings
. ./temp/host_init/calculate_db_memory.sh

@Ekhorn Ekhorn Nov 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

temp/env is loaded just above, so this script and the OR_DB_RAM export below overwrite any value a project set in .ci_cd/env or a secret, which .ci_cd/README.md describes as the way to configure compose variables. A host tuned by hand would change on its next deployment. Calculating only the values that are unset would keep those settings, and the README's variables section should mention the new behavior.


# Calculate host RAM
export OR_RAM="$(( $(free -m | awk '/^Mem:/{print $2}') * 40 / 100 ))"
export OR_KC_RAM="$(( $(free -m | awk '/^Mem:/{print $2}') * 20 / 100 ))"
export OR_DB_RAM="$(( $(free -m | awk '/^Mem:/{print $2}') * 25 / 100 ))"
Comment on lines +226 to +228

@Ekhorn Ekhorn Nov 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The heredoc at line 191 is unquoted, so these $(free ...) and $(( )) expressions run on the CI runner before the script reaches the host. Elsewhere in the heredoc, host-side expansions are escaped as \$. With a stubbed 16 GB runner and 8 GB host, OR_DB_RAM came out as 3997 instead of 1975.

No compose file in this repo reads OR_RAM or OR_KC_RAM. Is a compose change still coming for those, or can they be removed?

Suggested change
export OR_RAM="$(( $(free -m | awk '/^Mem:/{print $2}') * 40 / 100 ))"
export OR_KC_RAM="$(( $(free -m | awk '/^Mem:/{print $2}') * 20 / 100 ))"
export OR_DB_RAM="$(( $(free -m | awk '/^Mem:/{print $2}') * 25 / 100 ))"
HOST_RAM="\$(free -m | awk '/^Mem:/{print \$2}')"
export OR_RAM="\$(( HOST_RAM * 40 / 100 ))"
export OR_KC_RAM="\$(( HOST_RAM * 20 / 100 ))"
export OR_DB_RAM="\$(( HOST_RAM * 25 / 100 ))"

@Ekhorn Ekhorn Nov 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OR_DB_RAM is 25% of host RAM regardless of the chunk-based sizing, while the system administration guide treats P = 4 x N as the RAM the container needs: the largest chunk should never exceed 25% of the container limit. On an 8 GB host the limit is about 2 GB, so chunks above about 512 MB break that rule. With a 3 GB chunk, shared_buffers alone exceeds the limit, which also breaks the guide's golden rule and leads to the OOM kill it warns about. The fallback branch also says Compose defaults are used, but OR_DB_RAM is still replaced there.

Should OR_DB_RAM be at least MINIMUM_RAM_SIZE? When that does not fit on the host, the guide's remedy is a smaller chunk_time_interval, so a warning in the deployment log may be more useful than starting with settings the container cannot honor.


# Login to Private ECR Registry to download restricted images
if [ -n "$OR_AWS_ECR_ACCOUNT_ID" ] && [ -n "$OR_AWS_ECR_REGION" ]; then
aws ecr get-login-password --region $OR_AWS_ECR_REGION | docker login --username AWS --password-stdin $OR_AWS_ECR_ACCOUNT_ID.dkr.ecr.$OR_AWS_ECR_REGION.amazonaws.com &>/dev/null
Expand Down
30 changes: 30 additions & 0 deletions .ci_cd/host_init/calculate_db_memory.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#!/bin/bash
set -eo pipefail

@Ekhorn Ekhorn Nov 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

deploy.sh sources this file, so set -eo pipefail stays on for the rest of the deployment, whichever branch below runs. Commands that may fail today now abort it:

  • docker volume rm or_deployment-data fails when the volume does not exist. A new host never gets past this, because the stack that would create the volume never starts. The root docker-compose.yml has no such volume, so deployments using it would fail every time, after the existing stack has been stopped.
  • TOKEN=$(curl ...) fails on hosts without an EC2 metadata service, also with the stack stopped.
  • aws s3 cp fails on hosts without that bucket, which reports a failed deployment after a successful start.

The docker volume rm case was reproduced with this script and a stubbed curl in a shell fed from a heredoc. The curl pipeline already has || true, so nothing here needs these options:

Suggested change
set -eo pipefail
# Sourced by deploy.sh, so this must not change shell options


# Retrieve largest uncompressed chunk size from query-exporter service
CHUNK_SIZE="$(curl -s http://localhost:8406/metrics | grep '^pg_datapoint_largest_uncompressed_chunk' | awk '{printf "%.0f\n", $NF}' || true)"
if [ -z "$CHUNK_SIZE" ]; then
echo 'Could not retrieve chunk size from query-exporter service, using defaults from Docker Compose'
else
# Calculate minimum RAM size and round up to the next whole GB (1.2 GB becomes 2 GB)
MINIMUM_RAM_SIZE="$((((4 * CHUNK_SIZE + 1024 - 1) / 1024) * 1024))"

@Ekhorn Ekhorn Nov 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A metric value of 0 passes the -z check above and turns every setting into 0. The query rounds to whole MB, so a largest chunk under 0.5 MB reports 0, as can happen on a new or lightly used install. PostgreSQL then exits with FATAL: 0 8kB is outside the valid range for parameter "shared_buffers". Treating 0 like a missing metric avoids that:

if [ "${CHUNK_SIZE:-0}" -eq 0 ]; then


# Set shared buffers to 25% of minimum RAM size.
export OR_DB_SHARED_BUFFERS="$((MINIMUM_RAM_SIZE / 4))"

@Ekhorn Ekhorn Nov 4, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exporting bare MB values works when compose adds the unit, as in shm_size: ${OR_DB_SHM:-156}MB. profile/deploy.yml and the compose example in the system administration guide still have the unit inside the default (${OR_DB_SHM:-156mb}), and this PR does not change them. In that form, a 1 GB chunk on an 8 GB host is read as:

Variable Value Read as
OR_DB_RAM 1975 1975 bytes, which Docker rejects with Minimum memory limit allowed is 6MB
OR_DB_SHM 2048 2048 bytes
OR_DB_SHARED_BUFFERS 1024 8 MB (8 kB pages)
OR_DB_CACHE_SIZE 3072 24 MB (8 kB pages)
OR_DB_MAINTENANCE_WORK_MEM 512 512 kB
OR_DB_WORK_MEM 48 48 kB, below the 64 kB minimum, so PostgreSQL 17.9 exits with FATAL

profile/deploy.yml and the guide need to change together with this script. With the values above, this form was checked against Compose and PostgreSQL:

    shm_size: ${OR_DB_SHM:-156}MB
    deploy:
      resources:
        limits:
          memory: ${OR_DB_RAM:-512}MB
    command:
      - postgres
      - -c
      - shared_buffers=${OR_DB_SHARED_BUFFERS:-128}MB
      - -c
      - effective_cache_size=${OR_DB_CACHE_SIZE:-384}MB
      - -c
      - maintenance_work_mem=${OR_DB_MAINTENANCE_WORK_MEM:-56}MB
      - -c
      - work_mem=${OR_DB_WORK_MEM:-4}MB

This also affects custom projects. A compose file that follows the guide's current example gets the misread values above, and in the new form a value such as OR_DB_RAM=2G becomes 2GMB, which Compose rejects with invalid suffix. Both are worth a release note.

# Calculate SHM size by adding 20% to the shared buffers and round up to the next whole GB (1.2 GB becomes 2 GB)
export OR_DB_SHM="$((((OR_DB_SHARED_BUFFERS * 120 / 100 + 1024 - 1) / 1024) * 1024))"
# Set maintenance work mem to 12.5% of minimum RAM size and round up to the next multiple of 128 MB.
export OR_DB_MAINTENANCE_WORK_MEM="$((((MINIMUM_RAM_SIZE / 8 + 128 - 1) / 128) * 128))"
# Set worker mem to 1% of minimum RAM size and round up to the next multiple of 16 MB.
export OR_DB_WORK_MEM="$((((MINIMUM_RAM_SIZE / 100 + 16 - 1) / 16) * 16))"
# Calculate effective cache size
export OR_DB_CACHE_SIZE="$((MINIMUM_RAM_SIZE - OR_DB_SHARED_BUFFERS))"

echo 'Chunk size:' "$CHUNK_SIZE MB"
echo 'Minimum RAM size:' "$MINIMUM_RAM_SIZE MB"
echo 'Shared buffers:' "$OR_DB_SHARED_BUFFERS MB"
echo 'SHM Size:' "$OR_DB_SHM MB"
echo 'Maintenance worker mem:' "$OR_DB_MAINTENANCE_WORK_MEM MB"
echo 'Worker mem:' "$OR_DB_WORK_MEM MB"
echo 'Effective cache size:' "$OR_DB_CACHE_SIZE MB"
fi
2 changes: 1 addition & 1 deletion .github/workflows/deployment.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,7 @@ jobs:
with:
repository: ${{ steps.is_core_repo.outputs.repository }}
path: core
ref: ${{ steps.is_core_repo.outputs.version }}
ref: refs/heads/update/deploy.sh
sparse-checkout: .ci_cd
sparse-checkout-cone-mode: false

Expand Down
Loading