Engineering docs

Secrets rotation (Railway)

Last updated Oct 24, 2026 · 5 sections

Written 2026-10-24. The three production secrets on Railway (SECRET_KEY, OPS_PASSWORD, the Postgres password inside DATABASE_URL) were shared in chat during the build and must be rotated before launch and after any testing round where they were shared. This is the order that does not break anything. Do it in a quiet hour: every advertiser, ops and wallet session signs out when SECRET_KEY changes.

What each secret protects

Secret	Protects	If you just change it
`SECRET_KEY`	Fernet encryption and HMAC lookup of wallet phone numbers; every JWT (advertiser, ops, wallet); OTP code hashes	Every wallet phone becomes unreadable and no wallet can log in. Run the rotation script first.
`OPS_PASSWORD`	The ops@dooh.local login	The backend updates the ops password on the next boot (since 2026-10-24). Safe to change any time.
Postgres password	The database	Railway rewrites `DATABASE_URL` when you regenerate it on the Postgres service, if the backend reads it as a reference variable. Check before you do it.

1. Rotate SECRET_KEY (needs the script)

Generate a new key on your laptop and keep it in your password manager:
```
python3 -c "import secrets; print(secrets.token_urlsafe(48))"
```
Open a shell inside the backend container (the database is reachable only from there). The Railway CLI is linked at backend/:
```
cd backend && railway ssh
```
Inside the container, dry-run the rotation. Nothing is written yet. It prints counts only, never a phone number:
```
cd /app && OLD_SECRET_KEY="<current SECRET_KEY>" NEW_SECRET_KEY="<new key>" /app/.venv/bin/python scripts/rotate_secret_key.py
```
Expect unreadable (skipped): 0. If it is not 0, the old key is wrong; stop.

Apply it:

cd /app && OLD_SECRET_KEY="<current SECRET_KEY>" NEW_SECRET_KEY="<new key>" /app/.venv/bin/python scripts/rotate_secret_key.py --apply

Leave the container. In Railway → dooh-platform → Variables set SECRET_KEY=<new key>. Railway redeploys. Everyone signs in again; wallets ask for a new OTP.
If a wallet row was skipped as unreadable, that shopper registers again with a fresh OTP; their coupons stay attached to the wallet id.

The script is safe to re-run: rows already on the new key are counted as "already on new key" and left alone.

2. Rotate OPS_PASSWORD

Generate a password (16+ characters) and store it in the password manager.
Railway → Variables → OPS_PASSWORD=<new>. Railway redeploys; on boot the backend sees the stored hash no longer matches and updates it.
Sign in at /login as ops@dooh.local with the new password. Tell the ops team one to one; never in a group chat.

3. Rotate the Postgres password

In Railway, open the Postgres service → Variables. Check how DATABASE_URL on the backend is defined: if it shows ${{Postgres.DATABASE_URL}} it is a reference and updates itself; if it is a pasted string you must paste the new URL after step 2.
Regenerate the password on the Postgres service (Railway: Variables → POSTGRES_PASSWORD → regenerate, or recreate the credentials per Railway's current UI).
If the backend URL was pasted, update DATABASE_URL on dooh-platform. Redeploy and check https://api.freebeeads.com/healthz answers 200 and /login works.

4. After rotating

Remove WALLET_OTP_DEV once WhatsApp delivery is on (see docs/DEPLOY.md).
Delete any chat messages that contained the old values.
Record the date here:

Date	What	By