Freebee Ads docs

Engineering docs

Testing — local and CI

Markdown

Last updated Oct 20, 2018 · 4 sections

Three layers, all green before anything merges to main (.github/workflows/ci.yml runs them on every push and pull request):

LayerWhereWhat it proves
Backend unit + APIbackend/tests (pytest, 358 tests)contract shapes, ledger rules, money path (test_e2e_money_path.py), Sentry scrubber
Web typecheck + buildweb (tsc --noEmit, next build)the app compiles and every route renders at build time
Playwright journeysweb/e2e (@playwright/test, 19 specs / 84 tests, about 2 minutes locally)the product works end-to-end in a browser against a real backend

Backend

cd backend
python -m venv .venv && .venv/bin/pip install -r requirements.txt -r requirements-dev.txt
.venv/bin/python -m pytest -q -p no:warnings

Tests pin DATABASE_URL / MEDIA_STORE_DIR to a temp dir in tests/conftest.py; nothing touches var/dev.db.

Web

cd web
npm ci
npx tsc --noEmit -p .
npm run build

Playwright journeys

The suite lives in web/e2e and codes to the seeded dev data (backend/scripts/seed.py: four screens at Inorbit, demo@dooh.local / ops@dooh.local). Every journey creates its own advertiser, creative and booking through the API first, so tests are independent and safe to retry; only the seeded screens and the ops account are shared.

SpecJourney
01-signupfresh sign-up → onboarding → Home shows the four metric bands under "Explore performance breakdowns"
02-creative-uploadupload a generated PNG → appears in the grid, "Waiting for review"
03-bookingbook the first seeded screen for tomorrow → "Booked" card
04-approvalops sees the Approvals badge → Review opens the slide-over → tick the three checks (Approve stays off until then) → Approve → advertiser sees Approved
05-wall-pairing/player shows a code → /ops/pair claims it → player loops → /v1/metrics/live lists the screen
06-walletshopper requests a dev-mode OTP, verifies with consent → My coupons (masked number)
07-smoke (@smoke)every page for public (incl. legal, forgot/reset, docs) / advertiser (incl. billing, messages, team) / ops (incl. devices, API keys) answers 200 with no console errors; /healthz and /api/health answer
08-billingops Mark paid on Bookings → advertiser Billing shows the invoice Paid → the printable invoice renders
09-messagesapproval lands two messages → the bell counts them → Mark all read clears it without a reload
10-teamowner invites a viewer (dev echo link) → sign-up with the invite joins the team → the viewer's booking is refused with the view-only message
11-devicesa paired player's health shows on Devices → Reload → the player acks at its next check-in → the card shows the result
12-proofDownload proof of play → /v1/proof/verify accepts the file and rejects an edited copy
13-password-resetforgot → dev echo reset link → new password works, old one does not, the link is single use
14-jobs-hardeningcontent security policy (@smoke: nonce policy on pages, engine pages load with no violation, game editor / kiosk / player too) → ops Today "Background jobs" Run now → owner changes a teammate's role and removes them after the confirm line
14-languagesthe phone page switches to Telugu and Hindi (html lang, ?lang=, remembered across a reload); /w?lang=hi opens in Hindi and English is one tap away
14-mallops invite a mall contact → the contact signs in and lands on /mall → Overview bands, Screens, Statement → creates a partner API key that reads its screens; an advertiser opening /mall is sent to /dashboard and /v1/mall/* answers 403

Run locally

Start a backend on a throw-away DB with dev OTP on, seed it, start the web against it, run:

# terminal 1 — backend
cd backend
DATABASE_URL=sqlite:////tmp/e2e/dev.db MEDIA_STORE_DIR=/tmp/e2e/media \
  .venv/bin/python scripts/seed.py && \
DATABASE_URL=sqlite:////tmp/e2e/dev.db MEDIA_STORE_DIR=/tmp/e2e/media WALLET_OTP_DEV=1 \
  EMAIL_DEV_ECHO=1 WEB_URL=http://localhost:3000 .venv/bin/python -m uvicorn app.main:app --port 8001

# terminal 2 — web
cd web && NEXT_PUBLIC_API_URL=http://localhost:8001 npx next dev -p 3000

# terminal 3 — the suite (first time: npx playwright install chromium)
cd web
E2E_BASE_URL=http://localhost:3000 E2E_API_URL=http://localhost:8001 npx playwright test
npx playwright show-report          # HTML report; traces are kept for failures

Any ports work — the suite only reads E2E_BASE_URL (default http://localhost:3000) and E2E_API_URL (default http://localhost:8001). 06-wallet needs WALLET_OTP_DEV=1 on the backend; 10-team and 13-password-reset need EMAIL_DEV_ECHO=1 (with the default EMAIL_PROVIDER=log). Bookings land on a random future day, so the suite can run again and again on the same database without selling a seeded screen out.

Against production — read-only smoke only

cd web
E2E_BASE_URL=https://freebeeads.com \
E2E_API_URL=https://api.freebeeads.com \
npx playwright test --grep @smoke

The @smoke subset never writes. The role pages need credentials for that environment (E2E_ADVERTISER_EMAIL / E2E_ADVERTISER_PASSWORD, E2E_OPS_EMAIL / E2E_OPS_PASSWORD); when a login fails the role block is skipped, never failed. Do not run the @journey specs against production: they create real advertisers, bookings, devices and wallet users.

What CI does (.github/workflows/ci.yml)

Three jobs in parallel, pip and npm caches on:

  1. backend — Python 3.12, pip install -r requirements.txt -r requirements-dev.txt, pytest.
  2. web — Node 20, npm ci, tsc --noEmit, next build.
  3. e2e — starts the backend on :8001 (temp SQLite under /tmp/e2e, WALLET_OTP_DEV=1, EMAIL_DEV_ECHO=1, scripts/seed.py), builds and starts the web on :3000 with NEXT_PUBLIC_API_URL, installs Chromium (cached), runs the suite with one retry, and on failure prints the server logs and uploads playwright-report/ + test-results/ (traces) as the playwright-report artifact.

No GitHub secrets are needed for CI. Typical wall time: backend ~3 min, web ~3 min, e2e ~6–8 min.

Product screenshots (web/public/shots)

npm run shots (in web/, with the same two dev servers up and scripts/seed_templates.py run so the game ads exist) regenerates the landing-page screenshots at 1440×900 with Playwright and converts them to WebP with cwebp when it is installed. See web/scripts/shots.mjs.

Testing and CI · Docs