# meowtrace — Project Status & Session Handoff (STATUS.md)

> ↩ **Back to canonical entry:** [SPEC.md](./SPEC.md)
> Living status tracker + handoff for the next session. Read this first on wake.

**Last updated:** 2026-07-27
**Overall:** Phases 0–6 shipped. **Live in PRODUCTION** on meowtrace.com (www/app/admin →
`api.meowtrace.com` → ALB → VM → RDS), unified on the prod API. Live-mode Stripe. Plus a
Phase-5c portal feature batch (password recovery, org-named signup, dashboard analytics,
request history).

---

## 1. Where things stand

| Phase | Status |
|---|---|
| 0 Planning | ✅ |
| 1 Data & lookup core | ✅ Postgres 18; **full 10.1M-row `geo_ranges` imported to prod RDS** |
| 2 Lookup API | ✅ Fastify: single + bulk (207 partial), error envelope, in-app lookup cache |
| 3 Accounts & credit ledger | ✅ auth, API keys, credits, 429 hard-cap, usage logging |
| 4 Billing & Stripe | ✅ **LIVE-mode** products/prices/webhook; Checkout (+ **promotion codes**), Portal, provisioning |
| 5 Frontend | ✅ www (SEO static), docs, portal SPA (user + admin) |
| 5b Organizations | ✅ orgs own credits/keys/billing; invites; org switching |
| 5c Portal feature batch | ✅ password recovery, org-named signup, dashboard chart (recharts), request history |
| 6 Deploy & harden (prod) | ✅ **prod VM + RDS + ALB + S3/CloudFront live**; still TODO: Redis, read-replicas, X-Forwarded-For IP |

## 2. Live surfaces (all HTTPS)

**PRODUCTION** (see [DEVOPS.md](./DEVOPS.md) § Production for full topology):
| URL | What | Backing |
|---|---|---|
| https://api.meowtrace.com | Lookup + account/billing API | ACM TLS → **ALB** → prod VM `13.228.3.109` nginx :80 (default_server) → Fastify `:8090` → **RDS** |
| https://app.meowtrace.com | User Portal | S3 `s3.meowtrace.com/app/prod` → CloudFront `E1Z2MH4SYRCMJH` |
| https://admin.meowtrace.com | Admin Portal (same build, host-gated) | S3 `.../admin/prod` → CloudFront `ETB82PO7ZG7BG` |
| https://www.meowtrace.com | SEO marketing site + docs.html | S3 `.../www/prod` → CloudFront `E9OXGMG86LRC0` |

**STAGING** (dev box, Let's Encrypt; local Postgres):
| URL | Backing |
|---|---|
| https://api.staging.meowtrace.com | Fastify `127.0.0.1:8080` (systemd `meowtrace-api` on dev box) |
| https://app / admin / www.staging.meowtrace.com | `/var/www/<host>/` static |
| https://docs.staging.meowtrace.com | raw markdown (autoindex) · https://viewer.staging… MD viewer |

> **ALB health check:** target group probes **`/status`** (DB-verified readiness, 200/503),
> not `/health`. nginx on the prod VM is the **default_server** on :80 so host-less ALB
> checks reach the backend. Prod DNS: `api.meowtrace.com` → ALB (Alias); resolver caches
> may briefly show the old staging A record.

## 3. Source layout (`projects/meowtrace/`)

- `src/backend/` — Node+TS+Fastify API. Key modules: `config.ts` (TOML loader),
  `db.ts`, `routes.ts` (lookup), `accounts_routes.ts` (auth/keys/me/org-switch),
  `org_routes.ts` (members/invites), `billing.ts`+`billing_routes.ts` (Stripe),
  `admin_routes.ts`, `credits.ts`, `orgs.ts`, `auth.ts`, `mail.ts` (SendGrid),
  `tiers.ts`. Run scripts: `import`, `lookup`, `benchmark`, `stripe:setup`, `serve`.
- `src/portal/` — React+Vite+MUI SPA (user + admin, hostname-selected). Shared
  `components/TigerMark.tsx` = traced chibi tiger logo.
- `src/www/` — SEO static site (`index.html`, `docs.html`, `robots.txt`,
  `sitemap.xml`, `assets/logo-mark.svg`).
- `src/frontend/` — the Minimal `vite-ts` kit (reference base; portal was built lean instead).
- `reference/` — Minimal v5.4.0 kit (read-only) + `data-sample/`.
- `scripts/` — `schema.sql`, `schema_accounts.sql`, `schema_orgs.sql`, `retired/`.

## 4. Config & secrets (IMPORTANT)

- **Loader:** `config.ts` resolves `/etc/meowtrace.<env>.toml` from `MEOWTRACE_ENV`;
  `MEOWTRACE_CONFIG` overrides path; env vars are fallback. Sections: `[server]`
  (port/host/app_base_url/session_secret), `[database]` (url), `[stripe]`
  (secret/public/webhook/price_hobby/price_enterprise), `[sendgrid]` (api_key/mail_from).
- **STAGING config: `/etc/meowtrace.staging.toml`** (dev box, owner `ubuntu`, 600) — test-mode Stripe, local Postgres.
- **PROD config: `/etc/meowtrace.production.toml`** (prod VM, owner `ubuntu`, 600, `MEOWTRACE_ENV=production`)
  — **LIVE-mode** Stripe keys, RDS DSN, prod webhook secret. **DSN gotcha:** the password
  must be **percent-encoded** in the URL (see [DEVOPS.md](./DEVOPS.md) § Prod config).
- `/etc/meowtrace-api.env` (mode 600) still exists as fallback + sets `MEOWTRACE_ENV=staging`.
  *(Open question: slim it to just MEOWTRACE_ENV, or keep belt-and-suspenders.)*
- `mail_from` = **no-reply@meowtrace.com**. SendGrid domain auth for meowtrace.com is **valid** (deliverable).
- **⚠️ The `***` masking-artifact trap:** never write a secret via a value that could be
  masked to `***`; source real secrets straight from the config/DSN, and guard
  destructive steps with a "real value confirmed" check (bit us 3× this session).
- **Stripe checkout** sets `allow_promotion_codes: true` — the hosted page shows an "Add
  promotion code" field; coupons/promo codes are managed in the Stripe dashboard.

## 5. Admin access

- **Admin portals:** https://admin.meowtrace.com (**prod**) + https://admin.staging.meowtrace.com
  — login-only (admins are provisioned, never self-register).
- **Prod admin exists:** `ben@meowtrace.com` (role `admin`) was copied staging→prod (same
  `password_hash`), so the same credentials work on both. `testadmin@` was **not** copied.
- **Credentials are NOT stored in docs** (this file is served publicly at
  `docs.staging.meowtrace.com`). Admin login is held out-of-band by the owner. To grant
  admin on a fresh account: `UPDATE accounts SET role='admin' WHERE email='...'`
  (see [DEVOPS.md](./DEVOPS.md)).

## 6. Data model notes (orgs refactor, 5b)

- **Orgs are the tenant.** Credits, API keys, usage, subscriptions all scope to `org_id`.
- Each account gets a **personal org** on signup (owner, optional custom `org_name`). Orgs
  invite members by email; invitee auto-joins on next signup/login. Session carries current
  org (default = last login); `/v1/orgs/switch` re-tokens.
- **`account_id` on org-owned tables (`credit_ledger`/`api_keys`/`usage_records`/`subscriptions`)
  is nullable/legacy** — the app writes `org_id` only. `schema_orgs.sql` `DROP NOT NULL`s all
  four (drift D7 fix + **D10** which added `subscriptions`; a fresh DB with `NOT NULL` there
  rejects signups / crashes the Stripe checkout webhook). Org is the tenant everywhere.
- Signup is **always Free** (no tier param); tier upgrades via Stripe billing.
- **13 tables**, all captured in `scripts/schema_*.sql`; a fresh DB builds from them alone
  in order (`schema.sql → schema_accounts.sql → schema_orgs.sql`). **Verify with the drift
  guard `scripts/verify-schema.sh`** (diffs a fresh build's full column signature — type +
  **nullability** — against a live DB) **plus a signup smoke test** before shipping schema
  changes (see DEVOPS.md § migrations).

## 7. DB state

- `geo_ranges`: **staging** = 306-row sample; **prod RDS = full 10,141,528 rows**
  (IP2Location, `dataset_version=ip2location-2026-07`, coverage `0.0.0.0..255.255.255.255`).
- Test accounts exist from today's proving (org/owner/member/cfgtest + admin id 5 on staging,
  `ben@` admin id 9 on prod). Org 6 (`Digital Consultancy`) is a **live paying** hobby org.
  Safe to `TRUNCATE accounts, api_keys, credit_ledger, usage_records, refill_state,
  subscriptions, processed_webhooks, organizations, org_members, org_invites
  RESTART IDENTITY CASCADE;` if a clean slate is wanted — but that removes the admin;
  re-create + promote after.

## 8. Design / brand

- **Canon palette (Báizhēn character sheet):** Deep Green `#1F3D36`, Jade `#6BAA9B`,
  Gold `#D1B46A`, White Tiger `#F3F4F6`, Metal Gray `#BFC5C9`. Applied to www, docs, portal.
- **Logo:** traced chibi tiger from user artwork → `src/www/assets/logo-mark.svg` +
  portal `TigerMark.tsx`. Live on every surface.
- **Frontend workflow** saved as a **pending Skill Workshop proposal**
  `frontend-render-verify` (not yet applied — awaiting explicit approval).

## 9. Open threads / next steps

> Portal feature backlogs now live in their own specs:
> **User** → [PORTAL-PLAN.md §4](./PORTAL-PLAN.md); **Admin** → [ADMIN-PLAN.md §4](./ADMIN-PLAN.md).

- [ ] `usage_records.caller_ip` logs the proxy/ALB IP — honor `X-Forwarded-For` for the real client IP.
- [ ] **Phase 6 scaling remainder:** Redis cache (hot IPs + credit-balance write-behind),
  RDS read replicas, ALB req/sec rate limiting. (VM/RDS/ALB/S3/CloudFront + live cutover DONE.)
- [ ] Optional: more email triggers (payment receipt, credit-exhausted).
- [ ] Prune `src/frontend/` (unused Minimal-kit reference base) if it's truly dead weight.

**DONE since last handoff (2026-07-27):** full prod stack (VM+RDS+ALB+S3/CloudFront),
live-mode Stripe (products/prices/webhook), full 10.1M geo import to RDS, unified prod-API
cutover (www/app/admin), new traced tiger brand + inverted squircle mark, signup confirm-
password + org-name, password recovery, dashboard analytics chart, request-history page,
migrations made inline (13 tables, fresh-build clean), and two prod hotfixes (signup
`account_id NOT NULL` drift D7; org-rename PATCH response shape).

## 10. Ops cheatsheet

> **Deploy order (Rule 4): staging → verify live → commit/push → prod.** Prod frontends
> and staging frontends are **separate deploy actions** (different pipelines). See
> workspace `TOOLS.md` Rule 4 + [DEVOPS.md](./DEVOPS.md) § Production for full detail.

```bash
# ── STAGING (dev box) ──
sudo systemctl status|restart meowtrace-api      # staging API (:8080)
sudo journalctl -u meowtrace-api -f
cd src/portal && VITE_API_BASE=https://api.staging.meowtrace.com npm run build
sudo cp -r dist/* /var/www/{app,admin}.staging.meowtrace.com/   # + prune old index-*.js
sudo cp -r src/www/* /var/www/www.staging.meowtrace.com/

# ── PROD ── (agent has SSH to VM + scoped AWS key: S3 write + CloudFront invalidate)
ssh -i ~/.ssh/meowtrace_prod_vm ubuntu@13.228.3.109
  cd /opt/meowtrace && git pull && sudo systemctl restart meowtrace-api   # backend
  # migrate RDS: psql "$DSN" -f scripts/schema.sql → schema_accounts.sql → schema_orgs.sql
# frontends (from dev box, AWS profile meowtrace-deploy, region ap-southeast-1):
cd src/portal && npm run build         # .env defaults VITE_API_BASE=https://api.meowtrace.com
aws s3 sync dist/ s3://s3.meowtrace.com/app/prod/   --delete
aws s3 sync dist/ s3://s3.meowtrace.com/admin/prod/ --delete
aws s3 sync src/www/ s3://s3.meowtrace.com/www/prod/ --delete
aws cloudfront create-invalidation --distribution-id <E1Z2…|ETB8…|E9OX…> --paths '/*'

# ── Prod DB shell (DSN from prod TOML; password is %-encoded in the URL) ──
sudo cp /etc/meowtrace.production.toml /tmp/t.toml && sudo chown ubuntu /tmp/t.toml
DSN=$(cd /opt/meowtrace/src/backend && node -e "const{parse}=require('smol-toml');console.log(parse(require('fs').readFileSync('/tmp/t.toml','utf8')).database.url)")
psql "$DSN"   # then rm /tmp/t.toml
```

---

## Changelog
- **2026-07-27 (late)** — **Post-launch prod hardening: drift D8–D10, coupons, schema guard, admin-on-prod.**
  Fixed three live prod incidents, all the same staging↔prod drift class: **D8** — lookup
  API 500'd on every authed call (`accounts.suspended` missing on prod RDS; captured +
  applied). **D9** — prod RDS was missing the whole `processed_webhooks` table (Stripe
  webhook idempotency would 500); caught by the **new `scripts/verify-schema.sh` drift
  guard** on its first prod run. **D10** — a **real paid checkout** left the org on FREE:
  the webhook crashed on `subscriptions.account_id NOT NULL` (the D7 pass missed this
  table); dropped the constraint, provisioned the org (→ hobby, 200k credits, active sub
  via a native webhook replay), and **hardened the guard to diff nullability** so this
  class can't hide again. Also: **Stripe promotion codes** enabled on Checkout
  (`allow_promotion_codes`); **admin `ben@meowtrace.com` copied staging→prod** (same
  password). Docs reconciled to shipped reality (SPEC §5.3/§10, API.md org+admin endpoints
  + `/v1/me` shape + live result envelope, DEVOPS guard, DATA-IMPORT D8–D10) and a
  [CHANGELOG.md](./CHANGELOG.md) added.
- **2026-07-27** — **Production launch + portal feature batch + hotfixes.** The whole stack
  went **live on meowtrace.com**: prod VM `13.228.3.109` (nginx default_server → Fastify
  :8090), **RDS** (full 10.1M `geo_ranges`), **ALB** (ACM TLS, `/status` health), and
  **S3/CloudFront** for www/app/admin — all unified on `api.meowtrace.com`. **Live-mode Stripe**
  (products/prices/webhook `we_1TxUjh…`). Shipped a **portal feature batch**: signup
  confirm-password + optional org name; **password recovery** (`/v1/auth/forgot`|`reset` +
  `/reset` page, SendGrid); **dashboard requests-over-time chart** (recharts, from
  `usage_rollup_minute`); **request-history** page (`/requests`). New **traced tiger** brand
  across all surfaces + `TigerMark variant="inverted"` squircle for light surfaces.
  **Migrations made inline** — fresh DB builds from `scripts/schema_*.sql` alone (captured
  `processed_webhooks`, fixed cross-file ordering, split GRANTs; drift D6). **Two prod
  hotfixes:** signup 500 from `credit_ledger.account_id NOT NULL` staging↔prod drift
  (D7 — dropped NOT NULL on org-rescoped tables); org-rename white-screen from `PATCH
  /v1/orgs` returning a partial shape (now returns full `OrgDetail`). Docs updated: SPEC §5.3
  (password_resets/usage_rollup_minute + account_id nullable), API.md (auth reset + usage
  endpoints + org_name signup), DEVOPS.md (Production topology + inline-migration procedure),
  DATA-IMPORT.md (D5–D7). **New workspace rule:** TOOLS.md Rule 4 (staging→verify→commit→prod).
- **2026-07-20** — **Docs restructure + auto-deploy + lookup contract v2 + schema fix.**
  (1) **Portal specs split out:** added [PORTAL-PLAN.md](./PORTAL-PLAN.md) + [ADMIN-PLAN.md](./ADMIN-PLAN.md)
  (full living specs); migrated portal feature detail out of SPEC (usage-tracking *model*
  stays in SPEC §9.1). (2) **Docs auto-deploy (Path C):** `.git/hooks/post-commit` →
  `scripts/deploy-docs.sh` (rsync mirror) — repo is source of truth, served copy self-heals,
  no drift; fixed a stale-served-SPEC bug. (3) **Lookup response contract v2 (BREAKING):**
  single `{ result:{...} }`, bulk `{ results:[{status,result?}] }`, body `country:{code,name}`
  + `region` + `city` (dropped `_name`); built in `src/backend`, www demo updated, verified
  live over HTTPS. (4) **SPEC §5.3 completed:** added the orgs-era tables that had drifted
  (organizations/org_members/org_invites/refill_state/processed_webhooks), reconciled to live
  schema. (5) **Portal feature decisions** recorded in the new plans (confirm-pw signup,
  sidenav both surfaces, org-details mgmt; dropped billing alerts; lookup UI single-IP only).
- **2026-07-20** — **API CORS fix.** Admin/user portal login was browser-blocked (no `Access-Control-Allow-Origin` from the Fastify API). Installed `@fastify/cors`, registered in `server.ts` (before routes) with explicit origins `admin.`/`app.`/`www.staging.meowtrace.com` + headers `Content-Type`/`Authorization`/`x-api-key`; bearer auth so `credentials` off. Restarted `meowtrace-api`, verified preflight `204`+ACAO (allowed) and refusal (stranger). Added `src/portal/.env` for `VITE_API_BASE`. Ref: [DEVOPS.md](./DEVOPS.md) §API.
- **2026-07-20** — Created at session close. Captures Phases 0–5 + orgs (5b) on staging:
  Postgres data core, Fastify API, accounts/credits/429, Stripe billing, www/docs/portal
  frontends, organizations (invites/switching), TOML config consolidation
  (`meowtrace.staging.toml`), SendGrid (no-reply@), canon palette + traced chibi tiger logo.
  Admin provisioned (credentials held out-of-band, not in docs). Phase 6 deferred.
- **2026-07-20** — **Security: purged a leaked admin password + email** from §5 (this
  file is publicly served). Credentials are no longer stored in any doc.
  ⚠️ **Rotate that admin password** — it was committed to git history and served over
  HTTPS; treat it as compromised.