mirror of
https://github.com/Tria-plc/edr-platform.git
synced 2026-09-07 22:25:42 +00:00
test(integration): boot freight API in-process across parallel shards
The suite drove a containerized freight API, so every code change needed an image rebuild before a test could see it, and there was no way to attach a debugger. Files also ran strictly in sequence against one shared database, which is the root of the warm-stack gotchas the README documents: stowaway paid bookings climbing back aboard, a short consist on the fifth file. The freight app now boots inside each vitest worker from dist/, and each worker owns a whole shard of the topology - its own database, payment API, gateway mock and broker vhost - so nothing mutable is shared and files run in parallel. Full suite drops from roughly 20 minutes to 196s at 4 shards. - main.ts exports createFreightApp() so the harness applies the same prefix, pipes, filters and interceptors as production instead of replaying them by hand; self-start is guarded by require.main so the Dockerfile CMD still boots - booking-window tick cadence is env-driven (BOOKING_WINDOW_TICK_CRON), */1 in the suite, */10 unchanged in production - prepare-shards.mjs seeds a template database (boot seeders, then the SQL fixtures that depend on them) and clones it per shard; it.mjs re-clones on every run, so each run is hermetic - gateway mock and payment API are generated per shard: the mock keeps modes and orders process-global and 20 of 25 specs reset it in beforeAll, and the inbound CBE bill query has to reach one specific shard's app - poll() samples every 250ms instead of 2000ms, keeping the caller's deadline - authz.it.ts seeds its own invoice; it previously read another spec's leftover and returned early, which silently passed on a pristine database Known: an unlocked MAX(sequence_no)+1 in train-scheduling.service.ts races under concurrent allocation and leaves a short consist, so 1-3 specs fail intermittently. Pre-existing and reproduces at the production tick cadence.
This commit is contained in:
Nathnael
@@ -3,29 +3,77 @@
|
||||
Headless, API-level tests for the freight API running against the **real**
|
||||
`edr-payment-api`. Only the bank/wallet gateways are stubbed.
|
||||
|
||||
The freight API is **not containerized** here — it boots inside each vitest
|
||||
worker from `apps/edr-freight-api/dist`, which `it:test` rebuilds every run. A
|
||||
code change needs no image rebuild, and a breakpoint in the API is hit by the
|
||||
test that provoked it.
|
||||
|
||||
```
|
||||
pnpm it:up # build + start the stack (first run ~5 min)
|
||||
pnpm it:test # vitest run (auto-ups the stack if needed)
|
||||
pnpm it:test -- --reporter=basic src/payment-happy.it.ts
|
||||
pnpm it:logs payment-api-it
|
||||
pnpm it:down # -v, wipes the throwaway DB
|
||||
pnpm it:test --no-reset -- src/payment-happy.it.ts # skip the DB re-clone
|
||||
pnpm it:logs payment-api-it-0
|
||||
pnpm it:down # -v, wipes the throwaway DBs and the generated shard file
|
||||
|
||||
IT_SHARDS=1 pnpm it:up # 1 shard when RAM is tight or output must be readable
|
||||
```
|
||||
|
||||
## Stack
|
||||
|
||||
`docker-compose.it.yaml` is an **overlay** on `docker-compose.e2e.yaml` — same
|
||||
freight API, Postgres (tmpfs), MinIO, Fayda/eTrade mocks; plus RabbitMQ, the
|
||||
real payment API, and one gateway mock. It is a separate compose project
|
||||
(`edr-freight-it`) on offset ports, so the Cypress e2e stack can run alongside.
|
||||
`docker-compose.it.yaml` is an **overlay** on `docker-compose.e2e.yaml`, plus a
|
||||
third **generated** file (`integration/.it-shards.yaml`, written by
|
||||
`scripts/gen-shards.mjs`, gitignored) holding the per-shard services. It is a
|
||||
separate compose project (`edr-freight-it`) on offset ports, so the Cypress e2e
|
||||
stack can run alongside.
|
||||
|
||||
Each vitest worker is a whole shard of the topology. Nothing mutable is shared
|
||||
between shards, so spec files run in parallel:
|
||||
|
||||
```
|
||||
freight-api-e2e :3111 ──HTTP──> payment-api-it :3113 ──HTTP──> gateway-mock-it :4600
|
||||
^ │
|
||||
└────────── RabbitMQ :5772 ─────┘ (outbox → payment.events → consumer)
|
||||
shard i (vitest worker i, VITEST_POOL_ID)
|
||||
|
||||
freight app in-process, host :3111+i
|
||||
│ ▲
|
||||
│ └──── HTTP ──── payment-api-it-{i} :3131+i inbound CBE bill query,
|
||||
│ │ via host.docker.internal
|
||||
└── HTTP ──────────────► │ ──HTTP──> gateway-mock-it-{i} :4600+i
|
||||
▲ │
|
||||
└── RabbitMQ vhost payment_s{i} ──┘ (outbox → payment.events → consumer)
|
||||
|
||||
database edr_it_s{i} ← CREATE DATABASE … TEMPLATE edr_freight_e2e
|
||||
freight tables AND the payment API's edr_payment
|
||||
schema live in it, so `db()` and `paymentDb()` are
|
||||
one connection
|
||||
```
|
||||
|
||||
Shared by every shard: Postgres (one container, one database each), RabbitMQ
|
||||
(one container, one vhost each), MinIO, and the one-shot migration.
|
||||
|
||||
`IT_SHARDS` defaults to **4**. RAM is the ceiling, not cores — a shard is an
|
||||
in-process Nest app plus two containers.
|
||||
|
||||
Never `docker compose -f docker-compose.it.yaml` on its own — it needs the base
|
||||
file first. Use `it.mjs`.
|
||||
file first, and the generated shard file last. Use `it.mjs`.
|
||||
|
||||
## Setup order
|
||||
|
||||
The sequence is load-bearing, which is why `scripts/prepare-shards.mjs` exists
|
||||
rather than vitest's `globalSetup`:
|
||||
|
||||
1. `freight-migration-e2e` migrates the **template** database `edr_freight_e2e`.
|
||||
2. `prepare-shards.mjs` boots the freight app once against the template and
|
||||
calls `app.init()` — that fires `onApplicationBootstrap`, the always-on
|
||||
seeders (org, units, positions, permissions).
|
||||
3. …then applies the SQL fixtures, which declare exactly those as prerequisites
|
||||
(`seed-users.sql`: *"created by the API's always-on boot seeders"*).
|
||||
4. `CREATE DATABASE edr_it_s{i} TEMPLATE edr_freight_e2e` per shard — a file
|
||||
copy on the tmpfs Postgres, not a re-run of migrations and five seeders.
|
||||
5. The per-shard payment APIs start; each creates its own `edr_payment` schema
|
||||
inside its shard database.
|
||||
|
||||
`it:test` re-runs step 4 on **every run** (stop payment APIs → drop → clone →
|
||||
start), so each run is hermetic. `--no-reset` skips it for a quick rerun.
|
||||
|
||||
## Gateway mock
|
||||
|
||||
@@ -46,7 +94,15 @@ negative test. The suite drives **CBE Birr** end to end (plain HMAC, no key
|
||||
material); other providers answer a generic stub until a scenario needs them.
|
||||
|
||||
Editing `server.js` needs a container restart (`docker compose … restart
|
||||
gateway-mock-it`) — the code is a read-only mount, not baked into an image.
|
||||
gateway-mock-it-0`) — the code is a read-only mount, not baked into an image.
|
||||
|
||||
**One mock per shard.** `modes`, `calls` and `orders` are process-global in
|
||||
`server.js`, and 20 of 25 specs call `POST /__control/reset` in `beforeAll`. On a
|
||||
shared mock every starting spec would wipe every running spec's live orders —
|
||||
and that fails as a wrong *assertion*, not an error: an unknown order still gets
|
||||
a correctly-signed webhook, just carrying the mock's default amount. Per-shard
|
||||
mocks are why the suite can run `payment-failure` (which forces providers into
|
||||
`fail`/`timeout`) beside a spec that settles normally.
|
||||
|
||||
## Files
|
||||
|
||||
@@ -151,21 +207,34 @@ what it actually does and says so in a comment, so a fix fails loudly:
|
||||
ignores `deleted_at`, so a soft-unlinked booking can never be re-batched).
|
||||
- **Arrange is slow.** Contract → booking → clearance → ops accept → batch is
|
||||
30–60s of real API work per booking, so files share one schedule day.
|
||||
- Files run sequentially (one DB); concurrency is exercised inside a test with
|
||||
`Promise.all`.
|
||||
- **Files run in parallel across shards, sequentially within one.** A shard owns
|
||||
its whole topology, so two files never contend; two files on the *same* shard
|
||||
still run one after the other, which is what the day-offset partitioning and
|
||||
the per-file `releaseUnpaidHolds()` / `resetCorridorDay()` hygiene still
|
||||
assume. Concurrency inside a single test is exercised with `Promise.all`.
|
||||
- **Day offsets are the de-facto partition key** (`departureAt(n)` per file).
|
||||
`payment-happy` and `expired-invoice-late-settle` both claim day 4 — harmless
|
||||
now that a shard has its own database, but do not read the offsets as unique.
|
||||
- **A retired fixture keeps its shipment day at its peril.**
|
||||
`rescueStrandedPaidForDay` sweeps every unlinked booking whose
|
||||
`payment_status` is PAID and whose `scheduled_date` falls on the day being
|
||||
filled, and re-places it on the fresh train. A previous run's paid bookings
|
||||
therefore climb back aboard — 18 stowaway wagons on a 28-wagon day, until
|
||||
`releaseUnpaidHolds` / `resetCorridorDay` started nulling `scheduled_date`.
|
||||
filled, and re-places it on the fresh train. Within a run this still bites
|
||||
across files on one shard; *between* runs it no longer can — `it:test`
|
||||
re-clones every shard database from the template first.
|
||||
- **Only a FULL train rests at DONE.** An under-filled day CONCLUDES and
|
||||
REOPENS (`window_phase` back to OPEN, `booking_cycle_no` 2), so waiting for
|
||||
DONE there waits forever — use `pollCycleConcluded`.
|
||||
- **Wagon stock is finite and shared.** Paid bookings keep their wagons, so
|
||||
`releaseUnpaidHolds()` also frees every earlier `CTR-IT-%` allocation —
|
||||
without it the fifth or sixth file on a warm stack silently gets a short
|
||||
consist.
|
||||
- **Wagon stock is finite and shared** *within a shard*. Paid bookings keep
|
||||
their wagons, so `releaseUnpaidHolds()` also frees every earlier `CTR-IT-%`
|
||||
allocation — without it the fifth or sixth file on one shard silently gets a
|
||||
short consist.
|
||||
- **`poll()` samples every 250ms, but keeps the caller's deadline.** Callers pass
|
||||
`attempts × intervalMs`; that product is the timeout, and only the sampling
|
||||
rate changed. Override with `IT_POLL_INTERVAL_MS`.
|
||||
- **The tick cadence is env-driven.** `BOOKING_WINDOW_TICK_CRON` is `*/1` here
|
||||
and `*/10` in production. Every window phase, allocation and expiry the suite
|
||||
waits on lands on that tick. A spec that proves sensitive to it should pin its
|
||||
own value rather than the whole suite reverting.
|
||||
- **One tenant per booking.** `seedTenantContracts` mints a company per booking
|
||||
because a company may hold only one unpaid reservation at a time; staff book
|
||||
and pay on their behalf, which is also the real Path B flow.
|
||||
|
||||
Reference in New Issue
Block a user