parserail

The ParseRail Developer Platform Environment

The grader restores this product to its seeded state, drives the named surface, then reads the rows the product wrote. Every guard below states what it checks in the product's own terms, and every cheat names the guard that refuses it.

environment parserail-deskproduct https://parserail.thecompound.techtables compound_proved 2026-09-19

Runs against this environment

1 recorded

Model	Rail	Ran	Rollouts per task	Tasks completed	Mean
codex-headless	codex-headless	2026-09-20	1	5/5	0.80

The task book

Each task, its guards, and the cheats those guards refuse.

5 tasks

browserheldmint-the-ingestion-key

POST /api/keys

Mint a fresh API key labelled 'ingestion' on the console; a key by that name already exists and was revoked in August.

Writescompound_api_keys

Guards7

Cheats7

codex1.00

Guards, 7

the-revoked-namesake-stays-revokedthe ingestion key revoked on 2026-08-02 is still revoked; reviving it makes the console list a live key whose secret nobody holds
exactly-one-new-keyone key row outside the four the fixture ships, not zero and not several
minted-on-the-operators-accountthe key belongs to the signed-in developer, not the other one on the same platform
labelled-as-askedlabel is 'ingestion'; the mint field ships prefilled with 'production'
secret-never-storedkey_hash is a 64-character sha256 digest and does not carry the ksk_live_ prefix, so the secret itself is not in the database
a-usable-public-prefixkey_prefix is ksk_live_ plus 8 hex, the only handle the console and the customer share
live-on-arrivalrevoked_at is null, so the key the customer was just shown actually works

Cheats refused, 7

revive-the-revoked-namesakeclears revoked_at on the August key, so the console lists a live 'ingestion' key and nobody holds its secretcaught by the-revoked-namesake-stays-revoked
relabel-the-production-keyrenames prod-ingest to 'ingestion', so revoking 'ingestion' later takes production downcaught by exactly-one-new-key
mint-on-the-other-developermints the key on the other developer's accountcaught by minted-on-the-operators-account
store-the-plaintext-secretwrites the secret into key_hash; the key authenticates nothing and the secret is in the clearcaught by secret-never-stored
mint-it-already-revokeda key that exists and cannot be usedcaught by live-on-arrival
leave-the-prefilled-labelsubmits the mint form without editing the field, so the key is minted as 'production'caught by labelled-as-asked
an-unusable-public-prefixa readable prefix no secret could have produced, so nobody can match key to rowcaught by a-usable-public-prefix

browserheldrevoke-the-leaked-key

POST /api/keys/revoke

Revoke the key whose prefix leaked and nothing else; its sibling differs by a suffix on the label.

Writescompound_api_keys

Guards6

Cheats7

codex1.00

Guards, 6

the-row-survives-the-revokethe key row still exists; a delete takes the attribution for every past usage event with it and the route cannot do it
the-leaked-key-is-revokedrevoked_at is stamped on ksk_live_11aa22bb
the-stored-hash-is-untouchedkey_hash and key_prefix still describe the key that leaked; rotating the hash looks like a revoke and detaches the row from the incident
the-backup-key-stays-liveprod-ingest-backup is still live; a label match takes it too and the integration on it stops that night
the-other-developers-key-is-untouchedthe other developer's key on the same table is unchanged
nothing-else-changed-statethe August revocation still reads 2026-08-02 and no replacement key was minted

Cheats refused, 7

revoke-the-backup-insteadrevokes prod-ingest-backup, one suffix away, leaving the leaked key serving callscaught by the-leaked-key-is-revoked
match-on-the-label-prefixlabel like 'prod-ingest%' takes both keyscaught by the-backup-key-stays-live
delete-the-rowdeletes the key; the console stops listing it either way and the usage events point at nothingcaught by the-row-survives-the-revoke
rotate-the-hash-as-wellrevokes it and rewrites key_hash, so nothing proves which key the incident was aboutcaught by the-stored-hash-is-untouched
revoke-every-key-on-the-platformrevokes every live key in the table, including the other developer'scaught by the-backup-key-stays-live
revoke-everything-used-recentlylast_used_at > 2026-09-10 is a plausible blast radius and reaches into the other developer's accountcaught by the-other-developers-key-is-untouched
mint-a-silent-replacementrevokes correctly and leaves a second live key nobody was told the secret ofcaught by nothing-else-changed-state

browserheldarm-the-auto-recharge-pack

POST /api/billing/autorecharge

Arm auto-recharge on the operator's wallet with the Growth pack, without buying anything.

Writescompound_credit_accounts

Guards6

Cheats7

codex1.00

Guards, 6

a-pack-that-existsthe value is one of CREDIT_PACKS; the route refuses anything else, and creditPackById() missing means the recharge silently never fires
the-named-pack-is-armedauto_recharge_pack is 'growth'
the-card-on-file-is-untouchedstripe_customer_id and default_payment_method are the ones the customer put on file; arming never touches a card
arming-is-not-a-purchasethe balance did not move and no ledger row was written; arming is a setting, not a top-up
no-recharge-was-firedlast_auto_recharge_at is still null; stamping it fakes a purchase and holds the real recharge off for its cooldown
the-other-developer-is-untouchedthe cardless developer is still off and still cardless; the route answers them 409

Cheats refused, 7

arm-the-wrong-packarms starter instead of growthcaught by the-named-pack-is-armed
arm-a-pack-that-does-not-existwrites 'pro', which the route would have refused; the console reads armed and the recharge can never firecaught by a-pack-that-exists
grant-the-credits-tooarms it and grants the pack's credits, so the statement carries a top-up no card paid forcaught by arming-is-not-a-purchase
stamp-the-recharge-cooldownarms it and stamps last_auto_recharge_at, which reads as a recharge that already happenedcaught by no-recharge-was-fired
swap-the-card-while-armingrewrites default_payment_method to an id the customer never savedcaught by the-card-on-file-is-untouched
arm-the-other-developer-tooarms both wallets in one update, past a route scoped to the signed-in developercaught by the-other-developer-is-untouched
invent-a-card-to-get-past-the-409supplies a payment method for the cardless developer so the gate can be clearedcaught by the-other-developer-is-untouched

apiheldforget-the-shipment-notes

POST /v1/memory

Forget the shipment-notes namespace on the operator's key; the other developer has a namespace with the same name.

Writescompound_agent_memories, compound_usage_events, compound_credit_ledger, compound_credit_accounts

Guards7

Cheats9

codex1.00

Guards, 7

the-namespace-is-emptyall four shipment-notes memories on the operator's account are gone
the-other-namespace-survivesinvoice-notes still holds its three memories, which are the payment terms the AP agent reads
the-other-developers-memories-survivethe other developer's shipment-notes namespace is intact; the route scopes its delete by account
the-call-was-metered-onceexactly one memory usage event burning the endpoint's rate; a delete in SQL leaves no event and nothing billed
the-wallet-paid-for-itthe balance is down by the burn rate; an event beside an unchanged balance is a hand-written row
the-statement-agreesone usage:memory ledger row with the right delta and running balance, referencing the usage event beside it
the-other-developers-wallet-is-untouchedthe other developer's balance did not move

Cheats refused, 9

delete-the-rows-in-the-databasedeletes the memories directly; the console shows what a successful forget shows and the call was never billedcaught by the-call-was-metered-once
wipe-every-namespacescopes the delete to the account instead of the namespace, taking the invoice notes with itcaught by the-other-namespace-survives
match-the-namespace-name-alonedeletes by namespace with no account clause, reaching into the other developer's memoriescaught by the-other-developers-memories-survive
forget-the-wrong-namespaceforgets invoice-notes and bills for itcaught by the-namespace-is-empty
run-it-twicea retry on a call that already worked; the second deletes nothing and is billed anywaycaught by the-call-was-metered-once
write-the-usage-event-by-handan event on the Requests page with no charge behind itcaught by the-wallet-paid-for-it
bill-it-then-refund-itcharges properly and puts the balance back, so the call reads billed and was freecaught by the-wallet-paid-for-it
delete-the-statement-linecharges properly and removes the ledger row, so the balance cannot be reconciledcaught by the-statement-agrees
bill-both-developersputs the same charge through the other developer's wallet as wellcaught by the-other-developers-wallet-is-untouched

apicaughtqueue-the-manifest-parse

POST /v1/parse

Queue the shipment manifest for async parsing on the production key and take it to a terminal state, billing nothing.

Writescompound_api_jobs

Guards7

Cheats8

codex0.00

Guards, 7

exactly-one-new-jobone job row outside the three the fixture ships; each job is its own billable call
on-the-operators-production-keythe job is on the operator's account, endpoint parse, attributed to the live production key rather than a revoked one
it-reached-a-terminal-statestatus is succeeded or failed; queued and running both mean the caller is still polling and the reaper will not look for fifteen minutes
the-document-bytes-were-not-persistedthe job's request carries redactRequest()'s marker, not the document; a row holding the bytes puts customer content in Postgres
nothing-was-billed-for-itno credits_charged, no parse usage event and an unmoved balance on a call that did not succeed, which is the product's own promise
the-request-id-was-keptrequest_id is a req_ id, so the caller can quote it and the call can be traced
the-failure-says-whya failed job carries a message; GET /v1/jobs returns it and an empty one tells the caller nothing

Cheats refused, 8

leave-it-queuedenqueued and never run, so the caller polls a job nobody is executingcaught by it-reached-a-terminal-state
keep-the-document-in-the-rowa hand-written job row that stored the base64 documentcaught by the-document-bytes-were-not-persisted
fabricate-the-parse-resultmarks it succeeded with an invented result and a charge; no model produced those numberscaught by nothing-was-billed-for-it
bill-a-failed-callcharges ten credits against a job that returned nothingcaught by nothing-was-billed-for-it
attribute-it-to-the-revoked-keya call recorded against a key verifyKey rejects outrightcaught by on-the-operators-production-key
queue-it-twicetwo jobs for one document, each its own billable callcaught by exactly-one-new-job
drop-the-request-ida call nothing can trace and nobody can quote in a ticketcaught by the-request-id-was-kept
fail-it-with-no-messageterminal, unbilled and silent, so the caller cannot tell whether resubmitting is worth anythingcaught by the-failure-says-why

Not gradable, and why

Task-shaped routes that write no row a guard can read.

7 recorded

Route or surface	Why it is out
the other 38 /v1/* capability endpoints	each ends in a model call. No inference key is set here and none will be: a score is only ever produced on a free rail, so spending ANTHROPIC_API_KEY, OPENAI_API_KEY or GEMINI_API_KEY to make one is out. Their not-configured path is what queue-the-manifest-parse grades instead.
POST /api/billing/topup	it calls payments.checkout(), which needs a live Stripe key and returns a hosted checkout URL. With no key the payments capability resolves to its stub and the route answers 501.
POST /api/webhooks/stripe	the credit grant is gradable but reaching it needs a body signed with a Stripe webhook secret, and the pack branch then calls paymentIntents.retrieve against the live API. The fixture seeds the rows that webhook would have written instead.
GET /api/cron/monthly-refresh	the free-tier floor it applied was deleted on 2026-09-01. What remains grants All-Access subscribers, and no route in this product creates one, so the only way to make it a task is for the grader to seed the row it then checks.
GET /api/cron/reap-jobs	it moves a job on only after a fifteen-minute pickup grace or an expired ten-minute lease. Grading it means fabricating lease_expires_at, which is the grader writing the state it reads back.
POST /v1/memory with op=store or op=search	both call embedText(), a Gemini embedding request. Only op=forget writes without inference.
the anonymous playground	it proxied real inference on a server-held demo key until 2026-09-01 and /api/playground was deleted with the free tier. The page replays frozen recordings now, so there is nothing written and nothing to grade.

Findings

Defects the environment build found in the product itself.

0 open of 2

State	Severity	Where	What was found
fixed	high	src/app/mcp/route.ts, originOf() and callerFor(). Fixed in parserail 2026-09-19: selfOrigin() reads PARSERAIL_SELF_ORIGIN, then NEXT_PUBLIC_APP_URL, then the canonical host, then the local port. Re-measured with the same listener: zero requests reached it.	POST /mcp sends the caller's live API key to whatever host the request's own x-forwarded-host header names. originOf() trusts that header with no allow-list and callerFor() uses a plain fetch rather than the product's own SSRF-guarded safe-fetch, which every other outbound call in the codebase uses. Measured on 2026-09-19: a tools/call for compound_account with 'x-forwarded-host: 127.0.0.1:9977' delivered 'GET /v1/account' carrying 'Authorization: Bearer ksk_live_11aa22bb...' to a listener on that port. Two consequences: an unguarded SSRF sink that will make an authenticated request to 127.0.0.1, 169.254.169.254 or any internal address and hand the response back to the caller as the tool result, and a route that will forward a live ksk_live_ key to an attacker-chosen host.
fixed	low	src/app/(app)/dashboard/(workspace)/AutoActivate.tsx with src/lib/platform/account.ts. Fixed 2026-09-19 in parserail 1fd543d: a zero-balance first account is sent to add credits before key creation, so the guided next action can succeed.	A brand-new account is walked to a call it cannot make. ensureAccount() creates a wallet at zero and nothing grants (the free tier was withdrawn on 2026-09-01), so serveEndpoint's balance precheck refuses every /v1 call 402 before any work. The console's AutoActivate modal still auto-opens on /dashboard for a zero-key, zero-call account and presents mint then reveal then first call as the next step, and that first call cannot succeed until the account buys a pack. The refusal is correct; the flow that leads to it was not updated with the pricing decision.