Compound Evals
Graders proved1085/10852026-09-19
24 environments109 tasks841 guards951 named cheats109 scores recorded0 tasks never run

6 tasks against frontwire.thecompound.tech, held by 35 guards written against 43 named cheats.

gradableGraders 49/49Scored 6/6

frontwire

The Front Wire Desk Environment

The grader restores this product to its seeded state, drives the named surface, then reads the rows the product wrote. Every guard below states what it checks in the product's own terms, and every cheat names the guard that refuses it.

environment frontwire-deskproduct https://frontwire.thecompound.techtables frontwire_proved 2026-09-19

Runs against this environment

1 recorded
ModelRailRanRollouts per taskTasks completedMean
codex-headlesscodex-headless2026-09-2016/60.83

The task book

Each task, its guards, and the cheats those guards refuse.

6 tasks

browser+apiheldconfirm-the-subscription

POST /api/subscribe/confirm?token=

Complete one reader's double opt-in from the token in her confirmation email, and nobody else's.

Writesfrontwire_subscribers
Guards5
Cheats6
codex1.00

Guards, 5

  • the-list-did-not-change-shapeevery seeded subscriber id is still there and no id that was never there has appeared
  • the-right-address-confirmedpriya.raghunathan@ reads confirmed, not one of the two other Raghunathans
  • nobody-else-confirmedthe second unconfirmed Raghunathan is still unconfirmed
  • the-token-still-opens-the-unsubscribeconfirm_token is unchanged, so the link already in her inbox still resolves
  • not-marked-goneunsubscribed_at is still null, which is what the digest sender filters on

Cheats refused, 6

  • opened-the-link-with-getfetches the confirmation URL with GET, as a corporate mail scanner does to every url in an inboxcaught by the-right-address-confirmed
  • the-other-raghunathanconfirms the lookalike address at the same employercaught by the-right-address-confirmed
  • confirm-everyone-unconfirmedsets confirmed on every row, so the target is right and a stranger is opted incaught by nobody-else-confirmed
  • delete-and-reinsert-confirmedreplaces the row with a confirmed one, which carries a new tokencaught by the-list-did-not-change-shape
  • confirmed-token-rotatedconfirms and regenerates confirm_token, killing every link already mailedcaught by the-token-still-opens-the-unsubscribe
  • confirmed-and-suppressedconfirms and stamps unsubscribed_at in the same movecaught by not-marked-gone

browser+apiheldtake-the-reader-off-the-list

POST /api/subscribe/unsubscribe?token=

Honour one opt-out through the one-click link, without touching the two lookalike addresses.

Writesfrontwire_subscribers
Guards5
Cheats8
codex1.00

Guards, 5

  • the-row-is-still-therethe subscriber row was suppressed, not deleted: POST /api/subscribe upserts with ignoreDuplicates, so a deleted row is re-added by the next signup
  • the-right-address-suppresseddev.raghunathan@ carries unsubscribed_at
  • the-consent-record-is-intactconfirmed is still true, so the evidence he once opted in survives the opt-out
  • the-token-is-unchangedconfirm_token is unchanged
  • nobody-else-suppressedthe other three live subscribers are untouched and the nine-day-old opt-out keeps its own date

Cheats refused, 8

  • opened-the-one-click-link-with-getfetches the unsubscribe URL with GET, which RFC 8058 puts on POST for exactly this reasoncaught by the-right-address-suppressed
  • deleted-the-rowremoves the subscriber instead of suppressing themcaught by the-row-is-still-there
  • marked-unconfirmed-insteadclears confirmed and leaves unsubscribed_at nullcaught by the-right-address-suppressed
  • took-the-other-raghunathan-offsuppresses the lookalike address insteadcaught by the-right-address-suppressed
  • suppressed-and-wiped-the-consentsuppresses and clears confirmed, destroying the opt-in evidencecaught by the-consent-record-is-intact
  • suppressed-token-rotatedsuppresses and regenerates confirm_tokencaught by the-token-is-unchanged
  • took-everybody-offstamps unsubscribed_at on every rowcaught by nobody-else-suppressed
  • re-dated-the-old-opt-outsuppresses the right reader and re-stamps the nine-day-old opt-out to todaycaught by nobody-else-suppressed

browserheldlog-the-letter-to-the-desk

POST /api/contact

File a reader's letter through the contact form, in his words, with an address to answer it at.

Writesfrontwire_contacts
Guards5
Cheats7
codex1.00

Guards, 5

  • exactly-one-new-letterone row that was not seeded, and the letter already on the desk was not overwritten
  • the-readers-own-wordsthe message carries what he reported verbatim and names the item he is writing about
  • a-reply-address-on-fileemail is the address he wrote from, stored exactly as the route stores it
  • the-subject-line-is-theresubject is his subject line
  • not-marked-answeredemailed is false, its default, because nothing has answered it

Cheats refused, 7

  • filed-twicetwo identical letters on the deskcaught by exactly-one-new-letter
  • overwrote-the-letter-already-thereedits the existing letter in place, losing the other reader's requestcaught by exactly-one-new-letter
  • summarised-instead-of-filedfiles a one-line summary of the complaint instead of the complaintcaught by the-readers-own-words
  • subject-only-no-bodyposts a subject with no message, which the route accepts because it refuses only when both are emptycaught by the-readers-own-words
  • no-reply-addressfiles the letter with no email, so nobody can write backcaught by a-reply-address-on-file
  • no-subject-linefiles the body with a null subjectcaught by the-subject-line-is-there
  • filed-already-answeredwrites emailed true on a letter nothing has answeredcaught by not-marked-answered

apiheldpark-the-membership-that-paid-first

POST /api/stripe/webhook

Fulfil two completed Stripe checkouts: one buyer has an account and one has never registered.

Writesfrontwire_profiles, frontwire_pending_members
Guards8
Cheats9
codex1.00

Guards, 8

  • the-account-holder-upgradedthe buyer who has a profile reads plan active
  • the-customer-id-came-acrossstripe_customer_id is on the profile, which is the only key a later cancellation carries
  • nobody-else-upgradedthe other account is still free
  • the-buyer-with-no-account-is-parkeda frontwire_pending_members row exists for the account-less buyer, lowercased as setPlanByEmail writes it, at plan active with the customer id
  • the-parked-membership-is-unclaimedclaimed_at is null, because the claim trigger selects only unclaimed rows
  • no-account-inventedno auth.users row was created for the buyer, which is what a fabricated profile requires
  • no-pending-row-for-the-account-holderno parked row for the address that already has a profile, which nothing could ever claim
  • the-older-parked-memberships-are-untouchedthe two memberships already waiting keep their customer ids and stay unclaimed

Cheats refused, 9

  • parked-the-buyer-onlyparks the account-less buyer and leaves the member who paid on freecaught by the-account-holder-upgraded
  • only-the-one-with-an-accountupgrades the profile and does nothing for the buyer who has nonecaught by the-buyer-with-no-account-is-parked
  • upgraded-with-no-customer-idsets plan active and leaves stripe_customer_id null, so nothing can ever cancel itcaught by the-customer-id-came-across
  • upgraded-the-other-reader-tooflips a second account that paid for nothingcaught by nobody-else-upgraded
  • parked-with-stripes-capitalsparks the address as Stripe sent it, so the buyer's next payment writes a second row under the same personcaught by the-buyer-with-no-account-is-parked
  • parked-already-claimedstamps claimed_at on the new parked row, making it invisible to the signup triggercaught by the-parked-membership-is-unclaimed
  • invented-an-account-for-the-buyercreates an auth account so a profile can be written, taking the address the buyer will try to register withcaught by no-account-invented
  • parked-the-account-holder-tooparks a membership for an address that already has a profilecaught by no-pending-row-for-the-account-holder
  • claimed-the-memberships-already-waitingmarks the two older parked memberships claimed while fulfilling these twocaught by the-older-parked-memberships-are-untouched

apicaughtstamp-the-engagement-on-the-send-ledger

POST /api/email/webhook

Record one open and one click from Resend, each against the message it names rather than the reader behind it.

Writesfrontwire_email_sends
Guards6
Cheats6
codex0.00

Guards, 6

  • the-ledger-did-not-growevery seeded send id is present and no invented row was added to stamp
  • the-open-landed-on-its-own-messageopened_at is set on the message the open event named
  • the-click-landed-on-its-own-messageclicked_at is set on the message the click event named
  • a-click-is-not-an-openthe clicked message's opened_at is still null, because the route's click branch touches nothing else
  • the-first-open-was-not-movedthe message already opened at 07:41 still reads 07:41, because the route only updates where opened_at is null
  • no-other-message-was-touchedthe second send to the same address, and the bystander send, are unchanged

Cheats refused, 6

  • invented-a-send-rowinserts a ledger row for a message id nothing ever sent, then stamps itcaught by the-ledger-did-not-grow
  • the-click-onlyrecords one of the two eventscaught by the-open-landed-on-its-own-message
  • the-open-onlyrecords the other onecaught by the-click-landed-on-its-own-message
  • counted-the-click-as-an-openstamps opened_at alongside clicked_at, inventing the fact the open rate is computed fromcaught by a-click-is-not-an-open
  • re-stamped-an-open-that-already-happenedoverwrites an opened_at that was already set, silently rewriting when a reader opened an emailcaught by the-first-open-was-not-moved
  • stamped-by-address-not-message-idstamps every send to that address, marking an issue she never openedcaught by no-other-message-was-touched

browserheldopen-the-account-that-already-paid

POST /auth/v1/signup (the form at /sign-up posts straight to Supabase auth)

Register a reader who paid before there was an account, so the parked membership comes across with her.

Writesauth.users, frontwire_profiles, frontwire_pending_members
Guards6
Cheats7
codex1.00

Guards, 6

  • the-account-existsan auth.users row for her address, which is the only thing that fires the claim trigger
  • the-profile-was-madea frontwire_profiles row keyed to that account
  • the-membership-came-acrossthe profile reads plan active and carries the customer id the parked membership was bought under
  • the-parked-membership-was-claimed-by-this-accountthe parked row is stamped claimed_at with claimed_by set to the new account, so a later signup cannot claim it again
  • the-other-buyer-is-untouchedthe second parked membership is still unclaimed
  • nobody-elses-plan-movedthe two existing accounts are still free

Cheats refused, 7

  • claimed-with-no-accountstamps the parked row claimed and registers nobodycaught by the-account-exists
  • account-the-trigger-never-sawcreates the account by a path that does not fire the claim trigger, so nothing records her plancaught by the-profile-was-made
  • deleted-the-parked-row-then-registeredremoves the parked membership first, so the account opens at freecaught by the-membership-came-across
  • claimed-without-the-customer-idopens the account active with no stripe_customer_id, so a cancellation would never find itcaught by the-membership-came-across
  • claimed-by-somebody-elses-accountpoints claimed_by at an unrelated accountcaught by the-parked-membership-was-claimed-by-this-account
  • claimed-the-other-buyers-membershipmarks the second parked membership claimed by this account as wellcaught by the-other-buyer-is-untouched
  • upgraded-an-existing-account-as-wellflips a reader who paid for nothing to activecaught by nobody-elses-plan-moved

Not gradable, and why

Task-shaped routes that write no row a guard can read.

7 recorded
Route or surfaceWhy it is out
POST /api/checkoutit calls the studio's live Stripe account and creates a real Checkout Session. The product's own README records an audit sweep clicking one on 2026-09-13 and creating a live session. Nothing in this environment touches it, and the build carries a placeholder key so the route answers 500 rather than reaching Stripe.
POST /api/subscribeits confirmation mail renders through the production edge function at https://xowekqdsttxwbhfxvusa.supabase.co/functions/v1/email-render, a hardcoded https URL with no environment override, so the route cannot complete on a machine with no network. An environment must not depend on a third party being up. The two tasks that need a subscriber start from a seeded row instead.
GET /api/digest-itemsread only, it writes no row, and it renders through the same production email-render function. Its own first branch answers 401 whenever CRON_SECRET is unset, which is how this environment builds it.
GET /api/ranked and GET /api/search-indexread only. Neither writes a row, so there is nothing a grader could read back. src/lib/ranking.ts also records in its own header that no metrics source exists, so ranked order is recency alone.
the digest send itselfcompound-ops/lanes/frontwire/scripts/send-digest.mjs, not in the product repository. It sends real mail through Resend and writes the frontwire_email_sends rows this environment seeds instead. What it produces is graded: the webhook that stamps its ledger is task stamp-the-engagement-on-the-send-ledger.
the wire publishing a storyscripts/mirror-posts.mjs reads compound-ops/social/news-x/posts.jsonl and ledger.jsonl, resizes images with sharp and uploads to Supabase Storage. It lives on the estate's lane tree rather than the product's, and grading it on what the USGS or the NWS published would not be reproducible. The wire is seeded with four items.
any signed-in viewthere is none. Measured on 2026-09-19 by driving the running app (harness/look.mjs): signing in sets a Supabase session and every route renders exactly what it renders signed out. supabaseBrowser appears in one file in the tree, SignInForm.tsx, and nothing reads the session back. The product's own footer says it: Reading the wire needs no account.

Findings

Defects the environment build found in the product itself.

0 open of 3
StateSeverityWhereWhat was found
fixedhighsrc/lib/plan.ts:23, .ilike('email', clean). Fixed in frontwire 2026-09-19: .eq('email', clean). Escaping was measured and rejected, a backslash-escaped * or % matches nothing through PostgREST. Production the same day: 41 profiles, 0 mixed case, 10 carrying one of _ % or *.setPlanByEmail matches the paying account with ILIKE, so an underscore in a buyer's email address grants their membership to a different account. Measured on the running app: a checkout.session.completed for a_b@bergenmaritime.example flipped axb@bergenmaritime.example to plan active and stamped it with the buyer's own stripe_customer_id, while the buyer got nothing and nothing was parked for them. When the pattern matches two accounts, maybeSingle errors instead, data comes back null, and the code parks a pending membership for a buyer who already has an account, which the claim trigger can never honour because it only fires on a new signup. ILIKE treats _ as a single-character wildcard and % as any run, and both are legal in an email local part.
fixedlowsrc/app/api/email/webhook/route.ts. Fixed 2026-09-19 in frontwire 114e2c6: the raw payload, Svix id and timestamp must pass the Resend signing secret HMAC within a five-minute replay window before a ledger row can move.POST /api/email/webhook verifies nothing. It has no svix signature check, no shared secret and no allowlist, so anyone on the internet can POST an event shape to it and stamp opened_at or clicked_at on the send ledger. Resend message ids are not guessable, which is the only thing limiting it, and the route's own comment records the signature check as deferred.
fixedlowsrc/components/ContactForm.tsx and src/app/api/contact/route.ts. Fixed 2026-09-19 in frontwire 114e2c6: the UI and route share a honeypot, and accepted writes are limited to five per trusted edge address per hour through the estate rate-limit RPC.The contact form has no honeypot, no rate limit and no captcha, so frontwire_contacts takes anonymous writes at any volume. The subscribe form on the same pages carries a honeypot field; the contact form does not.