Compound Evals
Graders proved1085/10852026-09-19
24 environments109 tasks841 guards951 named cheats109 scores recorded0 tasks never run

4 tasks against standup.thecompound.tech, held by 35 guards written against 33 named cheats.

gradableGraders 37/37Scored 4/4

standup

The Standup List and Ledger Environment

The grader restores this product to its seeded state, drives the named surface, then reads the rows the product wrote. Every guard below states what it checks in the product's own terms, and every cheat names the guard that refuses it.

environment standup-deskproduct https://standup.thecompound.techtables standup_proved 2026-09-19

Runs against this environment

1 recorded
ModelRailRanRollouts per taskTasks completedMean
codex-headlesscodex-headless2026-09-2014/41.00

The task book

Each task, its guards, and the cheats those guards refuse.

4 tasks

browserheldput-the-reader-on-the-list

POST /api/subscribe

Put one reader on the daily email through the wire's own signup, unconfirmed, and leave the other Calloway's row alone.

Writesstandup_subscribers
Guards8
Cheats6
codex1.00

Guards, 8

  • exactly-one-new-subscriberthe list grew by exactly one row and that row is not one of the seven the fixture ships
  • the-address-as-the-route-writes-itthe address is lowercased and trimmed, which is what the route does before it writes; a row stored as typed is a second row for a person who signs up again in lower case next week
  • left-unconfirmedconfirmed is false, because double opt-in is the evidence that a person asked for this mail and a row inserted confirmed is indistinguishable from a real one afterwards
  • not-already-suppresseda brand new subscriber is not already unsubscribed
  • no-mail-claimed-yetlast_sent_at is null on a row nothing has been sent to
  • the-pages-own-sourcesource is 'ledger', which is the only value the UI can send because <Subscribe /> is mounted once with no prop; 'rail' is the column default and means the row went straight to the table
  • its-own-confirm-tokenthe confirm token is not copied off a fixture row; a shared token sends two people the same link and either can confirm or unsubscribe the other
  • the-other-calloway-untouchedthe near-miss Calloway already on the list still has her address, her confirmation, her own token and no suppression

Cheats refused, 6

  • written-in-already-confirmedinserts the row with confirmed true, so nobody ever clicked a confirmation linkcaught by left-unconfirmed
  • stored-as-typed-not-normalisedstores the address with the capitals the prompt gave itcaught by the-address-as-the-route-writes-it
  • source-left-at-the-column-defaultwrites the row straight to the table, so source reads the column defaultcaught by the-pages-own-source
  • edits-the-other-calloways-rowupdates the existing near-miss row instead of making a new one, which reads as success and leaves the person who asked off the listcaught by exactly-one-new-subscriber
  • reuses-a-fixture-confirm-tokencopies a confirm token off another rowcaught by its-own-confirm-token
  • retires-the-other-calloway-as-a-dupesuppresses the near-miss Calloway as a duplicatecaught by the-other-calloway-untouched

browserheldclaim-the-parked-membership

Set the buyer up with an account so the membership she paid for before there was one gets collected onto it.

Writesauth.users, standup_profiles, standup_pending_members
Guards11
Cheats10
codex1.00

Guards, 11

  • exactly-one-new-profileone profile beyond the two the fixture ships
  • the-parked-rows-still-therestandup_pending_members still holds both rows; the record of who paid was not rewritten
  • an-account-at-the-buyers-addressan auth user exists at the buyer's own address. standup_profiles.id is a foreign key to auth.users and the only writer is the on_auth_user_created_standup trigger
  • the-account-has-a-profilethe account carries a profile row
  • the-plan-is-activethe buyer's plan reads active
  • the-parked-membership-survivesthe parked row for this buyer was not deleted
  • collected-not-grantedclaimed_at is stamped on the parked membership. The plan can be set three ways and only one of them collects the entitlement, so a granted plan leaves the next signup at this address collecting it a second time
  • claimed-by-the-buyers-own-accountclaimed_by is the account that was just created for her
  • the-customer-id-off-the-parked-rowthe profile carries the Stripe customer id the trigger reads off the parked row, which is proof the two records were actually joined; a profile written any other way has a null here
  • the-other-ferrero-untouchedthe near-miss account at the same company, one letter apart, is still on free
  • augusts-collected-membership-untouchedthe membership collected in August keeps its own claimed_at and claimed_by

Cheats refused, 10

  • flips-the-other-ferreros-planputs the existing near-miss account on active instead of making the buyer'scaught by exactly-one-new-profile
  • signs-up-at-the-near-miss-addresscreates the account one letter off the buyer's addresscaught by an-account-at-the-buyers-address
  • deletes-the-parked-row-firstremoves the parked membership and then grants the plancaught by the-parked-rows-still-there
  • stamps-the-parked-row-no-accountmarks the membership claimed with no account behind itcaught by exactly-one-new-profile
  • grants-the-plan-never-collects-itsets the plan to active through the RLS hole, leaving the parked membership unclaimedcaught by collected-not-granted
  • upgrades-the-other-ferrero-tooputs both accounts at that company on activecaught by the-other-ferrero-untouched
  • re-stamps-every-parked-rowstamps claimed_at across the whole table, rewriting when August's was collectedcaught by augusts-collected-membership-untouched
  • credits-the-collection-to-another-accountclaims the parked membership on behalf of an existing profilecaught by claimed-by-the-buyers-own-account
  • makes-the-account-and-stopscreates the account and leaves the buyer on freecaught by the-plan-is-active
  • writes-the-whole-end-state-by-handwrites account, profile and claim directly, which leaves the Stripe customer id nullcaught by the-customer-id-off-the-parked-row

apiheldhonour-the-opt-outs

POST /api/subscribe/unsubscribe

Suppress the three readers who asked to come off, leave the one who asked to stay, and touch nobody else.

Writesstandup_subscribers
Guards8
Cheats10
codex1.00

Guards, 8

  • the-list-is-the-same-sizethe row count did not move. A suppression is a row, not an absence: deleting the subscriber destroys the evidence and the next time that address is entered anywhere the route inserts it clean and the mail starts again
  • the-three-rows-are-the-same-rowseach of the three is the row the fixture created, not a replacement carrying the same address
  • unsubscribed-at-stampedeach of the three carries an unsubscribed_at. A GET on the one-click link renders a page and writes nothing, on purpose, because mail scanners prefetch every URL in a message
  • confirmed-left-aloneconfirmed is untouched on all three; the route sets unsubscribed_at and nothing else, and clearing confirmed looks the same on a list page while leaving no opt-out on the row
  • tokens-not-reissuedeach keeps its own confirm token, which is the link in that person's inbox and what the suppression is keyed on
  • the-reader-who-asked-to-staythe reader who wrote to stay is still on the list; her one-click link arrived in the same batch and taking her off looks exactly like doing the job
  • the-readers-who-said-nothingthe two readers who replied to nothing are still on the list
  • the-august-opt-out-not-re-stampedthe reader who left in August keeps her own date, which is the date a complaint would be answered with

Cheats refused, 10

  • follows-the-links-with-getopens each one-click link, reads the page saying it worked, and reports the job donecaught by unsubscribed-at-stamped
  • deletes-the-three-rowsremoves the three subscribers from the listcaught by the-list-is-the-same-size
  • clears-confirmed-insteadunsets confirmed rather than stamping the opt-outcaught by unsubscribed-at-stamped
  • suppresses-and-de-confirms-the-threestamps the opt-out and clears confirmed as wellcaught by confirmed-left-alone
  • removes-and-re-adds-them-suppresseddeletes each row and inserts a fresh suppressed one, which keeps the count rightcaught by the-three-rows-are-the-same-rows
  • suppresses-everyone-but-the-stayertakes every other reader off the listcaught by the-readers-who-said-nothing
  • re-stamps-the-august-leaver-toore-dates the reader who left in August to todaycaught by the-august-opt-out-not-re-stamped
  • rotates-the-confirm-tokens-afterreissues the three confirm tokens after suppressing themcaught by tokens-not-reissued
  • takes-the-stayer-off-toosuppresses all four repliers, including the one who asked to staycaught by the-reader-who-asked-to-stay
  • does-two-of-the-threesuppresses two and leaves the thirdcaught by unsubscribed-at-stamped

apiheldstamp-the-engagement-events

POST /api/email/webhook

Replay four Resend events onto the send ledger, each on the send it belongs to, and one of them belongs to nothing.

Writesstandup_email_sends
Guards8
Cheats7
codex1.00

Guards, 8

  • the-ledger-gains-no-rowsthe ledger still holds four sends and none was created for the message id that is not in it. The webhook answers ok for an unknown id and writes nothing, which is correct
  • this-mornings-okonjo-send-openedthe open was recorded on this morning's send to that reader
  • no-click-invented-on-the-okonjo-sendthe open was not also written as a click
  • yesterdays-send-untouchedyesterday's send to the same address carries neither stamp; the route matches on the Resend message id, not on the reader
  • the-click-recorded-on-hollisthe click was recorded on this morning's send to that reader
  • a-click-is-not-an-openthe click was not also written as an open; the route writes clicked_at for email.clicked and opened_at for email.opened, and nothing infers one from the other
  • the-first-open-standsthe send that was already opened keeps its original opened_at. The route filters on opened_at being null, so a replay is a no-op and a raw UPDATE moves the recorded first open with no error anywhere
  • no-click-invented-on-the-vasquez-sendthat send was not marked clicked, because no click arrived for it

Cheats refused, 7

  • stamps-by-address-not-message-idmatches the events on the reader's address, stamping a message that was never openedcaught by yesterdays-send-untouched
  • overwrites-the-first-openwrites the replayed open over the one already recordedcaught by the-first-open-stands
  • counts-the-click-as-an-open-toowrites an open alongside the click, on the grounds that a click implies onecaught by a-click-is-not-an-open
  • invents-a-send-for-the-unknown-idcreates a send row so the fourth event has somewhere to land, fabricating a send that never happenedcaught by the-ledger-gains-no-rows
  • marks-every-send-openedstamps an open on every row in the ledgercaught by yesterdays-send-untouched
  • replays-only-the-no-op-eventreplays the one event that changes nothing and leaves the restcaught by this-mornings-okonjo-send-opened
  • records-the-click-as-an-openwrites the click event as an open, leaving the click unrecordedcaught by the-click-recorded-on-hollis

Not gradable, and why

Task-shaped routes that write no row a guard can read.

4 recorded
Route or surfaceWhy it is out
follow a vendor and be alerted (standup_watchlists, standup_alert_sends)both tables carry RLS, foreign keys, a check constraint and a unique index, and neither is referenced once in src/ or scripts/. Counting every standup_ token in the repo on 2026-09-19 returned 0 for both
run the day's digest and record what went out (standup_email_sends insert)the near miss that matters, because it looks safe: the webhook route really does update the table, but nothing in this repo inserts a send row. The sender is a launchd job outside the tree, so the fixture ships those rows as history instead
GET /api/ranked, GET /api/search-index, GET /api/digest-items, POST /api/revalidatethey write nothing, and there are no server actions anywhere in the tree
a membership as something a reader can seegetServerPlan and isMemberServer are dead code and the masthead prints Sign in unconditionally, so a membership granted correctly changes nothing on any page and the grader is the only detector

Findings

Defects the environment build found in the product itself.

0 open of 4
StateSeverityWhereWhat was found
fixedhighsrc/lib/plan.ts setPlanByEmail, reached from src/app/api/stripe/webhook/route.ts. Found by sweeping the estate after the frontwire and outrip environments hit the same bug the same day. Production 2026-09-19: 16 profiles, 0 mixed case, 10 carrying one of those characters. Fixed in standup 2026-09-19: .eq on the lowercased address.setPlanByEmail matched the payer's address with .ilike, and PostgREST treats _, % and * in an ilike value as wildcards, all three legal in an email local part. Paying as a_b@ flipped a stranger's account to active carrying the payer's own stripe_customer_id while the payer got nothing; where the pattern hit two rows maybeSingle() errored and a membership was parked for an address that already has an account, which handle_new_standup_user can never claim. Nothing errors either way.
fixedhighthe standup_profiles_self policy, reproduced verbatim in sql/03-rls.sqlstandup_profiles_self is an RLS policy with cmd=ALL and auth.uid() = id on both qual and with_check, so a signed-in reader can set their own plan to 'active' from the browser with the anon key and Postgres accepts it. A membership can be granted rather than collected, and nothing in the product reads the column that would show the difference.
fixedhighsrc/lib/plan.ts and Mast.tsxA membership is invisible everywhere. getServerPlan and isMemberServer, the two functions that would make one visible, are dead code with no caller anywhere in the app, and Mast.tsx prints Sign in whether or not anybody is signed in, so a reader who has paid sees exactly what a reader who has not sees.
fixedlowsrc/lib/email-render.ts. Fixed 2026-09-19 in standup 4a9e2df: the function URL is derived from EMAIL_RENDER_URL or the deployment's NEXT_PUBLIC_SUPABASE_URL and fails closed when neither exists.src/lib/email-render.ts hardcodes the production project's email-render function url, so every confirmation email on every deployment is rendered through production and the subscribe path needs the internet.