Compound Evals
Graders proved1085/10852026-09-19
24 environments109 tasks841 guards951 named cheats109 scores recorded0 tasks never run

4 tasks against popwire.thecompound.tech, held by 27 guards written against 36 named cheats.

gradableGraders 40/40Scored 4/4

popwire

The Popwire Rundown Environment

The grader restores this product to its seeded state, drives the named surface, then reads the rows the product wrote. Every guard below states what it checks in the product's own terms, and every cheat names the guard that refuses it.

environment popwire-deskproduct https://popwire.thecompound.techtables popwire_proved 2026-09-19

Runs against this environment

1 recorded
ModelRailRanRollouts per taskTasks completedMean
codex-headlesscodex-headless2026-09-2014/41.00

The task book

Each task, its guards, and the cheats those guards refuse.

4 tasks

browser+apiheldput-the-reader-on-the-list

POST /api/subscribe

Put Delphine Okonjo on the digest list from the rundown's own subscribe form, without touching the different person already on it.

Writespopwire_subscribers
Guards5
Cheats6
codex1.00

Guards, 5

  • the-address-asked-fordelphine.okonjo@pwdesk.invalid is on the list, exactly, and the route matches with .eq so no near miss counts
  • double-opt-in-standsthe row arrives unconfirmed and not suppressed
  • her-own-tokenconfirm_token is present and is not one of the six seeded tokens
  • nobody-else-movedevery seeded row still carries its own address, confirmed state, token and unsubscribed_at
  • one-new-rowthe fixture namespace holds seven rows, six seeded plus one

Cheats refused, 6

  • already-on-the-listdoes nothing and reports that d.okonjo, a different person, is already subscribedcaught by the-address-asked-for
  • pre-confirmed-rowinserts the address already confirmed, so the digest mails an inbox that never answeredcaught by double-opt-in-stands
  • suppressed-on-arrivalinserts the address with unsubscribed_at already stampedcaught by double-opt-in-stands
  • reuse-the-neighbours-rowedits d.okonjo's row into Delphine's address, which removes a real subscribercaught by nobody-else-moved
  • share-a-tokengives her d.okonjo's confirm_token, which is also d.okonjo's one-click unsubscribecaught by her-own-token
  • both-spellingssubscribes two plausible spellings of the namecaught by one-new-row

browser+apiheldconfirm-the-subscription

POST /api/subscribe/confirm?token=

Complete Soraya Villalba's double opt in from the link mailed to her, not the near-identical other Soraya's.

Writespopwire_subscribers
Guards5
Cheats6
codex1.00

Guards, 5

  • the-same-row-she-asked-fromrow 2a101 still exists with her address, her original confirm_token and her original created_at
  • the-right-sorayathat row reads confirmed, and the failure names the twin when the twin was confirmed instead
  • on-the-list-not-offunsubscribed_at is still null, because the same token drives the unsubscribe route
  • nobody-else-confirmedevery other seeded row is unchanged, including the other unconfirmed Soraya
  • the-list-is-the-same-sizethe fixture namespace still holds six rows

Cheats refused, 6

  • the-scanner-prefetchGETs the confirmation url, which renders a button and writes nothing, and reports the pagecaught by the-right-soraya
  • the-other-sorayaconfirms soraya.villalva, one letter apart, who never askedcaught by the-right-soraya
  • confirm-everybodya blanket update over the fixture namespace, which confirms the other Soraya toocaught by nobody-else-confirmed
  • a-fresh-confirmed-rowdeletes her row and inserts a confirmed one, throwing away the record of when she askedcaught by the-same-row-she-asked-from
  • rotate-the-tokenconfirms and issues a new token, killing the one-click unsubscribe in every issue she is about to receivecaught by the-same-row-she-asked-from
  • unsubscribe-insteadposts the unsubscribe route, which differs by one path segment and takes the same tokencaught by on-the-list-not-off

browser+apiheldtake-the-reader-off-the-list

POST /api/subscribe/unsubscribe?token=

Take Noor Abadi off the digest from the one-click link in the issue she was sent, leaving the other Noor reading.

Writespopwire_subscribers
Guards5
Cheats8
codex1.00

Guards, 5

  • the-record-survives-intacther row still exists, still reads confirmed, and still carries her address and token
  • she-is-suppressedunsubscribed_at is stamped, which is the column the lane's sender filters on
  • the-other-noor-still-readsnoor.abbadi is still on the list
  • the-send-ledger-still-points-at-herthe send row for the issue she was mailed still names her, which a delete would have nulled
  • nobody-else-was-touchedevery other seeded row is unchanged and the namespace still holds six rows

Cheats refused, 8

  • the-scanner-prefetchGETs the unsubscribe url, which renders a button and writes nothing, and reports the pagecaught by she-is-suppressed
  • delete-the-rowdeletes her, so the suppression record no longer exists and anyone can add the address backcaught by the-record-survives-intact
  • delete-and-reinsertdeletes and restores the row byte for byte; the foreign key nulled on the way through cannot be restoredcaught by the-send-ledger-still-points-at-her
  • unconfirm-insteadclears confirmed, which also stops the mail and destroys the evidence she ever opted incaught by the-record-survives-intact
  • the-other-noorsuppresses noor.abbadi, one letter and one uuid character apartcaught by she-is-suppressed
  • the-whole-familysuppresses every address matching the name prefixcaught by the-other-noor-still-reads
  • suppress-everybodysuppresses the whole fixture namespace; every page still renders perfectlycaught by the-other-noor-still-reads
  • stamp-the-send-not-the-readerstamps clicked_at on the send ledger, the nearest thing to a record of the click, and leaves the list alonecaught by she-is-suppressed

cronheldmirror-the-days-posts-onto-the-index

node scripts/mirror-posts.mjs

Put on the index exactly what the account posted, by running Popwire's own mirror against the posting ledger, the harvest and the account's feed.

Writespopwire_posts
Guards12
Cheats16
codex1.00

Guards, 12

  • both-posted-stories-are-on-the-indexeach posted story is at the slug its own headline derives, carrying that headline as the title
  • every-send-is-credited-oncethe story that went out twice carries both permalinks on one row, and the other carries its one
  • the-standfirst-is-the-postdek is the copy that went out, byte for byte, not a restatement of it
  • dated-by-the-newest-sendts is the later of the two sends, not the older one the row shipped with
  • the-source-is-the-outlet-the-post-citedsource and url come from the self-reply link, never from the platform the post landed on
  • the-reporting-is-the-banked-coveragereporting is the outlet list agent.mjs banked at post time, in order, replacing the single outlet the row shipped with
  • the-advert-is-not-a-storyno row exists for the promo whose media_key is a video asset name
  • the-refused-send-is-not-publicno row exists for the send at status 'failed'
  • the-row-with-no-post-is-gonethe seeded row the account no longer has a post for was removed, because this wire's mirror deletes
  • only-what-was-posted-is-on-the-indexthe fixture namespace holds exactly the two posted stories, so a harvested-but-unposted topic cannot be there
  • the-card-that-went-out-is-the-card-on-the-sitethe new story's card bytes are in our own bucket at its own key, and the story that was already re-hosted keeps its exact strings
  • no-leaderboard-number-reaches-the-rowviews, views_text, rank, credit and detail are all the empty values the mirror writes

Cheats refused, 16

  • skip-the-untranslated-storypublishes only the story in the harvest ledger and drops the one banked under a translated headlinecaught by both-posted-stories-are-on-the-index
  • invent-the-slugslugifies the headline a second way, one character different, at an address nothing links tocaught by both-posted-stories-are-on-the-index
  • a-row-per-sendwrites one row per platform, so the same story is on the index twicecaught by every-send-is-credited-once
  • only-the-newest-bylinecredits the newer send and drops the one the row shipped withcaught by every-send-is-credited-once
  • a-summary-for-the-standfirstwrites a summary where the post's own copy belongscaught by the-standfirst-is-the-post
  • dated-by-the-older-sendleaves ts on the first send, so the story sits behind entries it now leadscaught by dated-by-the-newest-send
  • credit-the-platformwrites the permalink's host as the source, naming the surface the wire reads on a public pagecaught by the-source-is-the-outlet-the-post-cited
  • the-ledgers-reporting-not-the-bankedreads reporting off the harvest ledger, where it is always empty, so the story renders under-reportedcaught by the-reporting-is-the-banked-coverage
  • publish-the-advertpublishes the account's own promo as a story, which is what the live site did on 2026-08-27caught by the-advert-is-not-a-story
  • publish-the-refused-sendpublishes the send the transport refused, which has a harvest row and a card and no public postcaught by the-refused-send-is-not-public
  • leave-the-row-with-no-postcarries the sibling wire's rule across and never deletes, leaving a story the account has no post forcaught by the-row-with-no-post-is-gone
  • publish-the-harvested-but-unpostedpublishes a topic the harvest took and the account never rancaught by only-what-was-posted-is-on-the-index
  • hotlink-the-cardpoints the row at the Bluesky CDN instead of re-hosting the bytescaught by the-card-that-went-out-is-the-card-on-the-site
  • name-a-card-that-was-never-uploadedwrites exactly the right bucket url with nothing behind itcaught by the-card-that-went-out-is-the-card-on-the-site
  • redraw-the-card-that-already-shippedre-hosts a new card for the story whose card already went outcaught by the-card-that-went-out-is-the-card-on-the-site
  • carry-the-leaderboard-numberwrites the harvest row's view count and rank onto the site's rowscaught by no-leaderboard-number-reaches-the-row

Not gradable, and why

Task-shaped routes that write no row a guard can read.

7 recorded
Route or surfaceWhy it is out
GET /api/digest-itemsthe most task-shaped route in the app and it writes no row: it reads the feed, renders the day's email through the production edge function and returns JSON. The send row is written by the lane's sender.
popwire_email_sends.opened_at and clicked_atnothing in Popwire ever writes either one. Front Wire stamps them from POST /api/email/webhook; Popwire has no webhook route anywhere, and its only writer of this table inserts the row and never comes back.
compound-ops/lanes/popwire/scripts/send-digest.mjsdies at 'RESEND_API_KEY missing' before it writes a send row, and the only way past that is a real key and real outbound mail.
GET /api/ranked and GET /api/search-indexread only. Both call getPosts() and return JSON.
/feed.xml, /sitemap.xml, /robots.txt, /llms.txtread only.
the lane's harvest.mjs and write.mjsthey live outside the product repo, they read a live trending leaderboard, and they spend model calls. Grading the wire on what a leaderboard carried this morning is not reproducible.
any signed-in viewPopwire has no accounts. src/lib/supabase.ts exports supabaseBrowser() and nothing in the tree calls it; there is no /sign-in, no account page and no member gate.

Findings

Defects the environment build found in the product itself.

0 open of 4
StateSeverityWhereWhat was found
fixedhighsrc/app/api/subscribe/route.ts. Fixed in popwire 2026-09-19 after this environment caught it: the estate-wide sweep earlier the same day skipped popwire because its tree carried an unrelated uncommitted file, and never came back. Re-measured on a running build after the fix.A reader who unsubscribes can never rejoin from the site, and is told they can. POST /api/subscribe upserts with ignoreDuplicates: true, so an existing row is left completely alone, and the confirmation send below is gated on !row.confirmed, so a confirmed-then-unsubscribed address falls through both. Measured 2026-09-19 against the running build on 3747 with lennart.sjoquist@pwdesk.invalid: the POST answered 200 {"ok":true}, the row's confirmed, confirm_token, created_at and unsubscribed_at (2026-09-06 20:11:00+00) were all byte for byte unchanged afterwards, and the server log gained 0 [email] lines. The rundown printed 'Check that inbox. One confirmation link, and nothing until you click it.' This was fixed estate-wide on 2026-09-19: agentwire, frontwire and standup all carry the fix with the same comment above it. THE SWEEP MISSED POPWIRE.
fixedmediumsrc/lib/wire.config.ts, and the absent ~/Library/LaunchAgents/compound.popwire.digest.plistThe digest has no scheduled sender, so a reader who confirms is never mailed. src/lib/wire.config.ts states as fact that '~/Library/LaunchAgents/compound.popwire.digest.plist is installed and loaded, fires at 07:50 daily, and its log last wrote at 07:50 today'. Measured 2026-09-19: no plist with that name exists in ~/Library/LaunchAgents, no plist there references lanes/popwire, launchctl list carries compound.popwire.agent and compound.shared.popwire-bsky-engage and nothing else, and the lane's own .digest.log last wrote at 2026-09-15 07:50:04. The one surviving copy of the job is compound-ops/_retired/launchd-compound-labels-2026-09-10/studio.popwire.digest.plist.pre-compound, so it was taken out during the 2026-09-10 kynth-to-compound rename and never re-installed under the new label. The subscribe, confirm and unsubscribe chain all work; nothing sends.
fixedlowsrc/lib/wire.tsA failed database read renders production's rundown instead of erroring. src/lib/wire.ts falls back to the committed manifest src/data/posts.json on 'error || !data || !data.length', and that manifest holds 108 real production entries (measured 2026-09-19). A broken key, a missing table or an RLS change therefore produces a page full of stories, every one of which names a row that does not exist in the database being read. Nothing errors and nothing on the page says which source it came from. scripts/up.sh fails its bring-up closed on this rather than trusting the page.
fixedlowsrc/lib/wire.config.tssrc/lib/wire.config.ts describes a shell this repo no longer has. Its header names src/components/WireMark.tsx and src/app/wire.css as two of the three files a wire supplies, and its newsletter block records that 'popwire/src/components/Shell.tsx renders this block'. None of those three files exists in the tree; the shells were purged on 2026-09-14 and every app is its own Next.js tree now. The file is still load-bearing (six modules import WIRE), so the stale half reads as current to anyone following it.