rmhladder Plan 3/5: Pipeline + Worker Implementation Plan¶
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: The every-4-hours pipeline: adapter hardening from the Plan-2 review backlog, then discover → dedupe → classify → score → verify → persist, recheck with 3-strike expiry and a mass-expiry circuit breaker, scrape-run logging, review-task creation, a ladder:run CLI, and the ladder-worker cron process.
Architecture: Pure decision logic lives in lib/rmhladder/pipeline/* (unit-tested, no DB); thin prisma glue composes it; the worker (server/ladder-worker/) and scripts/run-ladder-pipeline.ts both call one runPipeline(). Board fetches are memoized per run via a caching fetchImpl wrapper so verify never refetches what discover just fetched. End-to-end correctness is proven by a live smoke run against the seeded dev DB in the final task.
Tech Stack: TypeScript strict, Prisma 7, vitest, node-cron (the one permitted new dependency — named by the spec), existing adapters/classifiers/scoring/verification from Plans 1–2.
Global Constraints¶
No network in unit tests (injectable
fetchImpleverywhere). Live network + dev-DB writes are allowed ONLY in Task 8’s smoke run and are explicitly authorized there.Only new dependency allowed:
node-cron(+@types/node-crondev). Nothing else.Do not modify Plan 1 exports except the one addition named in Task 4 (a
VerificationStatusunion type added tolib/rmhladder/verification.ts— additive only).Pipeline code in
lib/rmhladder/pipeline/; worker inserver/ladder-worker/; scripts followscripts/seed-ladder.tsconventions (prisma import from@/lib/prisma.server,import 'dotenv/config',main().catch(...).finally(...), file-level/* eslint-disable no-console */for CLI output).Commit per green test cycle; suite:
pnpm exec vitest run lib/rmhladder; commit-scope discipline:git status --porcelainbefore every commit, add only the task’s files.Politeness unchanged:
politeFetcheverywhere, 300ms between live requests in scripts, robots rules from Plan 2.Expiry policy (spec): a job expires only after 3 consecutive failed checks OR explicit closed language. Fetch failures are never expiry evidence.
Task 1: Memoized fetch + real HTTP status through the adapters¶
Files:
Create:
lib/rmhladder/pipeline/memo-fetch.ts,memo-fetch.test.tsModify:
lib/rmhladder/adapters/greenhouse.ts,lever.ts,ashby.ts,smartrecruiters.ts(+ their tests)
Interfaces:
Produces:
memoFetch(fetchImpl?: typeof fetch): typeof fetch— returns a fetch that caches responses by URL string for GET requests within its lifetime (per source-run). CachedResponsebodies must be replayable: cache{ status, body }and manufacture a freshResponseper call.Modifies (all four adapters, same mechanical pattern): the internal
fetchBoardreturns{ jobs: T[] | null; status: number }(SR:{ postings, status, totalFound }— see Task 2);verifyJobevidence uses the REAL status:httpStatus: status, and adds API-adapter blocked semantics:blocked: status === 403 || status === 429.fetched: jobs !== null. detectExpired/discoverJobs behavior unchanged (null jobs → [] / false).
Steps (TDD):
memo-fetch tests first: same URL fetched twice → underlying impl called once, both callers get the body; different URLs → two calls; non-ok responses are cached too. RED → implement → GREEN → commit
feat(rmhladder): memoized per-run fetch.Adapter status propagation, one adapter at a time, tests first per adapter: extend each adapter’s verifyJob test to assert
httpStatusmatches the stubbed status (e.g. stub 500 → evidencefetched:false, httpStatus:500; stub 403 →blocked:true). Existing synthetic-404 assertions get updated to the real-status behavior — this is the sanctioned replacement of the Plan-2 synthetic sentinel. GREEN per adapter → single commitfix(rmhladder): adapters propagate real HTTP status; 403/429 marks blocked.
Task 2: Greenhouse entity decoding + SmartRecruiters pagination & expiry guard¶
Files:
Modify:
lib/rmhladder/adapters/greenhouse.ts+ test +__fixtures__/greenhouse-board.jsonModify:
lib/rmhladder/adapters/smartrecruiters.ts+ test +__fixtures__/smartrecruiters-postings.json(add a page-2 fixture file)
Requirements:
Greenhouse
contentarrives HTML-entity-escaped (recorded API shape). AdddecodeEntities(s)(module-local, the five standard entities< > & " 'plus numeric&#\d+;) applied innormalize()todescriptionHtml. Update the fixture to carry escaped content ("<p>Join our payments team as a PM intern.</p>") and the existing full-object test to expect the DECODED string — this proves the decode path.SR pagination: postings URL gains
&offset=${offset};fetchBoardloops pages of 100 untilcontent.lengthreachestotalFoundor a hard cap of 500 postings (politeness), aggregating postings; returns{ postings, status, totalFound }.SR expiry guard (review finding: empty board is ambiguous on SR because the endpoint 200s for any slug):
detectExpiredreturnsfalsewhenevertotalFound === 0— an SR board with zero postings is treated as insufficient evidence, never as expiry.Tests: pagination test with two stubbed pages (totalFound 150 → two fetch calls, 150 postings aggregated — use a counting stub keyed on the offset param); totalFound-0 guard test; entity-decode via updated fixture test.
Steps: RED (new/updated tests) → implement → GREEN → commit fix(rmhladder): greenhouse entity decoding, SR pagination + empty-board expiry guard.
Task 3: Robots fail-closed wildcards, scheme allowlist, probe flag validation¶
Files:
Modify:
lib/rmhladder/adapters/robots.ts+ test;lib/rmhladder/adapters/generic.ts+ test;scripts/probe-ladder-sources.ts
Requirements:
robots: rules whose prefix contains
*or$currently never match (fail-open). ChangeisPathAllowedto match such rules on the literal prefix up to the first special character (e.g.Disallow: /jobs/*/applymatches any path starting/jobs/). Tests:Disallow: /careers/*blocks/careers/x;Allowwith wildcard still wins by (literal) prefix length; plain rules unaffected.generic:
verifyGenericUrlrejects non-http(s) URLs up front —data:,file:,javascript:→ return base evidence (fetched:false, blocked:false) without any fetch (test: fetchImpl never called — use a throwing stub).probe script:
--limit abccurrently yieldsslice(0, NaN)→ silently does nothing. Validate: non-numeric or < 1 → print error +process.exit(1)(mirror the--platformvalidation style).RED → implement → GREEN (
pnpm exec vitest run lib/rmhladder/adapters) → commitfix(rmhladder): robots wildcard fail-closed, URL scheme allowlist, probe flag validation.
Task 4: Persistence decision logic + VerificationStatus union¶
Files:
Create:
lib/rmhladder/pipeline/ingest.ts,ingest.test.tsModify:
lib/rmhladder/verification.ts(additive only)
Interfaces:
verification.ts adds (and
computeVerification/passesAlertGatesignatures switch theirstatus: stringfields to it — call-compatible, additive):export type VerificationStatus = 'verified_active' | 'verified_probable' | 'unverified' | 'expired' | 'broken_link' | 'duplicate' | 'non_us_role' | 'blocked_or_inaccessible' | 'needs_manual_review';
ingest.ts produces PURE functions (no prisma — the glue in Task 5 applies their output):
interface JobAssessment { dedupeHash: string; fields: { /* every LadderJob column derivable from a NormalizedJob */ }; verificationInput: VerificationEvidence; // evidence enriched with US confirmation reviewReasons: Array<'ambiguous_us_location' | 'ambiguous_early_career' | 'low_confidence'>; } function assessJob(args: { normalized: NormalizedJob; companyName: string; companyId: string; companyPriority: number; platform: string; evidence: VerificationEvidence }): JobAssessment function summarizeDescription(html: string | null): { summary: string | null; text: string } // node-html-parser textContent, summary = first 500 chars, collapsed whitespace
Rules encoded in
assessJob(each gets a test):location:
classifyUSLocation({ locationRaw, country });countryfield from result or normalized input;isUS === null→ reviewReasons includesambiguous_us_location; non-US jobs keep flowing (they persist withnon_us_roleverification status — the alert gate already excludes them).early-career:
classifyEarlyCareer(title, text); classificationunclear→ambiguous_early_career;schoolYearTargetandgraduationYearTargetare nulled when classification isno(deferred Plan-1 review item).scoring:
computeBaseScoreon the assembled ScorableJob (companyIsTarget = priority ≤ 2); store base + urgencyFlag.evidence enrichment:
usConfirmedset from the location result (adapters already set it, but assessJob is authoritative — it overwrites with its own classification); aftercomputeVerification, confidence < 60 →low_confidencereview reason; non-US location result → verification status is overridden tonon_us_role(confidence from location classifier).dedupe:
dedupeHash(companyName, title, locationBucket(...))— bucket from classified city/state/remoteStatus (onlyremote_uswhenisUS === true).
RED (test each rule with hand-built NormalizedJobs: US intern verified_active; London analyst → non_us_role; ‘Main Campus’ → ambiguous review; senior title → grad-year nulled; low-evidence HTML page → low_confidence) → implement → GREEN → commit
feat(rmhladder): job assessment pipeline logic + VerificationStatus union.
Task 5: processSource glue¶
Files:
Create:
lib/rmhladder/pipeline/process-source.ts,process-source.test.ts
Interfaces:
interface SourceRunStats { discovered: number; created: number; updated: number; verified: number; reviewTasks: number; errored: boolean }
async function processSource(deps: { prisma: PrismaLike; fetchImpl?: typeof fetch }, source: { id; platform; slug; company: { id; name; priorityLevel } }): Promise<SourceRunStats>
Flow:
getAdapter(platform)(null → errored stat, no throw) → onememoFetchper call →discoverJobs→ per job:verifyJob(memoized — no extra network) →assessJob→ upsertLadderJobby dedupeHash (create full row | update: lastCheckedAt, failedCheckCount 0, refreshed scores/status, mergealternateUrlswhen originalPostingUrl differs) → insertLadderVerificationrow → createLadderReviewTaskper reviewReason (skip when an OPEN task with same jobId+reason exists) →alert_eligibleis NOT computed here (Plan 5).PrismaLikeis a structural type of just the model methods used (ladderJob.upsert,ladderVerification.create,ladderReviewTask.findFirst/create,ladderSource.update) so tests inject an in-memory fake WITHOUT mocking the prisma package — the fake is ~40 lines, asserted on real behavior (rows accumulated, upsert-by-hash semantics).Per-source try/catch: any throw →
errored: true+ rethrow-safe error object returned in stats (caller logsLadderSourceError); success updateslastSuccessAt.RED (fake-prisma tests: fixture board → 2 discovered/2 created/verification rows; re-run same board → 0 created 2 updated, no duplicate review tasks; adapter-null platform; throwing fetch → errored, no partial corruption) → implement → GREEN → commit
feat(rmhladder): per-source pipeline processing.
Task 6: Recheck + 3-strike expiry + circuit breaker¶
Files:
Create:
lib/rmhladder/pipeline/recheck.ts,recheck.test.ts
Interfaces:
// pure decision core — exhaustively tested:
interface RecheckDecision { action: 'skip' | 'reset' | 'strike' | 'expire'; }
function decideRecheck(args: { presentOnBoard: boolean; fetchSucceeded: boolean; failedCheckCount: number }): RecheckDecision
// exact contract:
// fetchSucceeded false → 'skip' (fetch failure is no evidence: no counter change)
// presentOnBoard true → 'reset' (counter → 0, status stays active)
// absent && failedCheckCount + 1 >= 3 → 'expire'; otherwise 'strike' (counter + 1)
function circuitBreaker(args: { activeCount: number; wouldStrikeOrExpire: number }): boolean // true = trip: activeCount >= 4 && ratio > 0.5
async function recheckSource(deps, source, activeJobs): Promise<{ struck: number; expired: number; reset: number; skipped: number; tripped: boolean }>
recheckSource: one memoized board fetch; compute decisions for all active jobs FIRST; ifcircuitBreakertrips → NO strikes/expiries applied; instead: sourcestatus: 'error'+ oneLadderReviewTaskwith reasonmass_expiry_suspected. That reason doesn’t exist yet:LadderReviewReasongains the additive enum valuemass_expiry_suspected(+ validate/generate/db push) — this is the plan’s one sanctioned schema change.Expiry application:
expire→ job statusexpired, verification row{ status: 'expired', confidence: 90, evidence: '3 consecutive checks found the posting absent from the <platform> board.' }.SR note: adapters’
detectExpiredalready guards totalFound===0 (Task 2), so recheck trustsdetectExpired.RED — decideRecheck table-test all arms (including boundary failedCheckCount 2 → expire on 3rd), circuitBreaker boundaries (3 active never trips; 4 active 3 absent trips; exactly 50% doesn’t trip), recheckSource fake-prisma tests (trip applies nothing; normal path strikes/expires/resets) → schema enum addition +
pnpm exec prisma validate && pnpm db:generate && pnpm db:push→ implement → GREEN → commitfeat(rmhladder): recheck with 3-strike expiry and mass-expiry circuit breaker.
Task 7: runPipeline orchestrator + manual-source aliveness + CLI¶
Files:
Create:
lib/rmhladder/pipeline/run.ts,run.test.ts,scripts/run-ladder-pipeline.tsModify:
package.json("ladder:run": "pnpm exec tsx scripts/run-ladder-pipeline.ts")
Interfaces:
async function runPipeline(deps: { prisma; fetchImpl?: typeof fetch }, opts: { trigger: 'cron' | 'manual'; limitSources?: number; platforms?: string[] }): Promise<{ runId: string; discovered: number; created: number; verified: number; expired: number; errors: number; reviewTasks: number; durationMs: number }>
Lifecycle: create
LadderScrapeRun(trigger) → loadactivesources (optionally filtered/limited) → API sources throughprocessSourcesequentially with 300ms spacing between sources →recheckSourcefor sources with active jobs → manual sources are program PAGES, not postings, so their check is aliveness only:checkRobotsthenpoliteFetchexpectingok; fetch failure → sourcestatus: 'error'+ review taskbroken_link; robots-disallowed → sourcestatus: 'blocked'+ review taskblocked→ per-source errors recorded toLadderSourceError(never abort the run) → finalize run row (counts, finishedAt, stats JSON of per-source lines).run.test.tswith the fake prisma + stub fetch: two API sources (one healthy fixture board, one 500) + one manual (200) → run row counts correct, error row for the 500, healthy source stats intact.CLI script: flags
--trigger manual(default),--limit N(validated),--platform X; prints the summary line; conventions per Global Constraints.RED → implement → GREEN → commit
feat(rmhladder): pipeline orchestrator + ladder:run CLI.
Task 8: ladder-worker + wiring + live smoke + wrap¶
Files:
Create:
server/ladder-worker/index.tsModify:
package.json(addnode-crondep +@types/node-cron; scripts:ladder-worker:devfollowing the existing*-worker:devpattern; append the worker to thedevconcurrently list, thebuildesbuild entry list, and thestartlist — mirrordoctrine-workerexactly in all three)
Requirements:
Worker:
import 'dotenv/config';cron.schedule(process.env.LADDER_CRON_SCHEDULE ?? '0 */4 * * *', ...)callingrunPipeline({ prisma }, { trigger: 'cron' }); overlap guard (skip tick if previous run still in flight — a simple boolean); startup log line; graceful SIGTERM (finish current run, stop cron, disconnect prisma). Structural mirror of the smallest existing worker inserver/— read one before writing.Install:
pnpm add node-cron && pnpm add -D @types/node-cron(lockfile change IS expected in this commit — the one place it’s legitimate; commit package.json + pnpm-lock.yaml together).Live smoke (authorized network + dev-DB writes):
pnpm ladder:run --limit 3 --platform greenhouseagainst the real dev DB (6 active greenhouse sources exist). Capture full output; then query and report:select count(*) from ladder_job,select status, count(*) from ladder_verification group by 1, scrape-run row contents. Discovered jobs from real boards (Adyen/Affirm/Airbnb…) should appear with classifications and scores. Zero unhandled errors required; empty boards are fine.Full wrap:
pnpm exec vitest run lib/rmhladderall green;pnpm exec eslint lib/rmhladder scripts server/ladder-worker0/0;pnpm exec prisma validate.Worker + wiring → smoke → wrap checks → commit
feat(rmhladder): ladder-worker cron process + plan-3 wrap.
Deferred to Plans 4–5 (recorded, not lost)¶
Dashboard “Run now” button wiring to runPipeline (Plan 4); alert dispatch reading new verified jobs via passesAlertGate (Plan 5); prisma migrate before any deploy; indexes for LadderSourceError(runId), LadderReviewTask(status), LadderJob(discoveredAt, relevanceScoreBase) once dashboard query patterns exist (Plan 4); Playwright JS-rendering fallback for generic verification (only if manual-source aliveness proves insufficient in practice).