Live Data Pipeline

Part of the Paulopus case study. Paulopus is a World Cup 2026 prediction site where an octopus persona locks a call before each match and grades itself afterward. None of that works without a trustworthy answer to a deceptively simple question: what was the actual score? This focus is about how I sourced that answer — the open data feeding fixtures, results, and scoring — and about the fallback I documented, planned for, and never actually wired.

One authoritative source, not a scoreboard scrape

Decision. For a project whose entire premise is grading predictions against reality, the results feed is the load-bearing dependency. I did not want to reverse-engineer a live scoreboard or juggle rate-limited commercial APIs. I wanted one canonical, versioned, inspectable source of FIFA World Cup 2026 fixtures and results.

Approach. I settled on openfootball/worldcup.json, a community-maintained repository that publishes the whole tournament as a single structured JSON file. It became the sole authority for fixtures, teams, and every scoreline — regulation, extra time, and penalties.

Artifact.

{
  "name": "openfootball/worldcup.json",
  "purpose": "authoritative FIFA World Cup 2026 fixtures/results, one-shot-seeded and later re-fetched for verdict grounding (regulation, extra-time, and penalty scorelines)",
  "endpoint": "WORLDCUP_2026_URL in lib/openfootball.ts = https://raw.githubusercontent.com/openfootball/worldcup.json/master/2026/worldcup.json"
}

Result. A single raw.githubusercontent.com URL in lib/openfootball.ts backed the whole pipeline. Because it is a plain file over HTTPS, it seeds, re-fetches, and diffs the same way every time — no auth handshake, no scraper to maintain.

A parser that knows a match can go past ninety

Decision. Football does not always end at the whistle. Knockout matches run to extra time and, when still level, to penalties. The parser had to preserve all three layers rather than flattening everything to a 90-minute number.

Approach. Early in the build I wrote the openfootball parser alongside the deterministic karma scorer, but that initial pass extracted only the 90-minute regulation score. The result model's extraTime, penalties, and decidedBy fields — and a scorer that derives the played score and real winner from them — arrived later, with the regrounding that fixed how knockout matches got graded.

Artifact.

Result. lib/scorer.ts grades against the score that was actually played and the team that actually advanced. The verdict ladder it assigns — Perfect, Inspired, Missed, RedCard — is computed deterministically, and a full verdict test matrix pins that behavior in vitest.

Seed, sync, backfill — three flows, one Mongo

Decision. During the live-tournament era the app read from MongoDB Atlas, not from a static file. I needed openfootball data to get into Mongo, stay current as matches finished, and let predictions attach to fixtures before kickoff.

Approach. I built three scripts against a cached Mongo client: an idempotent seed to pull fixtures and teams, a score-sync that pulled results and graded predictions with identity cross-checks, and a pre-kickoff prediction backfill. Sync ran on a schedule.

Artifact.

Result. Every 30 minutes, sync reconciled openfootball results into Mongo and re-graded finished matches — which meant a scoring rule change re-graded the whole field automatically, as when I later lowered the Perfect reward from +2 to +1.5 and sync recomputed existing entries.

The fallback I planned and never wired

Decision. One upstream source is a single point of failure. If openfootball lagged on a result, I wanted a second score feed to fall back to. So I designed for one.

Approach. I documented football-data.org as an optional fallback for sync-scores.ts, reserved a FOOTBALL_DATA_TOKEN secret, and left a slot for it in the sync workflow. Then I never implemented it. openfootball stayed reliable enough that the fallback never earned its keep.

Artifact.

{
  "name": "football-data.org (planned, never wired)",
  "purpose": "documented as an optional fallback score source (FOOTBALL_DATA_TOKEN) for sync-scores.ts; never implemented — openfootball remained the sole source",
  "endpoint": "FOOTBALL_DATA_TOKEN (unused workflow secret in .github/workflows/sync.yml)"
}

Result. An honest piece of dead scaffolding. The token still sits unused in sync.yml, a documented seam I chose not to build out — worth recording precisely because the temptation is to pretend the plan shipped.

Grounding verdicts on what really happened

Decision. Early scoring graded on the 90-minute regulation score the day-one parser captured. For nine knockout matches decided beyond regulation, that produced verdicts that contradicted who actually lifted through.

Approach. I reworked scoring to grade on the played score (extraTime ?? regulation, penalties excluded as goals) and the real winner (who advanced, via shootout when level), then re-fetched openfootball to reground the affected matches and rewrote their appreciation text to match.

Artifact.

Result. The nine wrong knockout verdicts were corrected and the running karma total moved from 15.5 to 25.0 — a jump that came entirely from grading against the outcome rather than the ninetieth minute.

Freezing the feed into a static snapshot

Decision. Once the tournament ended, a live Mongo-backed data layer and a 30-minute cron were pure liability — runtime secrets and moving parts guarding data that would never change again.

Approach. I snapshotted the final data into data/matches.json (104 matches) and data/teams.json (48 teams), rewrote lib/db.ts to read them in-memory behind unchanged async signatures, archived the Mongo-writing scripts, and disabled the sync and ingest workflows with if: false.

Artifact.

Result. The live app now carries no database and no runtime secrets; it reads a static snapshot. Only seed.yml survives as a runnable connection back to openfootball — a workflow_dispatch button to re-pull fixtures on demand, the last live wire in an otherwise frozen pipeline.