Imports

Everything the app knows about Magic arrives through one of the jobs below. Each reads a published bulk file or API, writes to specific tables, and is safe to re-run.

Two vendors do most of the work, and they cover different things. Scryfall is the source for card text, rulings and printings. MTGJSON is the source for sets, precons, product identifiers and price history. Where both cover something, the data model records which one owns the column.

The whole shape

Four outside sources feed the reference, and two tables are built from what those sources already brought in:

Two things to read off it. rules_examples has no vendor of its own: it is assembled from the examples already inside the Comprehensive Rules and from official rulings that describe an interaction, both of which arrive with other jobs. And the tagging and embedding passes sit downstream of everything, which is why a card import leaves them stale. Those are covered in tagging and embeddings.

Nothing on this page runs on a schedule. Every job here is started by hand. The two jobs that are meant to run on a timer are prices, and they are covered in scheduled jobs.

Cards and printings

seed:printings uses Scryfall's English-only default_cards file rather than the full multilingual one, so the printing catalogue is English printings.

seed:types imports the list of every supertype, card type and subtype. It does not touch cards. The parsed type columns and card_subtypes are derived by a trigger, in the same statement that writes a card's type line, which is what makes t:wolf match a parsed subtype instead of a substring the moment a card arrives. A subtype no catalog names yet is added to the vocabulary as unknown and gets its real category the next time seed:types runs.

Both bulk files are larger than a Node string can hold, so they are stream parsed and written in batches rather than loaded whole.

Sets, precons and products

pnpm ingest:mtgjson <subcommand> covers the MTGJSON side. Each subcommand is a separate job sharing one download client and one run log:

Every job records itself in data_job_runs, which is the place to look for what last ran and what it touched. A row whose finished_at is still empty belongs to a job that started and never recorded an ending.

Sealed products are separate again: sealed_products is seeded from MTGJSON by pnpm seed:sealed. They are catalogued but not priced. See where a price comes from.

Rules and reference text

seed:rules also tags each rule with the concepts it mentions, so a concept can link to the rules that define it.

seed:rules-examples truncates and rebuilds rather than merging, so a run replaces the table.

seed:set-guides is the one job here that writes model-generated text. It resolves a set's wiki page, parses what it can, and generates a short summary grounded in what it found. Where a page cannot be found it still writes a summary, from the model alone. Set blurbs are the only prose in the reference data that is not quoted from a source.

Clearing the cached pages

Card, set and precon pages are cached indefinitely rather than on a timer, so a finished import does not reach visitors on its own. The jobs that change what those pages show call the app's revalidation endpoint when they complete, which drops the cached copies:

seed:cards, seed:printings, seed:sealed, seed:set-guides, and all three tagging passes.

The call is deliberately non-fatal. A run that wrote its data correctly does not report failure because the cache could not be cleared, so a successful import and a cleared cache are two separate things. If a page looks like it is showing the previous import, that is the gap.

Prices are the exception, and by design: the live price on a card page is fetched in the browser rather than baked into the cached HTML, so a price refresh reaches you without any page being rebuilt.

Re-running

These jobs are written to be re-run. Ordering matters in one place: card text has to exist before the jobs that read it, so a full rebuild goes cards, then printings, then types, then tagging and embeddings.

Bulk writes to cards drop the write-heavy indexes, write, then rebuild them, because per-row maintenance of the vector and trigram indexes would otherwise exceed the statement timeout. A vacuum pass follows, since a bulk rewrite leaves dead rows that slow every later scan.