Brick Work 4.0: user journey on the revised model (revision 2)

Written 2026-10-23, as a walk through data-model-design.md and schema-4.0.sql in this folder. Same scenario as the initial journey: a fresh install, onboarding, an owned copy of 6081-1, three loose parts, two loose figures. Nothing personal: every number is a catalogue number.

Every row below was written, not just described. test_journey.py replays this journey against the DDL: a fresh collection file with the real 22 Oct catalogue attached read-only. It ends with a clean foreign_key_check and integrity_check (measured; log in the scratchpad revise-4.0/journey.log).

The ids are real. They are the running numbers a first build of the 22 Oct dump gives (colours 1 and 2 fixed, then colours by Rebrickable id; items as sets, then parts, then figures, each sorted by key; revise-4.0/build_dump.py) (measured). Another install, or a build from another dump, numbers differently (design § 3.1). That is why nothing below relies on a particular number.

BrickLink colour ids used below (White 1, Black 11, Red 5, Yellow 3, Dark Gray 10, Light Gray 9, Dark Brown 120) are what the shipped colors.csv will carry. I wrote them from memory: confirm them against the mapping build.

Budgets as in the code today: BrickLink 4,400 calls a day (sources/bricklink.py:49), Rebrickable 1,000 at 1 s spacing, Brickset 100 getSets, Brickognize 100. Image hosts img.bricklink.com and cdn.rebrickable.com need no key and have no budget.

0. Fresh container

docker compose up with an empty data/. brickwork setup creates only the collection file:

The user sees Get started (#83). Every Add path answers catalogue_not_loaded. This is the same offline and without keys.

1. Onboarding

1.1 Step 1: the catalogue (no account, no key)

The user sees Download today's catalogue, with the line "Rebrickable asks for at most one download a day" (Q14: tell, don't block). If files from today are already in data/catalogue-src/rebrickable/, the step offers Build from today's files first. Offline, the step offers Use files I downloaded and the link to rebrickable.com/downloads/. The user presses the button and sees one progress bar (download, build, mappings, index, activate), then "Catalogue of 22 Oct: 28,547 sets, 64,906 parts, 17,293 minifigures, 275 colours".

Job: job(source=rebrickable_dump, kind=catalogue_import) with steps download → build → mappings → index → activate. The download is 12 plain HTTPS GETs (17 MB). It uses no budget, and no source_budget row exists for it (Q14).

The catalogue file, built as catalogue.db.new and renamed into place. The build is seeded from the collection, which holds only colours 1 and 2, so everything else is numbered fresh.

The mappings step (build 2, kind=mapping):

Collection file, activate step:

• catalogue_build rows 1 and 2;
• meta.catalogue_id bound, and the high-water marks copied;
• the other 273 colours projected with their color_link rows;
• event catalogue/activated.

Cost: 11.6 s for the tables plus 0.3 s FTS on a Mac for this dump, giving a 129.7 MB file (measured). Links and the Pi timing are still to be measured.

From here, steps 2 to 4 below work without a key and without the network.

1.2 Step 2: keys (optional)

Without BW_SECRET_KEY, key entry is disabled as today; keys from the environment still work.

1.3 Tools & Admin: the BrickLink item lists (optional)

The user drops the eight XML files into data/catalogue-src/bricklink/ and presses Load. A seeded build writes a new catalogue file with:

• bricklink_item (all eight types);
• 6081-1's link upgraded inferred/set_number → matched/bricklink_list;
• 2339 gets (37722, P "2339", to_check, same_number) if P 2339 exists in the list (decision 8: never matched);
• theme-guessed kinds corrected where BrickLink's type disagrees (Q10).

Then reconcile writes BrickLink names as item_property(source=bricklink) and mirrors the upgrades for projected items. Nothing is called and nothing is budgeted. It works offline.

1.4 Step 3: bring a collection in

Not walked. Rebrickable or Brickset sync, a BrickTracker import and BrickLink XML all land through the same projection and materialisation. A synced item whose kind is not set becomes stock of that kind (design § 3.5).

2. The first set: an owned copy of 6081-1

The user opens Manage → Add, types 6081, and sees 6081-1 King's Mountain Fortress · 1990 · Lion Knights · 438 parts from catalogue.item_fts. They pick it, choose Owned, condition used, and see the review table.

The Part list picker (Q23). 6081-1 has one Rebrickable version. Without a BrickLink key there is no choice, so no picker is shown. With a key, the picker shows Rebrickable and BrickLink's list, preselected from set_preference (none), then from prefer_bricklink_inventory (false), so Rebrickable (§ 2.1 shows the other case). Every part row has a picture source, so the review shows no picture warning.

The user presses Confirm.

Collection file, one transaction (the Add receipt):

The copy can be checked the moment the transaction commits. Check Set Parts reads copy_part, copy_figure and the projected item (category and size for largest-first), without the catalogue.

Jobs, in order:

2.1 Variant: "prefer BrickLink's list" on, key present (Q13, Q23, decision F)

The user turns on Prefer BrickLink's list when a key exists in Settings (setting prefer_bricklink_inventory=true) and adds a second copy of 6081-1. The picker preselects BrickLink's list. No BrickLink list has been fetched for 6081-1 yet.

One transaction:

• job(source=bricklink, kind=subsets, item 15011, queued);
• copy(inventory_source=rebrickable, inventory_version=1, inventory_digest=9db79baf…, inventory_pending=bricklink, pending_job_id=<that job>), with Rebrickable's figure and part lines at once.

The copy page says "BrickLink list pending" and the copy can be checked offline.

When the job succeeds (1 call of 4,400), one transaction:

• source_record(bricklink, subsets, S:6081-1, <raw reply>);
• fetched_inventory(item 15011, source=bricklink, seq=1, is_latest=1, job, source_record);
• fetched_inventory_part lines, where BrickLink's quantity − extra_quantity is required and extra_quantity is spare (design § 4.2);
• fetched_inventory_figure lines.

Then the switch runs (design § 4.4). Lines with the same (figure, item, colour, spare) are updated in place with their check state carried and capped; new lines are inserted; vanished lines get retired_at. Then copy.inventory_source=bricklink, fetched_inventory_id=…, inventory_digest=<BrickLink list digest>, inventory_pending=NULL, pending_job_id=NULL. The receipt lists any marks that could not carry over.

In the replay, one Rebrickable line has no BrickLink counterpart and is retired, not deleted, and the copy gains one spare line. Figure parts stay on each figure's own Rebrickable list (Q15).

• Out of budget or offline: the job waits, the copy stays on Rebrickable's list with the note, and the user can cancel the pending switch.
• Without a BrickLink key: the picker never offers BrickLink and the switch has no effect.

A third copy added later takes the fetched BrickLink list at once, with no call.

3. Loose parts

The user opens Add → Parts and pastes 3023 x10, 2339 x2, 970c03 x1. They choose colours (White, Black, Red), Loose, used, and storage Box A (created inline). Every row has a picture source:

• 3023 White and 2339 Black have picture_url;
• 970c03 Red has one too, since fig-004576 uses it.

So no warning shows. A part in a colour no inventory uses, with no BrickLink link and no Rebrickable key, would show "no picture known for this colour" (G5, Q17).

Collection file (one Add receipt):

A second 3023 White used Box A line is refused by stock_merge_key, whose generated keys see the NULL lot as equal. stock.add_loose merges it into the first row instead, as today (stock.py:87-115).

Jobs:

Offline, all three rows exist and work in Part Picker, Matcher and Check Set Parts. 2339 Black is what 6081-1 needs twice, so the Matcher shows "2 in loose stock" on the copy.

4. Two loose minifigures

The user opens Add → Minifigures, types fig-004576 and fig-018222, chooses Loose, used, Box A. Both rows open with their four parts. The waiter shows "No BrickLink id".

Collection file (one Add receipt):

Jobs:

Offline, both figures exist with their parts. "Check figure parts" works on stock_part.

5. What ran, in one table

Every collection row carries its source:

• catalogue_build_id, the born key and the digest on items;
• (set, version) or fetched_inventory_id and the digest on copies, figure lines and loose figures;
• the line key on lines;
• origin and catalogue_build_id on links;
• provider_key on prices.

6. Gaps G1–G9: how each is closed