|
2 | 2 |
|
3 | 3 | Development history for Vitrine (a standalone project that vendors LichtFeld Studio as a pinned tool; formerly a fork — see ADR-021). |
4 | 4 |
|
| 5 | +## 2026-07-09 — Object pipeline convergence implemented (PRD v4 / ADR-025) + SAM3 root cause FIXED |
| 6 | + |
| 7 | +### R1 SOLVED: "SAM3 returns boxes" was an HWC/CHW bug in OUR wrapper — one-line fix |
| 8 | + |
| 9 | +The audit's blocker defect F1 is closed, and the root cause was ours, not SAM3's: |
| 10 | +`Sam3Processor.set_image`'s numpy branch reads `height, width = image.shape[-2:]` |
| 11 | +(correct for CHW tensors); our `sam3_segmentor` passed HWC frames, so it read |
| 12 | +`height=W, width=3`. Detection still worked (model input conversion handles HWC) |
| 13 | +but every output mask was interpolated to a **W×3 grid** and boxes were scaled by |
| 14 | +(3, W) — which downstream resizing smeared into the notorious "coarse boxes". |
| 15 | +Fix: pass PIL images (`_to_pil` in `sam3_segmentor.py`). Verified on rawcapdev: |
| 16 | +pixel-accurate silhouettes for vessel (fill 0.86, score 0.94), ketchup bottle |
| 17 | +(0.79/0.57) and wooden block (0.75/0.92) — overlay at |
| 18 | +`output/rawcapdev/sam3_fixed_overlay.jpg`. |
| 19 | + |
| 20 | +### First-ever validated automated per-object isolation on real data |
| 21 | + |
| 22 | +With silhouettes fixed + new binary-COLMAP support (`parse_cameras_bin`/ |
| 23 | +`parse_images_bin` + format-agnostic loaders — the direct-COLMAP path emits |
| 24 | +bin-only models, which would otherwise fail loudly), the full Phase-1/2 arc ran |
| 25 | +live on rawcapdev: segment → **object_crops** (new stage) → extract_objects |
| 26 | +MV-projected **1.09M/4M gaussians (vessel), 246k (bottle), 250k (block)** across |
| 27 | +6 views. Zero fallbacks, zero failures. Crops are generator-ready RGBA mattes at |
| 28 | +native res (vessel 3175²) with full provenance (frame, bbox, COLMAP pose, |
| 29 | +matting method, selection scores) in `object_crops/crops.json`. |
| 30 | + |
| 31 | +### ADR-025 implementation (all PRD v4 phases except live 3D generation) |
| 32 | + |
| 33 | +- **R2** `extract_objects`: full-scene-copy + world-XY fallbacks DELETED; per-object |
| 34 | + failures reported with cause (`object_failures` artifact). `full_scene` label |
| 35 | + (the environment) still copies by design. |
| 36 | +- **R3** new `object_crops` stage (`src/pipeline/object_crops.py` + config): |
| 37 | + best-frame selection = silhouette area × log-sharpness × centrality × edge |
| 38 | + clearance; SAM-matte alpha; rembg fallback for box-like masks; ≥1024² square pad. |
| 39 | +- **R4** multiview conditioning RETIRED: `trellis2_client.py` rewritten single-image |
| 40 | + (ComfyUI executor now, native-service contract ready); `view_completer.py`, |
| 41 | + `trellis2_multiview_pbr.json`, `flux2_turnaround.json` + all 3 stock-node Hunyuan |
| 42 | + workflow JSONs deleted; `multiview_renderer` kept preview-only and fixed to |
| 43 | + straight alpha (F4 un-premultiply). |
| 44 | +- **R5** `scripts/trellis2_native_service.py` scaffold (HTTP contract final; model |
| 45 | + calls marked VERIFY-ON-ENV-BUILD; client flips via `trellis2.native_url`). |
| 46 | +- **R6** generator GLB bytes persisted byte-identical (sha256 recorded, lineage |
| 47 | + sidecar per asset); `texture_bake` skips generator meshes (F7 closed). |
| 48 | +- **R6a** Hunyuan fallback = `reconstruct_from_image()` submitting the PROVEN |
| 49 | + Hy3D21 graph (hy3d_turnaround.py lineage) in code. |
| 50 | +- **R9** eval harness `eval/objects/run_eval.py` (+ blender turntable): mesh |
| 51 | + stats/regression gates; `--stats-only` validated against the 2026-07-02 brass |
| 52 | + vessel GLB (274,588 verts / 483,671 faces / PBRMaterial — exact match). |
| 53 | +- **Tests**: 30 new unit tests (crops, single-image client, object-arc |
| 54 | + regressions); pipeline subset **144/144 green** in-container. |
| 55 | + |
| 56 | +### LIVE RUNTIME VERIFICATION: crop → single-image TRELLIS.2 → PBR GLB |
| 57 | + |
| 58 | +The full ADR-025 arc ran end-to-end on GPU 1: the automated vessel crop → |
| 59 | +`Trellis2ImageToShape` (1536_cascade, seed 42) → **54.8 MB PBR GLB, 286,009 |
| 60 | +verts / 491,067 faces in 479 s**, persisted byte-identical (sha256 |
| 61 | +`496d9a84…`) with lineage sidecar — same output class as the manual |
| 62 | +2026-07-02 vessel (274,588 / 483,671). Workbench turntable confirms solid |
| 63 | +geometry + patina albedo (`output/object_e2e/`). Smoke script: |
| 64 | +`scripts/run_hull_e2e.py` (rewritten crop-based; the splat-render version is |
| 65 | +gone). |
| 66 | + |
| 67 | +### Infra root causes fixed along the way |
| 68 | + |
| 69 | +- **ComfyUI has been fighting DiffusionGemma for GPU 0 all along**: |
| 70 | + `comfyui_entrypoint.sh` passed `--cuda-device 0`, and ComfyUI implements |
| 71 | + that flag by UNCONDITIONALLY overwriting `CUDA_VISIBLE_DEVICES` — stomping |
| 72 | + `run_comfyui.sh`'s `COMFYUI_GPU` masking and un-masking the container back |
| 73 | + onto physical GPU 0 (~40 GB held by the resident LLM server). Explains the |
| 74 | + 2026-07-04 exit-137 OOM-kill and today's first Trellis2ImageToShape OOM |
| 75 | + (torch saw 19 MiB free on a "47 GiB" device). Fixed: flag dropped, GPU |
| 76 | + selection now genuinely `COMFYUI_GPU` (`COMFYUI_GPU=1 scripts/run_comfyui.sh` |
| 77 | + is the working setup while DiffusionGemma owns GPU 0). |
| 78 | +- `set -u` crash in the drtk self-heal (`$PYTHONPATH` unset) — entrypoint died |
| 79 | + before ComfyUI launch. Fixed (`${PYTHONPATH:-}`). |
| 80 | +- Blender 5 eval turntables: EEVEE renders TRELLIS.2's atlas-padding alpha as |
| 81 | + transparent patchwork → harness renders with WORKBENCH + texture color. |
| 82 | + |
| 83 | +**Open:** native service env stand-up (R5 acceptance); Pixal3D head-to-head |
| 84 | +(R8; NOTE: Pixal3D is MIT — ADR-025 amended 2026-07-09); R10 pose-solve at |
| 85 | +assembly (placement hints already recorded per asset); R9 references baseline |
| 86 | +committed from the first 3-object eval sweep. |
| 87 | + |
5 | 88 | ## 2026-07-02 — First LichtFeld-native E2E on real raw capture (rawcapdev) + build, pipeline, and web fixes |
6 | 89 |
|
7 | 90 | ### E2E milestone: rawcapdev gallery still-life |
|
0 commit comments