Skip to content

Commit c0c71da

Browse files
jjohareruvnet
andcommitted
docs(objects): convergence documented — proof assets, log entry, ADR-025 accepted+amended, PRD status
- docs/renders/object-pipeline-2026-07-09/: the high-value proof set (4 MB) — source frame, FIXED SAM3 silhouettes overlay, the three generator-ready crops, and turntables of all three generated PBR assets (incl. the model-completed vessel backside), with a README tying each image to the requirement it proves + the run metrics table. - docs/engineering-log.md: full 2026-07-09 entry (R1 root cause, first-ever validated automated per-object isolation, live crop→GLB verification, ComfyUI GPU-collision root cause, Blender 5 turntable notes). - ADR-025: Proposed → Accepted, with verification amendments — Pixal3D is MIT (the 'custom non-MIT' rejection ground is void; soak argument stands, R8 eval should be prioritised) and the cited Trellis2_multiimage fork does not exist (the decisive #103 quality evidence is confirmed verbatim). - PRD v4: per-requirement implementation status (R1-R6a+R9 done, R5 native env + R7 ladder + R8 eval + R10 pose-solve open). - architecture/overview + workflows/video-to-splat + architecture.md: stage diagrams and hull sections rewritten for the single-image object arc (object_crops stage; no geometry-from-partial-splat fallbacks for objects). Co-Authored-By: claude-flow <ruv@ruv.net>
1 parent 92dcb00 commit c0c71da

16 files changed

Lines changed: 182 additions & 31 deletions

docs/architecture.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -315,7 +315,7 @@ The `ModelLifecycleManager` hard-tier uses `docker stop` / `docker start` on `vi
315315

316316
The owner `vitrine-comfyui` is pinned to a known-good state via the **Salad add-on control API** (`vitrine-comfyui:3001`) — an in-container control plane for model probe, download, and lifecycle, not a cloud service. The `RecoveryController` (a stateless helper the orchestrator invokes) runs a per-object loop:
317317

318-
1. **Plan** — compose the FLUX.2-dev view-completion or TRELLIS.2 / Hunyuan3D-2.1 hull graph from a template, parameterised by object identity and the artifact report.
318+
1. **Plan** — compose the FLUX.2-dev recovery graph or the TRELLIS.2 / Hunyuan3D-2.1 SINGLE-image object graph (ADR-025 — conditioning is the object_crops matte, never splat renders) from a template, parameterised by object identity and the artifact report.
319319
2. **Submit**`POST /prompt` (graph API); poll `/history/{id}`; fetch outputs.
320320
3. **Evaluate** — scores the generated image or mesh render against object identity and artifact criteria. This step is *visual*, so it needs the staged gemma-4 vision fallback (or the `claude_code` backend) — text-only DiffusionGemma cannot see and handles only the planning/reasoning around it.
321321
4. **Decide**`accept` | `re-prompt` (adjust denoise/guidance/seed/mask, bounded retry budget) | `veto` (unrecoverable). Every attempt is annotated in the per-video ledger; nothing is silently dropped.

docs/architecture/overview.md

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ Gaussian Toolkit integrates multiple components into a unified 3D Gaussian Splat
6565
| SplatReady | 1.0.0 | Plugin | Video-to-COLMAP pipeline automation |
6666
| SAM3 | latest | Apache-2.0 | Concept segmentation (4M concepts, text+visual prompts) |
6767
| SAM2 | hiera-large | Apache-2.0 | Video segmentation (fallback, validated) |
68-
| TRELLIS.2-4B | 4B | MIT | **Primary** per-object hull (multiview shape+texture diffusion → PBR GLB) |
68+
| TRELLIS.2-4B | 4B | MIT | **Primary** per-object generator (SINGLE-image shape+texture diffusion → PBR GLB, ADR-025) |
6969
| Hunyuan3D-2.1 | 2.1 | Tencent-community | Per-object hull **fallback** (multi-view to textured mesh) |
7070
| ComfyUI | latest | GPL-3.0 | Node-based workflow engine (FLUX.2 / TRELLIS.2 / Hunyuan3D-2.1 / SAM3D) |
7171
| FLUX.2-dev | dev | Non-commercial | View completion + inpaint/background recovery (Qwen-Image-Edit = commercial-safe alt) |
@@ -106,10 +106,13 @@ Trained 3D Gaussian Splat Model (1M gaussians)
106106
│ 2D masks → 3D Gaussian labels (98.3% coverage)
107107
│ 33 per-object PLY files extracted
108108
109-
▼ [Stage 6: Per-Object Hull Creation]
110-
│ orbit render → coverage gate → FLUX.2 view completion
111-
│ → TRELLIS.2-4B multiview shape+texture diffusion → PBR GLB (primary)
112-
│ Hunyuan3D-2.1 fallback; TSDF floor fallback: Open3D fusion
109+
▼ [Stage 5b: Object Crops] (ADR-025 — generator conditioning)
110+
│ best-frame selection → SAM-matte crop ≥1024² + provenance
111+
112+
▼ [Stage 6: Per-Object Generation]
113+
│ TRELLIS.2-4B SINGLE-image shape+texture diffusion → PBR GLB (primary,
114+
│ bytes persisted verbatim); Hunyuan3D-2.1 single-image fallback
115+
│ (objects fail loudly — no geometry-from-partial-splat fakes)
113116
114117
▼ [Stage 7: Environment Mesh + Background Recovery]
115118
│ CoMe env mesh (MILo / GaussianWrapping / TSDF fallbacks)

docs/engineering-log.md

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,89 @@
22

33
Development history for Vitrine (a standalone project that vendors LichtFeld Studio as a pinned tool; formerly a fork — see ADR-021).
44

5+
## 2026-07-09 — Object pipeline convergence implemented (PRD v4 / ADR-025) + SAM3 root cause FIXED
6+
7+
### R1 SOLVED: "SAM3 returns boxes" was an HWC/CHW bug in OUR wrapper — one-line fix
8+
9+
The audit's blocker defect F1 is closed, and the root cause was ours, not SAM3's:
10+
`Sam3Processor.set_image`'s numpy branch reads `height, width = image.shape[-2:]`
11+
(correct for CHW tensors); our `sam3_segmentor` passed HWC frames, so it read
12+
`height=W, width=3`. Detection still worked (model input conversion handles HWC)
13+
but every output mask was interpolated to a **W×3 grid** and boxes were scaled by
14+
(3, W) — which downstream resizing smeared into the notorious "coarse boxes".
15+
Fix: pass PIL images (`_to_pil` in `sam3_segmentor.py`). Verified on rawcapdev:
16+
pixel-accurate silhouettes for vessel (fill 0.86, score 0.94), ketchup bottle
17+
(0.79/0.57) and wooden block (0.75/0.92) — overlay at
18+
`output/rawcapdev/sam3_fixed_overlay.jpg`.
19+
20+
### First-ever validated automated per-object isolation on real data
21+
22+
With silhouettes fixed + new binary-COLMAP support (`parse_cameras_bin`/
23+
`parse_images_bin` + format-agnostic loaders — the direct-COLMAP path emits
24+
bin-only models, which would otherwise fail loudly), the full Phase-1/2 arc ran
25+
live on rawcapdev: segment → **object_crops** (new stage) → extract_objects
26+
MV-projected **1.09M/4M gaussians (vessel), 246k (bottle), 250k (block)** across
27+
6 views. Zero fallbacks, zero failures. Crops are generator-ready RGBA mattes at
28+
native res (vessel 3175²) with full provenance (frame, bbox, COLMAP pose,
29+
matting method, selection scores) in `object_crops/crops.json`.
30+
31+
### ADR-025 implementation (all PRD v4 phases except live 3D generation)
32+
33+
- **R2** `extract_objects`: full-scene-copy + world-XY fallbacks DELETED; per-object
34+
failures reported with cause (`object_failures` artifact). `full_scene` label
35+
(the environment) still copies by design.
36+
- **R3** new `object_crops` stage (`src/pipeline/object_crops.py` + config):
37+
best-frame selection = silhouette area × log-sharpness × centrality × edge
38+
clearance; SAM-matte alpha; rembg fallback for box-like masks; ≥1024² square pad.
39+
- **R4** multiview conditioning RETIRED: `trellis2_client.py` rewritten single-image
40+
(ComfyUI executor now, native-service contract ready); `view_completer.py`,
41+
`trellis2_multiview_pbr.json`, `flux2_turnaround.json` + all 3 stock-node Hunyuan
42+
workflow JSONs deleted; `multiview_renderer` kept preview-only and fixed to
43+
straight alpha (F4 un-premultiply).
44+
- **R5** `scripts/trellis2_native_service.py` scaffold (HTTP contract final; model
45+
calls marked VERIFY-ON-ENV-BUILD; client flips via `trellis2.native_url`).
46+
- **R6** generator GLB bytes persisted byte-identical (sha256 recorded, lineage
47+
sidecar per asset); `texture_bake` skips generator meshes (F7 closed).
48+
- **R6a** Hunyuan fallback = `reconstruct_from_image()` submitting the PROVEN
49+
Hy3D21 graph (hy3d_turnaround.py lineage) in code.
50+
- **R9** eval harness `eval/objects/run_eval.py` (+ blender turntable): mesh
51+
stats/regression gates; `--stats-only` validated against the 2026-07-02 brass
52+
vessel GLB (274,588 verts / 483,671 faces / PBRMaterial — exact match).
53+
- **Tests**: 30 new unit tests (crops, single-image client, object-arc
54+
regressions); pipeline subset **144/144 green** in-container.
55+
56+
### LIVE RUNTIME VERIFICATION: crop → single-image TRELLIS.2 → PBR GLB
57+
58+
The full ADR-025 arc ran end-to-end on GPU 1: the automated vessel crop →
59+
`Trellis2ImageToShape` (1536_cascade, seed 42) → **54.8 MB PBR GLB, 286,009
60+
verts / 491,067 faces in 479 s**, persisted byte-identical (sha256
61+
`496d9a84…`) with lineage sidecar — same output class as the manual
62+
2026-07-02 vessel (274,588 / 483,671). Workbench turntable confirms solid
63+
geometry + patina albedo (`output/object_e2e/`). Smoke script:
64+
`scripts/run_hull_e2e.py` (rewritten crop-based; the splat-render version is
65+
gone).
66+
67+
### Infra root causes fixed along the way
68+
69+
- **ComfyUI has been fighting DiffusionGemma for GPU 0 all along**:
70+
`comfyui_entrypoint.sh` passed `--cuda-device 0`, and ComfyUI implements
71+
that flag by UNCONDITIONALLY overwriting `CUDA_VISIBLE_DEVICES` — stomping
72+
`run_comfyui.sh`'s `COMFYUI_GPU` masking and un-masking the container back
73+
onto physical GPU 0 (~40 GB held by the resident LLM server). Explains the
74+
2026-07-04 exit-137 OOM-kill and today's first Trellis2ImageToShape OOM
75+
(torch saw 19 MiB free on a "47 GiB" device). Fixed: flag dropped, GPU
76+
selection now genuinely `COMFYUI_GPU` (`COMFYUI_GPU=1 scripts/run_comfyui.sh`
77+
is the working setup while DiffusionGemma owns GPU 0).
78+
- `set -u` crash in the drtk self-heal (`$PYTHONPATH` unset) — entrypoint died
79+
before ComfyUI launch. Fixed (`${PYTHONPATH:-}`).
80+
- Blender 5 eval turntables: EEVEE renders TRELLIS.2's atlas-padding alpha as
81+
transparent patchwork → harness renders with WORKBENCH + texture color.
82+
83+
**Open:** native service env stand-up (R5 acceptance); Pixal3D head-to-head
84+
(R8; NOTE: Pixal3D is MIT — ADR-025 amended 2026-07-09); R10 pose-solve at
85+
assembly (placement hints already recorded per asset); R9 references baseline
86+
committed from the first 3-object eval sweep.
87+
588
## 2026-07-02 — First LichtFeld-native E2E on real raw capture (rawcapdev) + build, pipeline, and web fixes
689

790
### E2E milestone: rawcapdev gallery still-life
269 KB
Loading
445 KB
Loading
960 KB
Loading
582 KB
Loading
833 KB
Loading
274 KB
Loading
280 KB
Loading

0 commit comments

Comments
 (0)