Open this on the BEEF machine, tap Copy, paste the whole thing into OpenCode as its task. It builds the ComfyUI world-gen pipeline phase by phase and reports back.
# BUILD BRIEF: World-Generation Pipeline on the BEEF Machine **For:** the OpenCode agent running on the studio BEEF machine (48GB VRAM, OpenRouter models). **Goal:** stand up Mickmumpitz's open-source image-to-3D-world pipeline in ComfyUI, end to end, so a single 360 panorama (or a text prompt) becomes an explorable Gaussian-splat world. **You are the builder.** Do the install, dependency, download, and wiring work autonomously. Report after each PHASE and wait for a go before the next one. Do not guess when a real value is required — ask. --- ## OPERATING RULES (read first) 1. **No admin / user-space only.** This machine is IT-controlled. Install everything into a user-writable folder (propose `C:\Users\<me>\worldgen\` or the user's Documents). Use the ComfyUI **portable** build and a self-contained Python env so nothing needs administrator rights or system PATH changes. If a step genuinely requires admin, STOP and report exactly what and why, do not work around IT. 2. **Report per phase.** After each phase, print: what you installed, where, versions, disk used, and any errors. Then wait. 3. **Verify, don't assume.** After downloads, confirm file sizes and that models are in the exact folders. After installs, confirm the node/module imports. 4. **Keep a build log** at `worldgen\BUILD-LOG.md` — every command, path, URL, and result. This is how we debug remotely. 5. **Don't touch the network config.** This machine stays isolated. You only need outbound internet to download models and nodes. Do not install Tailscale, remote-access tools, or anything that phones home. --- ## PHASE 0 — Assets the human places first (DO NOT SKIP) Before you build, confirm these exist in `worldgen\_assets\`. If any are missing, STOP and list what's missing: - `worldgen_workflow.json` (and any `dataset_creation.json` / `panorama.json`) — Mickmumpitz's workflow files. - The trained LoRAs he provides (the 360-panorama LoRA and the green-screen outpaint LoRA). - His custom node pack if distributed as files (otherwise ComfyUI Manager resolves them in Phase 3). - Optional: a starting 360 panorama image (`start_pano.jpg`). If none, the workflow can generate one from a text prompt. **Note:** these are Mickmumpitz's own IP (Patreon / free release). You do NOT recreate the LoRAs or nodes. You install and orchestrate what the human placed here. --- ## PHASE 1 — ComfyUI portable + Python env 1. Download the latest **ComfyUI portable (Windows, NVIDIA)** into `worldgen\`. Extract. 2. Confirm it launches and detects the 48GB GPU (run the included `run_nvidia_gpu.bat`, check the console shows the CUDA device and VRAM). Kill it after confirming. 3. Confirm the bundled PyTorch is CUDA-enabled (`python -c "import torch;print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"` using the portable python). If it reports CPU-only, install the matching CUDA PyTorch wheel into the portable env. 4. Report GPU name, VRAM, torch + CUDA versions. ## PHASE 2 — ComfyUI Manager 1. Install **ComfyUI Manager** into `ComfyUI\custom_nodes\` (git clone). This is what auto-resolves missing nodes later. 2. Relaunch, confirm the Manager button appears in the UI. Report. ## PHASE 3 — Load the workflow, resolve custom nodes 1. Load `worldgen_workflow.json` into ComfyUI. 2. It will flag missing custom nodes. Use **Manager → Install Missing Custom Nodes**. Expect at least: **Splat Kit**, **Make Mopeds (Moge)**, Mick's node pack, and video/Wan helper nodes. 3. Restart, reload the workflow, confirm ZERO red "missing node" boxes remain. If any node can't be resolved by Manager, report its exact name so the human can point you at the repo. ## PHASE 4 — Models (into exact folders) The workflow's model-loader nodes contain the **download links and the target folder** for each model (Mick puts them right in the nodes). Read those, then download and place each: - **Wan 2.1** video model → `ComfyUI\models\` (per the node's stated subfolder, usually `diffusion_models` / `checkpoints`). - **Moge** depth model → its stated folder. - **Light X2V** speedup model → its stated folder. - The two **LoRAs** from `_assets\` → `ComfyUI\models\loras\`. - Any VAE / text-encoder the loader nodes call for. Verify each file's size matches the source. With 48GB VRAM these all fit comfortably. Report the full model list with paths + sizes. ## PHASE 5 — Speed stack (Sage Attention + Triton) + THE PATCH 1. Install **Triton** (Windows build) and **Sage Attention** into the portable Python env. 2. **CRITICAL:** also apply the **Sage Attention patch** for this workflow. Without it, the Wan video pass renders **all-black frames**. This is the #1 known failure of this pipeline. If you see black frames later, this patch is the cause. 3. Mick provides free auto-installers for Sage/Triton for ComfyUI — prefer those if present in `_assets\`. Report what you installed and the patch status. ## PHASE 6 — Test run (small, verify the chain) 1. In the workflow, set panorama mode. Either use `_assets\start_pano.jpg`, or set text-prompt mode with a simple prompt (use Mick's LoRA trigger caption prefix, then the scene, e.g. an alpine village). 2. Run panorama generation → confirm a seamless pano is produced (the seam-fix group should run). 3. `Compute Geometry` on the geometry node → confirm the top/side scene preview appears. 4. Draw ONE short drone path (look-forward mode), save the path video, confirm the preview flythrough looks right (not crashing through buildings). 5. Run the inpainting pass → **confirm the frames are NOT black** (if black → Phase 5 patch). Confirm the high-res reprojection composite runs. 6. Confirm a dataset folder appears in `ComfyUI\output\` in COLMAP format. Report the path + a couple sample frames. ## PHASE 7 — Train the splat 1. Install **Brush** (open-source Gaussian-splat trainer) from its GitHub releases into `worldgen\` (portable, no admin). (Postshot is the paid alternative; default to Brush.) 2. Point Brush at the Phase 6 output dataset folder, start training. 3. Confirm the point cloud forms and the scene becomes explorable. Export the trained splat (`.ply` / `.splat`) to `worldgen\output_splats\`. 4. Report training time and the export path. --- ## DONE = success criteria - One command/prompt in → an explorable splat world out, no black frames, water reflections intact. - Everything in `worldgen\`, no admin used, isolated machine untouched network-wise. - `BUILD-LOG.md` complete so the next run is one click. ## KNOWN FAILURE MODES (check these first if stuck) - **All-black video frames** → Sage Attention patch not applied (Phase 5). - **Missing red nodes** → run Manager → Install Missing; if unresolved, report node name. - **Soft/blurry world** → expected before the high-res reprojection; confirm that composite group ran. - **Drone crashing through geometry** → adjust the path height/points, re-preview before the full run. - **Out of VRAM** → unlikely at 48GB, but if so, enable the Light X2V speedup and lower video resolution. --- ## HAND-OFF NOTES FOR JOEL (not for the agent) - The one thing only you can provide is Phase 0: Mick's workflow JSON + his trained LoRAs. Grab the free release when it's public, or support his Patreon for the full guide + example files + LoRAs (worth it, he saved weeks of R&D). - You drive this from Prompt Walrus: paste this brief, then feed scene prompts and path intents as text. The agent does the rest on BEEF. - Results (splat files, renders) come off BEEF however IT permits (shared drive / export). Nothing needs to join your mesh.