Renderive/render_3D/datoviz/spec/data/WEBGPU_DATA_BUNDLES.md

WebGPU Data Bundles

Status: v0.4 implementation handoff Scope: prepared example data used by live WebGPU/WASM gallery routes

Implementation checkpoint, 2026-07-16:

1. web/wasm/data_loader.js validates and mounts fetched bundles before scenario creation.
2. tools/build_webgpu_data_bundles.py validates committed sources and stages declared live-route bundles for MkDocs; cache-only sources are rejected.
3. The WASM module exposes FS, and loader path, size, hash, cache, and unsafe-path behavior has a Node smoke test.
4. SVG Tiger uses the first approved committed fetched bundle. Terrain Relief has portable packet proof but still awaits an approved committed data bundle.

Live WebGPU examples that require data should load versioned static data bundles at runtime. Do not grow the base WASM scene module by preloading every promoted gallery dataset. The browser runner is host glue: it fetches prepared bytes, mounts them into the Emscripten filesystem, and then runs the same portable C scenario as native validation.

Decision

Move data-backed live examples from build-time --preload-file packaging to per-example fetched web bundles.

The canonical C example keeps reading ordinary Datoviz data paths:

data/examples/<example_id>/manifest.json
data/examples/<example_id>/prepared/<artifact>
data/assets/<asset_group>/<artifact>

The website serves those files from hash-versioned static URLs:

webgpu-data/examples/<example_id>/<bundle_version>/manifest.json
webgpu-data/examples/<example_id>/<bundle_version>/prepared/<artifact>
webgpu-data/assets/<asset_group>/<bundle_version>/<artifact>

Browser JavaScript fetches the bundle, verifies the declared artifacts where practical, writes the files into the Emscripten virtual filesystem at the canonical paths, and only then creates the scenario. Scenario code must not know whether data came from the local data submodule, a local cache, a WASM preloaded file, or a web fetch.

Non-Goals

1. Do not create browser-only data formats for gallery routes.
2. Do not parse or reinterpret scientific data in JavaScript beyond fetching, integrity checks, and filesystem mounting.
3. Do not silently synthesize fallback data for examples declared as prepared, real-data, or externally sourced.
4. Do not make every data-backed showcase live. Static media remains the default gallery artifact.
5. Do not commit generated site copies or packed runtime data to the parent repository unless a release-artifact policy explicitly approves those exact files.

Bundle Contract

The existing datoviz.example-data.v1 manifest remains the source of truth. Add a web section only when a bundle is eligible for browser serving:

{
  "schema": "datoviz.example-data.v1",
  "id": "showcases_choropleth",
  "status": "committed",
  "web": {
    "version": "sha256-0123456789abcdef",
    "virtual_root": "data/examples/us_state_choropleth",
    "max_bytes": 8000000,
    "required": true
  },
  "artifacts": [
    {
      "role": "points",
      "path": "prepared/points_xy_f64.bin",
      "format": "raw",
      "dtype": "f64",
      "shape": [123456, 2],
      "byte_size": 1975296,
      "sha256": "..."
    }
  ]
}

Required web fields:

Field	Meaning
version	Stable content version used in static URLs. Prefer a short aggregate SHA-256 prefix.
virtual_root	Emscripten filesystem directory where the bundle is mounted.
max_bytes	Browser promotion budget. The loader must reject bundles larger than this value.
required	Whether scenario creation must fail when the bundle cannot be loaded.

Artifact entries should already carry path, byte_size, sha256, format, dtype, and shape when known. The web loader should treat path as relative to virtual_root, and should reject absolute paths, .., empty path segments, and duplicate output paths.

Example Manifest Fields

examples/c/MANIFEST.yaml should point live routes to bundle manifests rather than relying on implicit preloaded files:

webgpu:
  status: webgpu-live
  route: examples/webgpu/live.html?id=showcases_choropleth
  data_bundles:
    - id: us_state_choropleth
      url: webgpu-data/examples/us_state_choropleth/sha256-0123456789abcdef/manifest.json
      virtual_root: data/examples/us_state_choropleth
      required: true

The generated examples/webgpu/live_examples.js entry should mirror only the browser host fields:

{
  id: "showcases_choropleth",
  label: "U.S. State Choropleth",
  scenarioId: "showcases_choropleth",
  dataBundles: [
    {
      id: "showcases_choropleth",
      url: "../../webgpu-data/examples/us_state_choropleth/sha256-0123456789abcdef/manifest.json",
      virtualRoot: "data/examples/us_state_choropleth",
      required: true,
    },
  ],
}

Loading Flow

The live route should load data before scenario creation:

examples/webgpu/live.html?id=<example_id>
  -> live.js resolves LIVE_EXAMPLES entry
  -> data_loader.js fetches each dataBundles[] manifest
  -> loader validates web metadata, total size, artifact paths, and optional SHA-256 hashes
  -> loader fetches each artifact as ArrayBuffer
  -> loader writes manifest/provenance/artifacts into Module.FS
  -> WasmSceneSession creates the portable C scenario
  -> scenario reads canonical data paths
  -> scene emits DRP2 packets
  -> WebGPU runtime renders

The loader should be idempotent. Re-entering the same live route may skip files already mounted with the same bundle id and version. Different versions should replace the mounted bundle directory before scenario creation.

WASM Runtime Requirements

The generated Emscripten module must expose filesystem access to the JS host. The implementation may use either direct Module.FS exposure or a small wrapper in web/wasm/scene.js, but the public browser runner should not depend on undocumented globals outside the WASM session module.

Recommended direction:

1. export FS and NODEFS/MEMFS support needed by the browser build;
2. add web/wasm/data_loader.js with mountDataBundles(Module, bundles, options);
3. call mountDataBundles() from WasmSceneSession.load() before DatovizWasmScene.createScenario();
4. remove example-data --preload-file entries from src/wasm/CMakeLists.txt once fetched bundles cover those live routes;
5. keep only tiny, truly shared runtime assets preloaded if they are required before any route is known.

This is an intentional API break for the browser host. It is better to break the experimental v0.4 WebGPU loader now than to let data-backed gallery routes depend on a growing monolithic WASM package.

Static Site Generation

The MkDocs build should copy only selected web-eligible bundles into the generated site:

examples/c/MANIFEST.yaml
  -> collect webgpu-live entries with webgpu.data_bundles
  -> read each source data manifest
  -> validate status, web metadata, artifact hashes, total size, and provenance links
  -> copy manifest, PROVENANCE.md, BLOCKERS.md when present, and prepared artifacts
  -> write site/webgpu-data/examples/<id>/<version>/
  -> optionally write site/webgpu-data/index.json

Suggested source-to-site mapping:

Source	Site output
data/examples/<id>/manifest.json	site/webgpu-data/examples/<id>/<version>/manifest.json
data/examples/<id>/PROVENANCE.md	site/webgpu-data/examples/<id>/<version>/PROVENANCE.md
data/examples/<id>/prepared/*	site/webgpu-data/examples/<id>/<version>/prepared/*
.cache/datoviz/examples/<id>/prepared/*	Not copied unless promotion policy explicitly allows it.

Cache-only data may be useful for local development, but it should not be promoted to public webgpu-live without redistribution and provenance review.

Serving Policy

The site can serve bundles as ordinary static files:

File kind	Cache policy	Notes
Versioned binary artifacts	Cache-Control: public, max-age=31536000, immutable	URL contains version.
Versioned manifest/provenance	Cache-Control: public, max-age=31536000, immutable	Content-addressed with bundle.
Unversioned index files	Cache-Control: no-cache or short TTL	Optional convenience only.

Use application/octet-stream for .bin, .npy, .npz, .gz, and other binary artifacts. Correct MIME types are useful, but loaders must rely on manifest metadata, not MIME sniffing.

Browser Cache

The first implementation can rely on HTTP caching only. A later optimization may add Cache API or IndexedDB storage keyed by:

datoviz-webgpu-data:<bundle_id>:<version>:<artifact_path>

Do not add browser persistence until there is a demonstrated repeat-download problem. If persistent cache is added, provide a small invalidation path and keep the no-cache path available for tests.

Failure Behavior

A missing required bundle should fail before scenario creation with a precise diagnostic:

Missing prepared data bundle for us_state_choropleth.
Run: python tools/data/prepare_us_state_choropleth.py

Other required diagnostics:

1. bundle manifest fetch failed;
2. artifact fetch failed;
3. total bundle size exceeds web.max_bytes;
4. SHA-256 mismatch;
5. artifact path escapes web.virtual_root;
6. manifest status is not web-eligible;
7. bundle requires data that is blocked by BLOCKERS.md.

The gallery page should still show static PNG or video fallback media outside the iframe. The live iframe may show the failure status, but it must not pretend that the real example rendered.

Validation

Add focused validation rather than exhaustive browser coverage:

1. python3 tools/data/validate_manifests.py data/examples/<id>;
2. a new static-site bundle checker for webgpu.data_bundles;
3. node --check examples/webgpu/live_examples.js;
4. just wasm-scene-smoke;
5. just webgpu-browser-smoke for at least one data-backed live route;
6. git diff --check.

The browser smoke should assert that a data-backed route reaches rendered status with a nonblank canvas when WebGPU is available, and reports the expected missing-bundle diagnostic when served without the copied bundle.

Migration Plan

1. Add webgpu.data_bundles metadata for showcases_choropleth and showcases_protein.
2. Add web/wasm/data_loader.js and route it through WasmSceneSession.
3. Expose the Emscripten filesystem through the WASM session module.
4. Teach tools/build_gallery.py or tools/mkdocs_hooks.py to copy declared web bundles.
5. Add a bundle validation script and hook it into the narrow gallery validation loop.
6. Remove dataset-specific --preload-file rules from src/wasm/CMakeLists.txt.
7. Keep texture/font/runtime assets only if they are shared enough to justify preload.
8. Promote additional examples only when their bundle is compact, redistributable, and documented.

Stop after each migrated example and record browser evidence in examples/webgpu/COMPAT.md.

Promotion Criteria

An example may be webgpu-live with data only when:

1. the scenario is the same portable C scenario used by native validation;
2. prepared artifacts are redistributable or generated with clear provenance;
3. the browser bundle is compact enough for gallery use;
4. source, license, citation, and preprocessing are linked from the example page;
5. missing data fails with a preparation command;
6. static media exists as the default gallery fallback.

If any criterion is missing, mark the example webgpu-planned, webgpu-deferred, or native-only instead of adding synthetic browser fallback data.