Design review 2026-08-02/03 (Kaetemi). Resolves the W2 blocker in tutorial roadmap — "SNP hash-list streaming is not integrated with the Emscripten build and isn't pipelined". Status: design agreed, nothing implemented. To be revised further.
.snp vs .bnp for the browser.snp is the right shape and the wrong interface.
The format is a manifest with no payload — SNPK + version + {Name, SHA1, Size, LastModified} per entry (streamed_package.cpp:38-57) — with bytes out-of-band at content-addressed paths /aa/bb/rest (streamed_package.cpp:59-65). That is what an HTTP cache wants.
What blocks it is one line of contract: IStreamedPackageProvider::getFile returns a filesystem path (i_streamed_package_provider.h:36) and CIFile then fopens it (file.cpp:227-235).
Why the shape beats .bnp on the web:
.bnp is all-or-nothing — CBigFile::getFile hands back FILE* + offset (big_file.h:79), so the whole archive must exist in the FS.CHttpPackageProvider re-implements by hand (temp files, local hash tree, exists-check at http_package_provider.cpp:62).CFile::getFileSize answers from the manifest without downloading (path.cpp:2077-2092).The honest alternative — .bnp + HTTP Range against the trailing index (big_file.h:96-99) — is feasible but loses dedup, invalidates the whole URL on any repack, and depends on browser range-caching behavior. .snp uses the cache instead of fighting it.
IStream-returning provider entry point plus a memory branch in CIFile alongside the snp branch at file.cpp:218. CIFile already has a memory-cache path (file.h:184,198) — that is the cheap seam.-s ASYNCIFY, no -pthread (gui/CMakeLists.txt:75 and all of samples/3d/*); p_thread.cpp:305 concedes single-threaded mode has no scheduling. Survivable via emscripten_fetch under Asyncify, but serial and stack-unwinding per asset. The manifest is the unwritten fix — every hash and size is known up front, so prefetch can fan out in parallel before the main loop, leaving Asyncify as the miss handler.CHttpPackageProvider is synchronous curl-easy (http_package_provider.cpp:75-110); the emscripten build gets only a stub curl (CMakeLists.txt:397-410). A browser provider is a new class regardless, and belongs outside nelweb.loadPackage calls nlerror on serial exception (streamed_package_manager.cpp:57), which aborts the wasm module. A 404'd or truncated manifest is a normal transient on the web; must become a warning + false.path.cpp:1414-1417 — CPath::addSearchStreamedPackage delegates to _FileContainer.addSearchBigFile. The static entry point does the wrong thing entirely. Only the auto-detect route (path.cpp:1182-1185, calling the member) works today.streamed_package_manager.cpp:44-47, 62-66 — m_Entries stores const CEntry* into the package's std::vector. Reloading a package name destroys the old vector; entries that vanished leave dangling pointers. The failure path at :58 erases while stale pointers may persist.streamed_package_manager.cpp:75-76 — list() dereferences find() without an end() check.streamed_package_manager.cpp:109-111 — unreachable return true.snp_make/main.cpp:226-233 — update scan is O(entries × files); ~10⁹ string compares at Ryzom corpus scale. :354-355 — -l output has no newlines.The delta engine ports: CXDeltaPatch::apply is internal C++ (login_patch.cpp:2929-2950 → login_xdelta.cpp:789) needing only zlib gzFile (login_xdelta.h:155-183). Everything around it does not.
navigator.storage.persist(), the UA may evict the origin. A patcher's premise is "the local copy is a known version N"; the browser reserves the right to delete that silently.CPatchThread::processFile applies chains sequentially, whole-file, materializing .tmp__j per step (login_patch.cpp:2797-2900). N patches on a 300 MB bnp = N full read+write cycles and 2× file size free quota at each step. Native already flinches — see the bail-out heuristic at :2712.(filesize, mtime ±2s) (:1784-1795) with SHA1 rescan as fallback, and applyDate (:1674) writes it back. Emscripten FS backends do not round-trip mtime reliably; if it does not survive, every startup falls through to hashing the entire data set.CCheckThread, CPatchThread, CScanDataThread, CDownloadThread, CInstallThread); curl (:1419); createBatchFile/executeBatchFile/reboot (:725, :1004, :1079) which are meaningless in a browser.Content addressing dissolves the problem rather than solving it. A changed file is a new hash, therefore a new immutable URL; unchanged files keep their URL and cache entry. Delta update at file granularity, free, with no patch generation, no version chains, no _.ref files, no batch files. patch_gen's server-side apparatus stops existing.
Bnp+xdelta only wins on intra-file deltas — a large file that changes in small ways. Rare; address case by case if it ever bites.
Not a new architecture — the native one with the installer replaced by a fetch. The client already has installed bnps plus streamed snp, and CPath already treats both as one namespace.
--embed-file, as samples/3d/font/CMakeLists.txt:13 already does. Kilobytes.CBigFile works as-is in MEMFS (plain nlfopen/seek/read with thread-local handle caching, big_file.cpp:150,713). Engine work: fetch into MEMFS, then CBigFile::add.Rule that keeps the split honest: anything with size goes to snp even if required (put it on the prefetch warm list); anything small goes to bnp even if optional. snp is "fetched individually", not "optional" — see the measured cost in §4b.
Why bnp rather than --preload-file (what samples do today, gui/CMakeLists.txt:66-71): a preload .data blob is welded to one binary, so changing one texture invalidates the blob and ties it to that build. A versioned bnp URL lets binary and assets version and cache independently.
The seam is free: CPath inserts both tiers into the same lookup map with the same pack@name convention and extension remapping (path.cpp:1445-1470), so files move between tiers with zero call-site change.
Every bnp and snp lives at an immutable versioned or content-addressed URL, Cache-Control: immutable. Consequences:
ryzom_%05d.idx plus all of patch_gen's apparatus collapse into one ~1 KB immutable JSON per client version.Exactly one mutable object in the system. Preferably push it up to the HTML wrapper itself (version baked in at deploy), so even the index is immutable.
Consider content-addressing the bnps too — <sha1>.bnp — so immutability is intrinsic rather than conventional and bnps dedupe across client versions.
The JS wrapper reads the version manifest and fetches the initial data before the wasm module starts.
fetch is natively async.Promise.all..wasm, the bnps, and the snp manifests download as one parallel wave (today --preload-file serializes them).Mechanism is emscripten's own: Module.preRun plus addRunDependency/removeRunDependency. Use FS.createDataFile(..., canOwn=true) to transfer the fetched ArrayBuffer rather than copying it — without it you hold both the fetch result and the MEMFS copy.
Note: MEMFS stores contents as JS-side typed arrays, not wasm linear memory. Bundles do not consume the wasm32 4 GB address space or interact with ALLOW_MEMORY_GROWTH realloc cost. Only bytes actually read land in the wasm heap. Resident for the session either way.
Keep the wrapper dumb — it fetches a list it was handed. All policy (category selection, the main_exedll_* platform filtering at login_patch.cpp:1250-1313, required-vs-optional) resolves at manifest-generation time, or you get asset policy in two languages.
Do not parse ryzom_%05d.idx in JS — it is the persistent_data format (DECLARE_PERSISTENCE_METHODS in bnp_patch.h). Emit a small JSON sidecar from patch_gen; .idx stays canonical for native.
Handoff back to C++: write the same manifest JSON into MEMFS and let startup code read it, then addSearchBigFile / addSearchStreamedPackage per entry. (Fix path.cpp:1414-1417 first, or call the container method.)
Skip a Service Worker initially — the HTTP cache does this job. It only earns its complexity for genuine offline play.
Streaming during play. CAsyncFileManager is a CTaskManager — a background thread (async_file_manager.h:29-34). Single-threaded wasm has none, so zone loads, async textures, and snp misses land on the main thread and block the frame. This is the next hard problem, and it decides whether a browser client is playable or merely bootable. Answer is either -pthread (SharedArrayBuffer, COOP/COEP headers on the host) or reworking the async loaders around Asyncify with a per-frame time budget.
Cold start is worse than a native install — first visit pays the full bnp tier before anything renders. That is the number to watch as the design ages.
~/snowballs_reference/data)457 files, 46,202,880 bytes. 225 dds, 96 zonel, 52 ig, 47 tga, 12 shape, 9 anim, 3 swt, 3 ps, 1 skel, 1 rbank, 1 gr, 1 bank, 1 farbank, fonts.
Whole set, bundled (tar):
| codec | output | ratio |
|---|---|---|
| zstd-19 | 12,646,633 | 3.65× |
| gzip-9 | 17,586,732 | 2.63× |
Per directory, bundled, zstd-19:
| dir | raw | comp | ratio |
|---|---|---|---|
| zones | 18,288,640 | 4,283,947 | 4.27× |
| maps | 9,943,040 | 3,231,208 | 3.08× |
| pacs | 8,376,320 | 3,677,015 | 2.28× |
| tiles | 7,301,120 | 1,176,082 | 6.21× |
| shapes | 2,058,240 | 190,020 | 10.83× |
| anims | 204,800 | 48,865 | 4.19× |
Window size (zstd-19, whole tar) — plateaus at 8 MB; transport-cap worries are noise at this scale:
| wlog | window | output |
|---|---|---|
| 21 | 2 MB | 13,395,037 |
| 22 | 4 MB | 12,914,909 |
| 23 | 8 MB | 12,646,633 |
| 24 | 16 MB | 12,615,613 |
| 27 | 128 MB | 12,615,715 |
Tiles (225 DDS, ~29 KB average), held-out test (trained on odd files, measured on even):
| ratio | 7.3 MB becomes | |
|---|---|---|
| bundled, zstd-19 | 6.21× | 1.18 MB |
| per-file, zstd-19 + 64 KB trained dict | 2.31× | 3.16 MB |
| per-file, zstd-19 no dict | 1.85× | 3.95 MB |
Bundling beats a dictionary by 2.7× — the dictionary is capped at 64 KB while a bundle gets the full window.
Dictionary gain by file size, held-out: 19.6% on 29 KB tiles, 3.4% on 190 KB zones. Dictionaries only earn their keep on small files that must be individually addressable — i.e. the snp tier and nowhere else.
Per-file vs bundled, zstd-19 no dict: shapes 9.99× vs 10.83×, anims 3.47× vs 4.19×, zones 3.61× vs 4.27×.
Consequence: every file moved from bnp to snp costs roughly 2.7× in bytes for that file. Bundle aggressively; compress the bundle.
Emit both .zst (zstd-19) and .br (brotli-11) for every bundle; let Accept-Encoding negotiate. zstd is the primary artifact — the native client consumes the same .zst, so one codec, one library in patch_gen, one artifact serving both targets. Brotli exists purely as a compatibility fallback nothing else depends on.
Content-Encoding: zstd support as of 2026-08: Chrome/Edge 123+, Firefox 126+, Opera 109+, Safari 26+ (partial) / 26.3+ (full). Gaps: Samsung Internet, Safari ≤25 (i.e. anyone on an older iOS). That tail is exactly what content negotiation covers.
Practicalities:
foo.bnp.br without Content-Encoding: br delivers garbage. Prefer brotli_static-style server config over encoding-in-filename.Content-Encoding or supply precompressed variants, and typically compresses only text MIME types — so a .bnp as application/octet-stream likely ships uncompressed (46 MB instead of 12.6 MB). Verify with curl -H 'Accept-Encoding: zstd, br' -I against a binary on the existing Pages site. Assume assets must go to cdn.ryzom.dev, which pulls CORS into round one.Hashes are incompressible, so brotli does almost nothing for a .snp. The lever is the encoding. Current per-entry cost (streamed_package.cpp:48-57): 1 (per-entry serialVersion) + 4+name + 4 (length prefix on a fixed-width CHashKey) + 20 hash + 4 Size + 4 LastModified ≈ 37 + name.
v2 target ≈ 18 + name:
LastModified from the client-facing manifest (keep it in a build-side sidecar for snp_make's incremental rebuild).At 30k entries: ~2 MB → ~1 MB, on the critical path before anything renders.
46.2 MB / 457 files — small enough to run a full build→publish→deploy cycle in minutes while the pipeline is still wrong. That is the whole reason; it has no unique correctness property (see §5d).
~/snowballs_source (105 MB — max, maps, tilebank, ligo, veget_set, sfx, fonts).~/snowballs_reference/data (46 MB), from snowballs-data-20030801.zip.Do not tier Snowballs. 12.6 MB compressed is one fetch. Ship one bnp. Generate a second, split manifest as a CI fixture (zones into .snp) so the snp path stays exercised — label it a fixture in code, or someone will "fix" the shipping config to match it.
net, not assetssnowballs2/CMakeLists.txt:5 requires NeL 3d misc net pacs sound. NeL networking is TCP — no browser equivalent without a WebSocket bridge.
Good news: network.cpp already guards every call site with if (!isOnline()) return; (:274,283,303,319,393). Phase one is an offline single-player build — landscape, character, camera, snowball physics, no services. Multiplayer via a WS bridge is a separate later project.
Sound is probably fine (emscripten ships OpenAL and vorbis ports); verify the NeL OpenAL driver builds, and stub it for phase one if it fights — SBCLIENT_DEV_SOUND is already a toggle (snowballs_config.h:54).
Agreed scope 2026-08-03: no dedicated asset server yet. Snowballs data build is a separate job in the existing GitHub CI, source assets pulled from a static CDN archive (the ryzomcore_graphics-rev5.7z pattern already at emscripten-samples.yml:54-74), and the emscripten publish job consumes the built data.
All tooling is Linux-native and unconditionally available: WITH_NEL_TOOLS + WITH_3D + WITH_PIPELINE_NATIVE_OLE turns on every pipeline_max_export_* plus zone_lighter_2003 (nel/tools/3d/CMakeLists.txt:1-49). No WIN32 gate, no Wine, no libgsf.
Preserve asset_revision as an explicit computed value even though in round one it is just a GH Actions cache key:
asset_revision = hash(source archive id + exporter binary hashes + lighter cfg + build config)
Emit it as a file in the artifact and a field in the manifest. Phase 3 (dedicated server) then becomes a backend swap, not a redesign. The key must include the exporter binaries, or an exporter fix silently serves stale data — hashing the built binaries is cruder than tracking a dependency subtree but is self-maintaining and cannot be wrong.
Risks:
zone_lighter is the schedule unknown — 96 zones with 8-ring dependencies on a 4-core hosted runner. Measure before wiring anything else; if a cold build is 40 minutes the plan depends on cache reliability.workflow_dispatch-able alone; a broken asset build must not take down the existing sample deploys.Byte-matching reference is not a publish gate — formats diverge (recompression, re-containering, possible mobile texture variants), so it would be disabled on the first real change. Exporter-output equality against reference is meaningful and already gated upstream by the pipeline_max ctest battery.
Round-one gates:
.shape/.ig/.zonel resolves inside the bundle set. Catches the actual common failure (missing file, extension remap that did not fire).Later:
Reference data keeps one narrow role: a one-time visual baseline for the first published revision, judged by eye. After that it is purely a development aid for the exporters.
preRun wrapper. Ends with a playable page.rev5 stops being a literal.Each phase independently useful and revertable.
Deferred past round one. When it lands:
asset_revision, compress=zstd:3, checksums on a long-lived corpus (silent corruption in a 2003 source archive would surface years later and be unattributable), send/receive to ship revisions. chattr +C the build scratch directory — repeated rewriting is exactly what fragments CoW filesystems. Alternative: XFS (reflink=1) for scratch on a separate volume.dedup=on; you would be paying a scanner to rediscover what the naming scheme already encodes.)algorithm_params, algo=zstd level=8 dict=/path) but is RAM-backed. EROFS's Z_EROFS_ZSTD_MAX_DICT_SIZE is almost certainly window size, not a trained dictionary. If dictionary-class gains are wanted: order similar files adjacently in a SquashFS/EROFS zstd image so the window catches cross-file redundancy, or use the zstd seekable format (contrib/seekable_format) — same shape as the snp tier with the archive doing the dictionary's job.needs:, outbound-HTTPS only, so NAT is a non-issue — but fork PRs must never reach it); repository_dispatch (server pushes); commit status / check run; custom deployment protection rules (verify availability on the github-pages environment, emscripten-samples.yml:404-413); polling last.zone_lighter_2003 wall time on Snowballs (blocks §5c sizing).curl -I GitHub Pages for Content-Encoding on octet-stream (decides §4c hosting).-pthread (only matters if persistent storage ever returns).