GPU Engine

WebGPU rendering, texture management, preview output, export capture, and GPU-backed analysis.

Linux/Mesa: GPU-accelerated canvas, worker OffscreenCanvas, and WebGPU paths fail silently on open-source Mesa drivers. Before changing any canvas/GPU code, read Linux / Mesa GPU.

---

WebGPUEngine is a thin facade over the rendering subsystems. It owns WebGPU context setup, texture and cache managers, the render loop, the render dispatcher, output windows, and export canvas state.

The engine supports:

• Main preview rendering
• Independent preview targets
• Popup output windows
• Sliced output / corner-pin rendering
• RAM preview and scrubbing caches
• Zero-copy export frames when the export canvas path is available
• CPU readback as a fallback path
• Waveform, histogram, and vectorscope GPU panels
• Native 3D scene and Gaussian-splat composition when their enabled runtime flags apply

---

• WebGPUContext handles adapter/device setup and device-loss recovery.
• RenderLoop runs the RAF loop, idle detection, scrub limiting, and watchdog recovery.
• RenderDispatcher handles per-frame texture import, compositing, output, and preview fallbacks.
• RenderTargetManager owns ping/pong compositing buffers plus independent preview buffers.
• TextureManager and MaskTextureManager manage image/video/mask textures.
• CacheManager owns scrubbing cache, composite RAM preview cache, and GPU frame cache.
• ExportCanvasManager owns the export OffscreenCanvas and stacked-alpha mode.

• The engine keeps a unified Map of target canvases.
• Preview canvases and output windows are reconfigured after device restore.
• HMR reuses the engine singleton when possible.

Startup distinguishes missing WebGPU, an unavailable adapter, adapter/device request timeouts, and renderer-resource failures. The preview displays the specific failure instead of describing every failure as incompatible hardware. A timed-out request can still be pending inside the browser, so recovery does not queue additional requests behind it. Late GPU devices are destroyed.

After unexpected device loss, the context drops the lost device after cache cleanup and makes up to three recovery attempts for settled failures. Recovery waits for asynchronous renderer resources before making the preview ready. Timeouts and unsupported runtimes terminate with a visible error rather than an indefinite loading state. Destroy/reinitialize invalidates older recovery work, and retained output canvases are rebound to the replacement device.

Only the first real loss per session is reported as an error; repeats are warnings. A loss while the page is hidden, frozen, or unloading (Android Chrome reports it as “Device was destroyed.”) is a warning and recovery waits until the page is visible again. Three real losses within a minute stop recovery and show a driver-reset message instead of looping lose/recover. That stop is its own incident and files one error, GPU device lost repeatedly; recovery stopped, with the loss message, its occurrence and the window (and firstLoss when an earlier loss of the same run was still held), since the GPU stays unavailable afterwards.

Each loss incident files at most one error. The first real loss’s error waits for the recovery outcome: a recovered device reports Device lost, while a recovery that gives up (for example when Chrome refuses a new adapter after repeated GPU-process crashes) reports only GPU recovery failed, including the loss message of that incident as lostDevice (also for a later loss whose Device lost was not filed again). The individual recovery attempts in between are warnings. The preview then says the GPU stopped responding and asks to save and reload the editor to restore it. Because the GPU already worked in this session, a recovery failure shows no setup advice: no Linux Vulkan banner and no driver/Vulkan help line (setEngineInitFailed(..., { setupAdvice: false })). The console capture in runtime diagnostics drops the Logger’s clock prefix so a logged failure keeps one stable message.

Targeted coverage: webgpuContext.test.ts, contextRecoveryWiring.test.ts, and outputCanvasRecovery.test.ts under tests/unit/.

Risky GPU startup records a release-scoped boot attempt in bootHealth.ts. Three interrupted attempts inside 24 hours activate a GPU hazard for that release and route canvas policy through the existing software fallback. An attempt becomes healthy only after the active render mode proves a readable frame; worker submission alone is not treated as pixel proof.

Device loss records the same hazard signal. The visible warning offers a manual Retry GPU action that clears the safety state and reloads the app. Clean pagehide completion and a three-second settled startup prevent normal reloads from being counted as crashes. Runtime owners and the engine singleton remain HMR-safe.

A render or compute pipeline can fail in validation, in the driver compiler (internal, e.g. CreateGraphicsPipelines failed with VK_ERROR_UNKNOWN on Android Vulkan drivers) or for lack of memory. WebGPU still returns a pipeline object; setting it, or a bind group made from its auto layout, invalidates the whole command buffer, so every layer of the frame disappears.

guardGpuDevice(device, source) (src/engine/core/gpuPipelineGuard.ts) is installed right after every requestDevice that renders: the editor (WebGPUContext), the render worker, thumbnails, look thumbnails and the photo guide preview. It scopes every synchronous pipeline creation for all three error kinds and remembers failed pipelines. Once one failed, render passes, compute passes and render bundles skip its setPipeline, draws, dispatches and bind groups derived from it, so only that feature is missing. Work encoded with a pipeline before its verdict arrives is not protected: by the WebGPU spec the pass invalidates its encoder, so every browser loses that command buffer, and shipping Safari also keeps the encoder locked after a pass that set the failed pipeline, so its next beginComputePass fails with encoder state is invalid. Features that encode right after creating pipelines wait for gpuPipelineSettled first (effect pipeline cache, flock session gate, flock renderer). Healthy devices pay nothing: passes are only wrapped after a failure. Each failure is logged with feature, label, kind and driver message (huge messages are condensed to their first line and compiler error lines: WebKit prefixes the whole generated Metal source); the preview shows “Not supported by this graphics driver” with the affected features (PreviewGpuFailureNotice, full messages as tooltip). A new device clears the records of its source.

Callers that must react themselves use createCheckedPipeline(device, create), which returns the pipeline and its verdict without opening competing error scopes on a guarded device: effects skip and report themselves in the effect card, and the HAP export encoder falls back to its CPU encoder. On an unguarded device createCheckedPipeline also records the verdict for gpuPipelineSettled. markGpuPipelineFallback(pipeline, note) declares that the owner switches to an equivalent implementation when the pipeline fails: it raises no “not supported” notice, and is reported once per page session as a non-fatal ERROR-level log starting with GPU pipeline fallback used (guarded and unguarded devices), carrying the condensed driver message. Only errors reach the production diagnostics feed; the classifier gives it the failure code gpu_pipeline_fallback, so triage can tell it from a skipped feature.

queue.writeBuffer() is staged before the command buffer it belongs to is submitted. A single buffer rewritten between draws or dispatches recorded into one encoder therefore gives every recorded pass the last written data (two layers of one effect both render with the second layer’s settings). Destroying or replacing a buffer that the unsubmitted encoder still references makes the whole submit fail.

New GPU nodes and renderers must take per-draw uniforms, settings, and vertices from src/engine/core/gpuUniformRing.ts instead of owning one shared buffer:

• GpuUniformRing.write(data) returns a buffer that no other call in the same frame receives. The pool grows when a frame needs more slots and rewinds at the next frame start.
• GpuFrameBuffers.ensure(key, bytes) keeps one growable buffer per key (for example per output target). A replaced buffer is retired and destroyed at the next frame start, never mid-frame. Use retire() for any other buffer that a recorded pass may still use.
• Frames are 'explicit' when the owner has a begin hook that runs after all of its encoders were submitted (GaussianSplatGpuRenderer.beginFrame() drives the splat cull, sort, particle, and effector passes). Use 'task' for nodes reached from several frame owners without a shared hook (pixel-particle renderer, slice vertex buffers): the frame ends with the current synchronous section, so record and submit an encoder without an await in between.
• Data that a later frame reads again needs its own key, never a shared scratch buffer. The splat GPU sort writes into one sorted-index buffer per draw stream (uploaded scene plus graph branch). With a sort frequency above 1, skipped frames redraw that stream’s own last order and count, bound to the current data buffer; before, every layer reused the single shared sort output (another layer’s order, or a buffer destroyed after growth). A stream’s buffer is retired when its scene is released or after 600 renderer frames without a draw. Sort frequency 0 draws the current cull/identity order, never a frozen one. Cull counts are tracked per stream too.
• The splat camera uniforms, splat mesh uniforms, and splat graph compute slots use their own per-call pools rewound in GaussianSplatGpuRenderer.beginFrame(). They follow the same rule (one slot per call, nothing destroyed before the frame start) without the shared helper.

Runtimes with per-instance GPU state (Analog Signal Lab, compute image graphs) key instances by render scope plus effect id. Nested compositions keep their original effect ids, so the scope keeps two visible occurrences apart.

Targeted coverage: gpuUniformRing.test.ts, gpuFrameScopedRenderers.test.ts (including two GPU-sorted splat layers across a skipped-sort frame), splatSortPass.test.ts, and effectsPipelineComputeInstanceScope.test.ts under tests/unit/.

---

• HTMLVideoElement is imported as an external texture when the browser supports it.
• VideoFrame can also be imported as an external texture.
• Firefox preview uses copied textures instead of external video import because imported frames can intermittently go black.
• Android Chromium copies HTML-video preview frames into persistent GPU textures during playback, pause, and seeking because its external video textures can otherwise present intermittent black frames.

• Images and canvases are copied into rgba8unorm GPU textures.
• Cached image views are reused when possible.

• motion-shape clips render through src/engine/motion/MotionRenderer.ts.
• Rectangle, ellipse, polygon, and star primitives are drawn with analytic WGSL SDFs into transparent rgba8unorm textures.
• A bounded uniform layout composites up to 8 ordered color-fill, stroke, linear-gradient, radial-gradient, or texture-fill appearances, with up to 8 stable stops per gradient and six shader blend modes.
• Replicated motion shapes use a per-shape instance buffer and instanced draws in the same shader path. The shader ceiling is 100,000 instances; effective counts are also constrained by the device, render target, and configured user limit.
• The resulting texture view is composited through the normal CompositorPipeline, so masks, effects, blend mode, nested comps, preview targets, and export share the same downstream path.

• Mask textures are uploaded per layer.
• A 1x1 white fallback texture is used when no mask is present.

• The zero-copy path only applies when the browser and source type support it. Preview and export still have explicit fallback paths.

---

• Idle mode starts after about 1 second of no activity.
• Idle detection is initially suppressed so browser video surfaces can warm up; it is lifted by the first play event or after about 3 seconds.
• Playback is limited to about 60 fps.
• Scrubbing is limited to about 60 fps unless a fresh video frame arrives, which bypasses the limiter.
• While a video plays, its decoded frames also fill the scrubbing cache. That capture keeps to about 1.5 ms of main thread per played frame on average: when createImageBitmap of a video converts synchronously (about 6 ms per 1080p frame on Linux/Vulkan), the following frames pass uncached, so playback keeps its frame rate. ScrubTextureCache.getPlaybackCaptureStats() reports the last capture cost and the frames that passed.
• A render request is consumed only by a frame that reaches the render callback. Frames skipped during device recovery, export, or while editor content publication is blocked (project open or composition switch) keep the request, so the first frame after the gate reopens shows the change. A direct renderHostPort.render() made while publication is blocked becomes such a request instead of being dropped.
• During playback and scrubbing the preview keeps one GPU submission in flight and coalesces newer requests until onSubmittedWorkDone() settles; a paused preview submits every request for a new picture at once, however many earlier paused submissions are still pending, so a seek onto a new clip is shown immediately even when WebKit settles earlier fences late. A paused request waits and coalesces only when the same picture (composition, timeline time and layer/clip identities) is already in flight, which keeps slider drags bounded; a safety bound of 6 distinct paused targets in flight defers further frames and logs Preview GPU queue saturated. getPreviewGpuBackpressureState(device) reports in-flight submissions, a pending request and the stalled-fence count. A completion fence that does not settle within 1 second releases the queue, renders the newest request, and logs Preview GPU completion did not settle with a running count, so a stalled fence can no longer freeze the preview until a reload. Until a device completed its first preview fence (pipeline and shader compilation at cold start), the window is 5 seconds and a release is logged at info, so the warning always means a stall of a warmed-up device.
• Paused and scrub-settle holds re-present the previous composite only while the current layers show the same clips and sources; a frame whose raster or nested composition is not ready yet on another clip renders empty instead of keeping the previous clip’s picture, and canvas/image rasters that were not collected request follow-up frames.

• A watchdog checks for stalls every 2 seconds.
• If the render loop stops producing frames for about 3 seconds while it should be active, it wakes the loop or restarts it.
• Stall time is measured only while frames are wanted and possible: idle periods, a hidden tab, export, device recovery and a held publication (project switch, editing handed to another tab) advance the baseline instead of counting as a stall.
• When the browser delivers no animation frames although the page reports visible (background or occluded window), the check is paused; a one-shot probe frame tells this apart from a loop that lost its callback, which is restarted with a warning.

• The render loop skips preview rendering while export is active.
• Export and RAM preview both set flags through ExportCanvasManager.
• These modes are separate from the normal preview path, but they share the same engine.

---

RenderDispatcher uses a best-effort chain for video preview:

1. Cached scrubbing frame
2. Copied fallback frame
3. Live external texture import
4. Last known frame or same-clip hold frame

This is why preview can stay stable during seeks while a fresh frame is pending.

• Firefox uses getCopiedHtmlVideoPreviewFrame() for HTML video preview stability.
• That path copies the current video frame into a persistent GPU texture and falls back to the previous stable frame when needed.
• Firefox 157 exposes WebGPU without importExternalTexture/texture_external. supportsExternalTextures(device) (src/engine/texture/externalTextureSupport.ts) detects this and logs one warning. The compositor then creates no external composite/copy pipelines (resources.external is null), TextureManager.importVideoTexture() returns null so collectors use copied textures, and TemporalFrameUploader copies decoded VideoFrames into its transfer texture; Slit Scan skips the external-texture shared-source block and uses the atlas path. Without this, startup raised uncaptured shader, bind-group-layout and pipeline validation errors.
• On Chromium, an import that fails because the video frame has no GPU backing yet (“doesn’t have back resource”, seen with a just-started live camera) is logged as a warning and retried next frame; other import failures stay errors.

---

• The output pipeline uses separate uniform buffers for transparency grid off/on and stacked-alpha export.
• Full-frame output and compositor passes draw one oversized triangle, avoiding a diagonal seam or missing half-frame when a mobile WebGPU driver drops one primitive from a two-triangle quad.
• renderToCanvas() catches canvas-context loss and simply skips that target for the frame.
• Bind-group caches are separate per output mode.

• Stacked alpha doubles the encoded export height.
• RGB is rendered in the top half and alpha grayscale in the bottom half.
• ExportCanvasManager creates the doubled-height OffscreenCanvas when stacked alpha is enabled.

• readPixels() is the CPU fallback path.
• It is used by export fallback paths and preview capture utilities.
• It is slower than the zero-copy export path and should be treated as a fallback, not the normal route.

---

ExportCanvasManager is responsible for export-state flags and the export canvas lifecycle.

• initExportCanvas() creates an OffscreenCanvas with WebGPU context.
• createVideoFrameFromExport() waits for device.queue.onSubmittedWorkDone() before constructing a VideoFrame.
• If the zero-copy path cannot be created, export falls back to readPixels().
• cleanupExportCanvas() clears the canvas and stacked-alpha state after export.

• Zero-copy export only works when OffscreenCanvas + WebGPU + VideoFrame creation are available and the export canvas can be configured.

---

Runtime flags are exposed on window.__ENGINE_FLAGS__.

• useFullWebCodecsPlayback and disableHtmlPreviewFallback are synced with the persisted settings toggle.
• useLiveSlotTrigger swaps slot-grid clicks from editor-open behavior to direct live triggering.
• useWarmSlotDecks prepares reusable slot-owned live decks for faster layer adoption.
• use3DLayers and useGaussianSplat are enabled in this branch.
• workerFirstRenderHost exists for worker-host presentation, but defaults to false; the main-thread render host remains the default.
• timelineCanvasWorker is enabled for eligible timeline rows; it is separate from the main preview render host.
• Motion shape/appearance rendering and the Grid Replicator MVP are always part of the native layer path.

---

• ScrubbingCache keeps up to 192 video scrub frames at 30 fps quantization, with a 192 MiB texture budget and a 960-pixel maximum source dimension.
• Composite RAM preview caches are capped at 900 frames and 512 MB.
• GPU RAM preview cache is capped at 60 frames.
• These are hard cache limits from code.

• useEngine() scales the active-composition base resolution by the persisted preview-quality multiplier before calling engine.setResolution(...).
• Full, Half, and Quarter therefore change the GPU render size for engine-backed preview targets.
• Export output resolution still comes from the composition/export settings, not from preview quality.

---

Problem	What To Check
Black preview after reload	Video GPU surfaces may not be warm yet. The engine suppresses idle until first play, but browser readiness still matters.
Firefox preview instability	HTML video fallback should be using copied textures, not external import.
Export canvas missing	Verify WebGPU context creation on OffscreenCanvas and device validity.
Preview stuck on an old frame until reload	Look for Preview GPU completion did not settle (stalled GPU fence; the queue now reopens after 1 second) and RenderLoop debug lines about deferred requests.
Device lost	The engine attempts recovery and reconfigures canvases, but a manual reload may still be needed.

---

Key implementation files:

• src/engine/WebGPUEngine.ts
• src/engine/render/RenderDispatcher.ts
• src/engine/render/RenderLoop.ts
• src/engine/render/htmlVideoPreviewFallback.ts
• src/engine/motion/MotionRenderer.ts
• src/engine/motion/shaders/motionShapes.wgsl
• src/engine/analysis/ScopeRenderer.ts
• src/engine/native3d/NativeSceneRenderer.ts
• src/engine/pipeline/OutputPipeline.ts
• src/engine/pipeline/SlicePipeline.ts
• src/engine/managers/ExportCanvasManager.ts
• src/engine/texture/ScrubbingCache.ts
• src/engine/core/RenderTargetManager.ts
• src/engine/core/gpuUniformRing.ts
• src/engine/featureFlags.ts
• src/engine/video/VideoFrameManager.ts
• src/services/render/renderHostPort.ts