Stylized Shadow Textures

The problem

Forward-shaded stylized renderers rarely get first-class access to shadow information the way deferred pipelines do. I set out to reconstruct a usable shadow mask post-hoc, then redraw it as a stylized hatch pattern instead of a soft shadow — while staying cheap enough for real-time use.

System overview

A global post-process volume material rebuilds the shadow mask on Unreal’s deferred renderer, reading scene color, base color, and surface normal from the GBuffer:

• Desaturation of both the scene color and base color buffers
• A ratio division between them to isolate occluded regions
• World-space tri-planar projection, driven by the surface normal, using a custom 2×2 rotation matrix pass — rather than Unreal’s built-in WorldAlignedTexture node — so hatch scale and stroke alignment stay stable regardless of surface orientation
• A single-channel grayscale hatching mask, rather than a full texture set, to keep memory bandwidth low

Light direction is tracked via a small event-driven subsystem hooked into Unreal’s native Transform Updated delegate — a minor supporting piece, not the core of the system.

Result

Benchmarked on an RTX 3070 at 1440p (2560×1440) using Unreal’s deferred renderer: 3.62 ms baseline frame time with the shader off, 3.43 ms with it on — the difference is normal profiling volatility, not a real gain. The isolated cost of the post-process pass itself, measured via the RDG visualizer, is 0.06 ms — and scales with resolution: roughly under 0.04 ms at 1080p, up to about 0.14 ms projected at 4K.

Benchmarking was desktop-only. The GBuffer access this system depends on is fundamentally incompatible with mobile forward shading, so no comparable mobile baseline exists.

Technical contributions

• Mathematical decoupling of world-space coordinate matrices — a custom 2×2 rotation matrix pass, replacing Unreal’s built-in WorldAlignedTexture node, keeps hatch scale and stroke alignment stable and drift-free regardless of surface orientation.
• Low-overhead, event-driven subsystem architecture — tracking anchored to Unreal’s native Transform Updated delegate rather than Tick, measured at 0.0 ms CPU overhead during static frames.
• Empirical verification of the “Post-Process Gap” — stress-testing on mobile hardware produced concrete evidence of where post-render buffer sampling breaks down on tile-based architectures, documented below.

Earlier approach: forward shading for mobile

An earlier version targeted forward-shaded mobile platforms, where the GBuffer isn’t available after the tile-resolve step on TBDR hardware. A custom C++ albedo-caching subsystem (UStylizedShadowsSubsystem, a UGameInstanceSubsystem) used a SceneCapture2D to manually extract and preserve scene albedo before tile resolution — with a directional-light transform delegate and a 30 fps timer feeding camera and light data into a Material Parameter Collection.

It didn’t work: the mobile GPU still cleared the custom render target caches during its own optimization sweeps, so texture lookups against those manually cached targets returned nothing. Combined with the extra draw-call cost of a second visibility pass on low-power hardware, this ruled out the mobile forward-shading approach entirely — the current system targets desktop deferred shading only.

Cross-engine implementation