Learning from an unbounded stream means updating weights at each observation, never revisiting the past. Real-Time Recurrent Learning (RTRL) is exact but costs quartic time and cubic memory in a dense network’s width; cheap successors add uncontrolled variance, discard cross-unit credit without an error bound, or abandon dense recurrence. We introduce Sketched RTRL (SK-RTRL), keeping the influence matrix as an exact block-diagonal part plus a deterministic low-rank sketch of the discarded residual, one rank knob spanning from that part alone to exact RTRL. The same shrinkage yields, at no extra asymptotic cost, a running scalar certificate bounding the gradient bias at every step, to our knowledge the first such bound for an RTRL approximation. Truncation records its discarded mass, so the rank resizes online without breaking the bound. The sketch holds a consistent gradient-cosine advantage over four approximate estimators on all nine tasks and three ranks tested, 108 of 108 paired comparisons, and cuts online training memory by 13.5 times at width 512. Task-level benefit varies: higher fidelity need not lower prediction error, there is no consistent advantage over tuned truncated BPTT, and rotation memory is an outright loss. The certificate holds in exact arithmetic and on every counted fp32 step, but is informative only at the clipped operating points measured, essentially vacuous without clipping (exactly zero on the chaotic and real-world tasks, 0.092 on copy) and on both gated cells. The low-rank premise holds for some task groups, not others; training-level evidence is on tanh units, the gated results are structural.
