Tweet by jsneedles
November 10, 2025
Another cool aspect of this architecture: We tolerate failures super well. Obviously the ingest layer is fallible, but also the simplest. Once we have a file on R2, we can re-ingest it, re-queue it, etc. Also, if for whatever reason we have errors in parsing, a separate pipeline exists (that basically mirrors this one) just to track errors with the raw payload and some client info. Thankfully it’s very dormant! Anytime we fail to ingest a file into CH that goes through a large number of retries (with back off) and eventually can land in a DLQ. This can be replayed through the original pipeline. We also (try to) insert info about that failure into a dedicated CH table for error tracking.
- Author
- jsneedles
- Date
- November 10, 2025
- Canonical URL
- /tweets/jsneedles-1987870046358581619-35db02