Tweet by marbemac
June 3, 2025
Type-safe resumable streams and multi-user chat, courtesy of @Cloudflare durable objects, @zero__ms, and @orpc - how does it work? Streaming every token through pg / zero is not a great option at scale, so we're using SSE and cloudflare durable objects to handle the token streaming, and pg / zero for the durable persistence once streaming is complete. 1. to kick off generating a reply, we insert a new message into postgres with status "streaming" and empty content 2. this new empty message is sync'd to the client via Zero 3. on the client, when rendering any message that has status "streaming", open an SSE connection via ORPC to a durable object instance representing the chat thread 4. we're opening an SSE connection per "streaming" message, not per thread. multiple messages can be streaming at the same time, within the same thread 5. token stream parts are kept in memory in the durable object, and streamed to connected clients over SSE 6. if a new client connects (or old client re-connects), it is immediately caught up to the present by streaming down the in memory parts on connect 7. on llm stream complete, update the message in pg with the final message parts, set status to "completed", and close sse connections 8. the message part and status change is sync'd to client via zero - client now uses those parts to render the message instead of the previously streamed parts Thus there are only two postgres operations per message - initial insert, update on streaming complete. This should hopefully scale to the moon.
- Author
- marbemac
- Date
- June 3, 2025
- Canonical URL
- /tweets/marbemac-1929922350549479564-83831e