How Wasm Simplifies Streaming Data Pipelines

Recorded August 17, 2023

View a Complimentary Live Webinar Hosted by Redpanda

Previously titled: Extending High-Performance Systems With WebAssembly

Today stream processing generally involves connecting disjunct distributed systems, such as a storage system and a stream-processing engine. However, 60-70% of use cases can be solved with WebAssembly (Wasm) powered functions built directly into a streaming storage system. This eliminates the data ping-pong, decreasing system latency as well as the resources needed. In this talk we'll dive into the challenges we encountered integrating a Wasm runtime into Redpanda, a distributed high-performance streaming storage engine. Redpanda is built with the Seastar framework, which has an execution environment with strict bounds on memory consumption and latency, posing unique challenges for supporting arbitrary user functions during both compilation and execution. We'll also detail our developer experience that encourages ease of use without compromising on performance in this strict environment.