Animated Architecture Patterns

Pipes and Filters

Split processing into independent stages connected by channels.

Also known as Pipeline architecture, Processing pipeline

Uploads 20 photos/s Catalogue product pages CDN image files Photo service one deployable; process(photo) runs every stage × 4 copies, for 4 resizers slow: 5/s in-process calls validate resize watermark tag publish tag fails: retry runs it all again 2× copies runneeded 41 44 41 41 41 copies of each stageat 20 photos/s Validatecheck file 1 × 50/s depth 0 6 12 18 24 30 1 Resizemake sizes ×4 1 × 5/s4 × 5 = 20/s depth 0 1 Watermark add logo ✕ crashed restarted 1 × 25/s 2× depth 0 1 Tagadd labels 1 × 40/s depth 0 1 Publishgo live 1 × 50/s depth 0 Safetyscan image NEW 1 × 30/s video pipeline Videos Transcode Tagsame filter Package dead-letter queue depth 01 try 1 try 2 try 3 ONE SERVICE, EVERY STAGE Pace. Every photo waits for resizing, the sloweststage at 5 photos a second. Scale. 20 photos/s need 4 resizers, so you run4 copies of the whole service. Change. A new watermark means testing andredeploying all of it. Failure. When tagging fails, the retry starts overand the costly resize runs again. FILTERS JOINED BY PIPES Filter: one job. It reads its input pipe, writesits output pipe and shares nothing else. Pipe: a queue between two filters. It holdseach photo until the next filter takes it. Source and sinks: uploads feed the pipeline;the catalogue and the CDN receive its output. On its own: each filter is built, deployed andscaled alone; only the message format is shared. ONE STAGE AT A TIMEunder each filter: instances × photos per second each Backs up. 20 photos arrive each second and oneResize does 5, so its pipe grows by 15 a second. Scale one. Resize alone goes to 4 instances: 4 × 5 = 20a second, and its pipe stops growing. The rest stay at 1. Insert one. Safety goes in between Validate andResize. No other filter changes. Reuse one. The same Tag filter also runs in asecond pipeline, for videos. WHEN A FILTER FAILS Watermark crashed holding 7f3. It went back to its pipeand only Watermark ran again, so make filters idempotent. 9c1 failed at Publish 3 times, so the brokermoved it to the dead-letter queue. A correlation ID travels with each photo and islogged at every hop, so one search finds them all. WHAT THE PIPES COST • Every hop serialises the photo and stores it again.• 6 filters and 5 pipes to deploy, scale and watch.• Each hop adds queueing and network latency.• No transaction spans the stages: 9c1 was resized, watermarked and tagged, but never published. 7f3 9c1

Step 1 · One big processing step. A single photo service validates, resizes, watermarks, tags and publishes each upload in one function. Resizing is the slowest stage (5 photos a second per copy), so taking 20 a second means running 4 copies of the whole service just to get 4 resizers, though one copy of every other stage would do. A new watermark means redeploying all of it, and when tagging fails the retry starts from the top and resizes the photo again.

Step 2 · Split into filters. Each stage becomes a filter: a small, separately deployed service with one job that reads from its input pipe (here a queue) and writes its result to its output pipe. Filters share nothing else, no memory, no database and no calls to each other, so the photo flows Validate → Resize → Watermark → Tag → Publish with one pipe between each pair, from the uploads (the source) to the catalogue and the CDN (the sinks).

Step 3 · Scale and change one stage. With 20 photos a second arriving and one Resize doing 5, Resize's input pipe grows by 15 a second, so Resize alone scales out to 4 instances (4 × 5 = 20 a second) and the pipe stops growing; every other filter stays at one instance. A content-safety filter is inserted between Validate and Resize: Validate now writes to a new pipe and Safety writes into Resize's existing one, with no change to any other filter. The same Tag filter is reused in a second pipeline, for videos.

Step 4 · Failures and the price. Watermark crashes while holding photo 7f3; the message was never acknowledged, so it reappears in its pipe and a restarted instance watermarks it again, without redoing the resize (at-least-once delivery, so filters must be idempotent). Photo 9c1 fails at Publish on every try and is moved to a dead-letter queue after 3, while a correlation ID in every message ties each photo's hops together (distributed tracing). The price: the data is serialised at every hop, there are more parts to run and watch, latency adds up over the hops, and no transaction spans the stages, so 9c1 was resized, watermarked and tagged but never published.

The problem

An online shop receives product photos at 20 a second. Each photo has to be checked, resized into the sizes the site uses, watermarked, tagged and published to the product catalogue and the CDN. The first version is usually one service with one function that runs the five steps in turn.

That works until the steps start to pull in different directions:

  • The slowest step sets the pace. Resizing is far heavier than the rest. It manages 5 photos a second, which caps each copy of the service, and every photo waits for it.
  • Everything scales together. Four resizers means four copies of the whole service, including four copies of validation, watermarking, tagging and publishing, where one of each would have been enough.
  • Everything ships together. A new watermark is a build, test and redeploy of the entire service, and a mistake in it can take every other step down with it.
  • Everything fails together. When tagging throws an error, the retry starts from the top, so the expensive resize runs again.
  • Nothing can be reused. A video pipeline that also needs tagging ends up with a copy of the tagging code.

How it works

Split the processing into a chain of filters joined by pipes:

  • A filter does one job. It takes an item from its input pipe, transforms it and writes the result to its output pipe. It doesn't know which filters come before or after it, only the format of what it reads and writes.
  • A pipe carries items from one filter to the next and holds them in between, so neighbours can work at different speeds and don't have to be running at the same moment.
  • A source feeds the first pipe (the uploads in the animation) and a sink receives what the last filter produces (the catalogue and the CDN).

Filters share nothing but pipes, so each one can be deployed, scaled, replaced or reused on its own, and you change the pipeline by rewiring pipes rather than by editing filters. In step 3 a content-safety filter goes in between Validate and Resize: Validate writes to a new pipe, Safety reads from it and writes into the pipe that Resize already reads, and no other filter notices.

Where it comes from

In a Bell Labs memo dated 11 October 1964, Doug McIlroy asked for a way to connect programs the way you connect lengths of garden hose, screwing in another piece whenever the data needs different treatment. According to Dennis Ritchie's history of Unix, pipes were added to the system in 1972 at McIlroy's urging. With them came the habit of small tools that read standard input and write standard output, so that the shell can chain them: cut -d, -f3 orders.csv | sort | uniq -c.

Pattern-Oriented Software Architecture, Volume 1 (Buschmann, Meunier, Rohnert, Sommerlad and Stal, 1996) catalogues Pipes and Filters as an architectural pattern for stream-processing systems, and supplies most of the vocabulary used here:

  • Push or pull. In a push pipeline the upstream side hands each item on; in a pull pipeline the downstream side asks for the next item when it is ready.
  • Active or passive filters. An active filter runs in its own process or thread and moves the data itself, pulling from its input and pushing to its output. A passive filter is called by a neighbour, which either pushes an item into it or pulls a result out of it.

Enterprise Integration Patterns (Gregor Hohpe and Bobby Woolf, 2003) applies the pattern to messaging. Its filters are message processors that receive on an inbound channel and publish to an outbound one, and its pipes are message channels. A queue between active filters, as in the animation, mixes push and pull: each filter pushes its output into a queue, and the next filter pulls from that queue at its own pace.

How it relates to other patterns

  • Competing Consumers is how a single filter scales out, and Queue-Based Load Leveling is what each pipe does for the filter behind it.
  • Medallion Architecture is the same idea in a lakehouse: each layer is the output of a refining step, kept as tables you can query and replay rather than as messages in flight.
  • Event-Driven Architecture uses the same brokers, but an event fans out to every subscriber that cares and nobody owns the sequence. A pipeline is a deliberate chain in which each item has one next step.
  • Saga (Orchestration) coordinates a business transaction across services and undoes completed steps when a later one fails. A pipeline transforms data and normally only moves forward.

When to use it

  • The work splits into independent steps that differ in cost, speed or resources: a CPU-heavy resize, a GPU for image tagging, a quick validation.
  • Steps change at different rates, are owned by different teams, or are worth reusing in other pipelines.
  • You expect to add, remove or reorder steps.
  • A high-volume stream where a slow or failing step must not hold up the others, and processing in the background is acceptable.

Don't use it when:

  • The steps are short, fast and tightly coupled. One function does them more simply, and every hop would add serialisation, latency and moving parts.
  • The steps must succeed or fail together in one transaction.
  • A caller is waiting for the result within the same request. A pipeline of queues is asynchronous.
  • Every step needs so much shared context that passing it along costs more than the split saves.

Trade-offs

  • More moving parts. Six filters and five pipes to deploy, configure, secure and monitor, instead of one service.
  • Latency per item. Each hop adds serialisation, a network round trip and time in a queue. A pipeline trades latency for throughput and isolation.
  • Serialisation and storage. Every hop encodes the data and the broker stores it again. Large payloads multiply that cost unless you pass references.
  • Duplicates. Redelivery after a crash means at-least-once processing, so every filter has to be idempotent.
  • No transaction across stages. When a later stage fails, the earlier results already exist, and you need a plan for them.
  • Harder to follow. One photo's story is spread over many services and logs. Correlation IDs and per-stage dashboards are part of the price.
  • Contracts between filters. Each message format is an interface. Changing one takes versioning and a careful rollout, just like an API.

Implementation notes

Designing filters

  • One job. If describing a filter takes the word and, consider two filters. Don't split too finely either: steps that are cheap, always change together or must scale together can share a filter. Azure calls grouping them the Compute Resource Consolidation pattern.
  • Stateless where possible. Keep what a filter needs in the message, or in storage the message points to, and not in memory between messages. Then any instance can take any message, and a crashed instance loses only the item it was holding.
  • Explicit input and output contracts. Give each pipe one documented message format with a version. A filter should pass on, unchanged, the fields it doesn't use, so a field added for a later filter doesn't break an earlier one.
  • Idempotent. Assume every message can arrive twice, as 7f3 does in step 4. Name outputs deterministically (7f3/800w.jpg, not a random file name) so a repeat overwrites its own result, and record message IDs where a side effect can't simply happen again, such as an email or a payment. See Idempotent Consumer.

What a pipe is in practice

Pipe Examples Good for Watch out for
In-process stream or channel Unix pipes, Go channels, Java streams, Reactive Streams libraries Cheap hops, no serialisation, built-in backpressure Every filter lives and dies with one process
Queue Amazon SQS, Azure Service Bus, Azure Queue Storage, RabbitMQ Separate services; each message goes to one consumer and comes back if it isn't acknowledged At-least-once delivery; usually no replay
Topic or log Apache Kafka, Amazon Kinesis Data Streams, Azure Event Hubs, Google Cloud Pub/Sub Durable streams that several pipelines can read and replay Ordering and parallelism tied to partitions

Frameworks build pipelines out of these parts:

  • Kafka Streams builds a processor topology: a graph of stream processors connected by streams, where source processors read from Kafka topics and sink processors write to them. The DSL provides common operations (map, filter, join, aggregations) and the Processor API takes custom processors. Processing is at-least-once by default (processing.guarantee). With exactly_once_v2, input offsets, state store updates and output topics are committed together, which covers what stays inside Kafka but not a call to an outside service.
  • Apache Beam describes a pipeline of PTransforms that read and write PCollections, and a runner such as Dataflow, Flink or Spark executes it. The pipes can be logical only: Dataflow fuses consecutive steps to avoid the cost of handing data between them, which can also reduce parallelism.
  • AWS Step Functions runs an orchestrated pipeline: each state's output becomes the next state's input, up to 256 KiB of it. Standard workflows execute each step once and can run for up to a year; Express workflows are at-least-once and run for at most five minutes.
  • Serverless functions joined by queues are a common cloud form, and the one in Azure's own image example: each filter is a function triggered by its input queue that writes to the next. When AWS Lambda reads from Amazon SQS, messages arrive in batches, so use partial batch responses to report the failed messages instead of failing the whole batch and repeating the ones that succeeded. See Serverless.

Scaling each filter, backpressure and buffers

Scale each filter on its own signals: the depth of its input pipe and the age of the oldest message in it. Within one filter, several instances read the same pipe as competing consumers, and an autoscaler adds and removes them (Autoscaling). In the animation Resize needs 20 ÷ 5 = 4 instances and nothing else changes. Matching the arrival rate only stops a backlog from growing; to work one off, run above the arrival rate for a while.

A pipeline moves no faster than its slowest filter, and the pipes decide how that shows. An in-process pipe has a fixed buffer: a Linux pipe holds 16 pages by default (64 KiB with the usual 4 KiB pages), and a writer that fills it blocks until the reader catches up, so the whole chain slows to the pace of its slowest command. Reactive Streams standardise the same idea across threads, so the receiving side is never forced to buffer more than it can take. A broker queue is effectively unbounded, so a slow filter doesn't slow its producer; the backlog grows instead. That is load leveling, and it only moves load in time. Give each pipe limits (length, size or message age), alert on its depth and age, and decide in advance what happens when it is full: reject at the source, shed load, or scale the filter.

Ordering

With one instance per filter and first-in, first-out pipes, items leave in the order they arrived. Once a filter has several instances, a quick item can overtake a slow one. If order matters only per key, such as all the photos of one product, keep each key's items in sequence. Kafka writes records with the same key to the same partition and a consumer reads each partition in order, and an Amazon SQS FIFO queue processes the messages of one message group one at a time, in order. Parallelism then comes from the number of keys or groups rather than from instances. Competing Consumers covers the options in more detail.

Large payloads: pass a reference

Don't push a 12 MB photo through every pipe. Store it once in object storage and put only its location and metadata in the message, which is the Claim Check pattern and exactly what Azure's image-processing example does. Brokers and orchestrators push you this way anyway: an Amazon SQS message can be at most 1 MiB (the Extended Client Library for Java and Python stores larger payloads in Amazon S3, up to 2 GB), and a Step Functions state passes at most 256 KiB. Each filter then writes its result as a new object and passes the new reference on.

Errors, retries and poison messages

  • Transient errors such as a timeout or a throttled API: retry inside the filter with backoff and jitter before giving up on the message.
  • Crashes: a message that was never acknowledged comes back on its own once its lease runs out (the visibility timeout in Amazon SQS, 30 seconds by default), and another instance processes it. That is what happens to 7f3 in step 4.
  • Poison messages: a message that fails every time would otherwise come back forever. Brokers count deliveries and park the message in a dead-letter queue after a limit: maxReceiveCount in an SQS redrive policy, or MaxDeliveryCount in Azure Service Bus (10 by default). Give each pipe its own dead-letter queue so you know which stage failed, alert on its depth, and plan how to redrive the messages once the cause is fixed.
  • Expected rejections are not errors. A file that fails validation belongs on a channel for invalid input, with the reason attached (Enterprise Integration Patterns calls it an Invalid Message Channel), not in a dead-letter queue that is there for bugs and outages.

Observability

  • Per stage: input pipe depth, age of the oldest message, processing rate, error rate, dead-letter depth and instance count. Depth and age show where the bottleneck is; comparing the rates in and out shows whether it is getting worse.
  • End to end: put a correlation ID in every message, pass it on unchanged and log it in every filter, so one search shows a photo's whole journey. Better still, propagate a W3C Trace Context traceparent and record a span per filter; see Distributed Tracing. Measure the time from upload to publish as well, because every stage can look healthy while the total is too slow.

Partial completion and compensation

No transaction spans the filters. When a late filter fails for good, the earlier ones have already done their work: in step 4, photo 9c1 has resized and watermarked files and tags, but no catalogue entry. Decide for each pipeline what that means:

  • Leave it when the leftovers are harmless, and clean them up later, for example with a storage lifecycle rule.
  • Finish it by redriving the message once the cause is fixed. Idempotent filters make that safe.
  • Undo it when an earlier step changed something that matters, such as stock or money, with a compensating transaction. Azure's guidance suggests pairing Pipes and Filters with compensating transactions as an alternative to a distributed transaction. When the steps form a business transaction rather than a transformation, coordinate them as a saga.

Routing between filters

A fixed chain is the simplest pipeline. When the next step depends on the item, for example videos to Transcode and photos to Resize, put a content-based router between the filters: it reads each message and sends it to the right pipe. When each item needs its own list of steps, a routing slip attached to the message names them in order, and the router at each step forwards the message to the next one on the list. Keep these decisions in routers or in the message rather than inside the filters, so the filters stay reusable.