Animated Architecture Patterns

Medallion Architecture

Bronze, silver, gold: raw data is refined in layers inside a lakehouse.

Also known as Multi-hop architecture

Lakehouseobject storage + open table format (Delta Lake, Iceberg, Hudi) SourcesConsumers Bronzeraw, as sent Silvervalidated Goldbusiness-ready job job orders_raw 1,000 12 dupes · 3 malformed partner_raw 8,300 CSV rows, as sent clicks_raw 52,400 JSON events, as sent orders 985 typed · deduped · USD amount_usd wrong · v12 rebuilt · v13 customers 8,300 ISO country codes clicks 52,400 typed events daily_revenue 5 rows · by region customer_features 8,300 rows · by customer product_summary 140 rows · by product Orders DBCDC events Partnerdaily CSV Clickstreamevent stream BI dashboardrevenue by region ML modelreads features Data APIproduct summaries FX bug fixed appended unchanged: nothing fixed, nothing dropped+ ingestion metadata_source_loaded_at_batch_id silver job: reads only new rows, parses and types fieldscountries → ISO codes · amounts → USD · joins customers 12 duplicates dropped 3 malformed → quarantine gold job: aggregates and models silver for each audiencebusiness names · fast to read · access per audience fixed job replays bronze into silver, then gold time travel: compare orders v12 (buggy) with v13 (fixed) bronze still has all 1,000 raw rows orders_quarantine empty 3 rows raw value_reason amount: "N/A"not a numberts: "2026-13-01"invalid date{"order_id": 48truncated JSON event record duplicate malformed replayed (fixed) counts: today's load Lineagewhich tables feed which orders_raw partner_raw clicks_raw orders customers clicks daily_revenue customer_features product_summary BI dashboard ML model Data API impact: 2 gold tables, 2 consumers fixed and rebuilt from bronze

Step 1 · Bronze: land as-is. Change events from the orders database, the clickstream and a partner's daily CSV file are appended unchanged to bronze tables, together with ingestion metadata (source, load time, batch ID). Today that is 1,000 order records, 12 duplicates and 3 malformed records included: bronze fixes nothing and throws nothing away, so it is the complete, replayable history of what the sources sent.

Step 2 · Silver: clean and conform. A silver job reads only the new bronze records, parses and types the fields, removes the 12 duplicates by order ID, converts country names to ISO codes and amounts to one currency, and joins customer data. The 3 malformed records move to a quarantine table with the reason, so silver holds 985 validated records (1,000 − 12 − 3): the trusted, record-level view that many uses share.

Step 3 · Gold: shape for use. A gold job aggregates and models silver into business-level tables for each audience: daily revenue by region for the BI dashboard, a customer features table for the ML model and a product summary (from the clickstream) for the data API. Gold tables are named in business terms, optimised for reading and governed per audience.

Step 4 · Fix and replay. A bug is found in the currency conversion. The silver job is fixed and replayed from bronze, which still holds every raw record. Lineage shows that only the orders path is affected (two gold tables, the dashboard and the model), and the table format's time travel lets the team compare the old and new versions. Day to day, every hop processes only new data, by streaming or incremental merges.

The problem

Analytics data arrives from many places: change events from operational databases, clickstreams, files that partners drop once a day. If every report and model reads those feeds directly, each team writes its own parsing, deduplication and business rules, and their numbers disagree. If the data is cleaned on the way in and only the cleaned copy is kept, each rule is applied once and for good: when a transformation turns out to be wrong (a bad exchange rate, a parser that mangles dates), the original records are gone and the history can't be rebuilt. Rejected records vanish without a trace, and nobody can say which dashboards were built from which inputs.

How it works

The medallion architecture, sometimes called a multi-hop architecture, organises a lakehouse into layers of increasing quality. Each hop is a job that reads one layer and writes the next, and each layer makes one promise:

Layer Holds Promises Read by
Bronze (raw) Every record exactly as the source sent it, appended, plus ingestion metadata A complete, replayable history: nothing fixed, nothing removed The pipelines that build silver; engineers and auditors
Silver (validated) Typed, deduplicated, conformed records; rejects set aside in quarantine At least one validated, non-aggregated version of every record Engineers, analysts, data scientists
Gold (business-ready) Aggregates, dimensional models and feature tables, named in business terms Shaped and governed for one audience, fast to read Dashboards, ML models, applications
  • Bronze keeps each source's data in its original form and only grows. Databricks recommends minimal validation here, with most fields stored as strings, VARIANT or binary so that an unexpected schema change can't lose data, and treats bronze as input for pipelines rather than something analysts query. Microsoft Fabric also keeps bronze in the source's format where possible, and suggests a shortcut instead of a copy when the files already sit in OneLake, ADLS Gen2, Amazon S3 or Google Cloud Storage.
  • Silver is where records are cleaned and conformed: schema enforcement, type casting, handling of nulls, deduplication, late and out-of-order records, quality checks and joins. Databricks advises against loading silver straight from the sources, because then a schema change or a corrupt record breaks the load instead of simply landing in bronze.
  • Gold holds the data products: dimensional models, aggregates and feature tables for specific consumers. Because gold mirrors business domains, some organisations run several gold layers, for example one each for finance, HR and IT.

All three layers live in one lakehouse: tables kept as open files (usually Parquet) in object storage, with an open table format such as Delta Lake, Apache Iceberg or Apache Hudi keeping a transaction log beside them. That log is what makes the hops safe. Commits are atomic, so a reader never sees half of a job's output. Writes are checked against the table's schema: Delta Lake rejects a write whose columns or types don't match, and adds new columns only when you allow the schema to evolve. And every commit creates a new table version, so in Delta Lake and Iceberg you can query a table as it was at an earlier version (time travel). Armbrust, Ghodsi, Xin and Zaharia argued for this design, warehouse-style management on cheap open storage, in the CIDR 2021 paper that set out the lakehouse architecture.

When to use it

  • Several sources (database change feeds, event streams, partner files) feed several kinds of consumer: BI, machine learning, data APIs.
  • You must be able to rebuild derived data after a bug or a rule change, or to show later exactly what a source sent.
  • Batch and streaming data meet on one platform, and different teams own different stages.

When not to use it:

  • Small data that fits comfortably in one database: a few views over the raw tables are enough.
  • A single warehouse where staged models already separate raw, cleaned and business-ready data. dbt's layering (below) is the same idea without a lake.
  • Serving an application's own screens with low latency. That's a read model inside the application (see CQRS), not an analytics pipeline.

Trade-offs

  • Copies and hops cost money. The same records are stored in bronze, in silver and often again in gold, and every hop is a job to run, monitor and pay for.
  • Every hop adds latency, unless the hops run as streams.
  • The names are a convention, not a guarantee. "Silver" means whatever your checks enforce. Without explicit expectations and owners, the layers drift into three copies of the same mess.
  • Personal data multiplies. Each copy falls under the same retention rules and deletion requests (see Governance below).
  • It's a recommendation, not a law. Databricks calls the pattern a best practice but not a requirement; pushing every dataset through three layers by rule can be pure ceremony.

Implementation notes

Ingestion

  • Land batch files, streams and change data capture feeds in bronze, and record the source, load time and batch or file name with every row.
  • Make loads idempotent and replayable, in the spirit of an idempotent consumer: a retried batch must not append the same rows twice. Delta Lake, for instance, ignores a write that repeats an application ID and transaction version it has already committed.
  • Expect late and out-of-order data. Resolve it in silver (merge by key, event-time windows), never by editing bronze.

Data quality

  • Declare checks as expectations. In Databricks' Lakeflow pipelines (formerly Delta Live Tables) each expectation either keeps and counts the bad records, drops them or fails the update. Fabric's materialized lake views carry data quality rules too.
  • Route rejects to a quarantine table with the reason and batch ID instead of dropping them. It is the dead-letter queue of a data pipeline, and someone must own reviewing and replaying it.
  • Agree data contracts with source teams (schema, meaning, freshness), so that a breaking change is caught at the boundary and not in a dashboard.

Deduplication, merges and history

  • Deduplicate by business key with a merge: Delta Lake's documentation shows an insert-only MERGE that skips records already in the table.
  • Apply CDC feeds to silver as upserts and deletes, again with MERGE.
  • Keep history where consumers need it. A type 2 slowly changing dimension keeps every version of, say, a customer's address with the dates it was valid, and is written with MERGE as well.

Modelling gold

  • Star schemas (facts and dimensions) for BI, pre-aggregated tables for hot queries (often a materialized view), and feature tables with one row per entity for ML.
  • Document each table's grain ("one row per region per day") and name it in business terms.

Incremental processing and backfills

  • Day to day, move only what's new: streaming reads from append-only bronze tables, Delta Lake's change data feed (row-level changes between table versions) or Iceberg's incremental reads of the data appended between two snapshots.
  • A backfill is a replay. Fix the job, rebuild the affected silver and gold tables from bronze, and compare the old and new versions with time travel before consumers switch over.

Governance

  • Register every table in a catalog and capture lineage. Unity Catalog, for example, records lineage automatically down to the column level, and Fabric shows lineage across layers for materialized lake views. With lineage, "what did this bug touch?" becomes a lookup instead of an investigation.
  • Grant access per layer: bronze to pipelines and engineers, silver to analysts and data scientists, gold per audience. Microsoft recommends a separate Fabric workspace for each layer for this reason.
  • Personal data: mask or tokenise it in silver and gold. A deletion request has to reach bronze too, and deleted rows survive in older table versions until those are cleaned up: Delta Lake removes old data files only when VACUUM runs (by default it keeps files deleted in the last seven days), and Iceberg when snapshots are expired.

Cost

  • Three layers can mean three copies. Keep bronze compressed, and retain it for as long as you may need to replay, not forever by default.
  • Compact the small files that streaming hops produce (Delta Lake's OPTIMIZE, Iceberg's rewriteDataFiles) and expire old versions on a schedule. Fabric notes that small files are acceptable in bronze, while gold, where most queries run, benefits from larger ones.

Variations

  • More or fewer layers: a landing zone before bronze, several gold layers for different domains, or silver and gold merged for a small dataset.
  • The same idea in a warehouse: dbt structures projects as staging (cleaned-up building blocks made straight from the sources), intermediate (purpose-built transformation steps) and marts (the business entities that consumers query).

How it relates to other patterns

  • A gold aggregate is often a materialized view; the medallion architecture is the layered pipeline around such tables.
  • CQRS read models answer one application's queries; gold tables serve analytics for many teams.
  • Event sourcing keeps an application's events as its source of truth and rebuilds state by replaying them; bronze plays the same role for analytics.
  • Each hop is a filter in a pipes and filters pipeline whose pipes are durable tables.