Skip to main content

What Is Data Integration Architecture? Layers and Patterns

Cover reading Data Integration Architecture: Patterns, Layers and Diagrams, beside six stacked layers, sources to governance.
Written byPrincipal Data/AI Architect
Technically reviewed byMuhammad Taha Senior AI Architect
Published
Reading time
12 min

Summarize with AI

Introduction

On November 12, 2026, Microsoft told partners it is retiring Purview Data Governance, with support ending on November 12, 2027. The retirement covers the Unified Catalog, Data Map and Workflows. Microsoft now points governance work to the OneLake catalog in Microsoft Fabric, so teams that built their governance around Purview have a year to move it.

So, how does a data integration architecture survive a change like this? It survives when each layer has one job and a clean boundary, so one tool can change without a rebuild. This article explains the layers of a data integration architecture, the six patterns teams build on, and reference diagrams for three company sizes.

What Is Data Integration Architecture?

Data integration architecture is the plan for how data moves from source systems, through ingestion and transformation, into storage where reports, applications and AI models can use it. The plan names each layer, the pattern that connects the layers, and the tools that run each step. A good plan lets a team replace one tool without rebuilding the others.

Most businesses need this plan once their tools stop agreeing. A CRM, an ERP and an ad platform each count customers in their own way, which creates data silos. The architecture decides where those counts get reconciled.

The Layers of a Data Integration Architecture

The architecture has six layers, and each one does a single job. Data enters through the source and ingestion layers. It becomes usable in the storage and transformation layers, and the serving layer hands it to people and systems. Orchestration, monitoring and governance run across all of them.

Many teams treat these six layers as their data integration architecture framework, since the layers stay fixed while the tools inside them change.

Source Layer

The source layer is every system that creates data the business wants to analyze. Typical sources are the CRM, ERP, billing tool, product database, ad platforms and spreadsheets. Each source exposes data through an API, a database connection or a file export.

Each source needs a named owner who knows when its fields change. SaaS sources are usually reached through API integration, and partners who ask for direct database access are better served by a governed API or a file drop.

Ingestion Layer

The ingestion layer copies data out of the sources and into storage, either on a schedule or as changes happen. Managed tools such as Fivetran and Airbyte cover common SaaS sources with ready-made connectors. Custom code covers the rest. Some teams call ingestion plus transformation the data integration layer.

For databases, change data capture (CDC) reads the transaction log instead of querying whole tables again. Debezium, an open-source CDC tool, records every row-level change as an event stream, in the order the changes happened.

Proprietary sources need custom pipeline development, and each pipeline follows its own data pipeline architecture.

Storage Layer: Data Warehouse and Data Lake Integration

The storage layer holds the integrated data, usually in a warehouse, a lake or a lakehouse. A warehouse such as Snowflake or BigQuery stores structured tables built for SQL. A lake stores raw files of any type at low cost. A lakehouse, such as Databricks, adds warehouse-style tables on top of lake storage.

Data warehouse integration suits teams whose questions center on sales, finance and operations. Data lake integration suits teams with logs, documents or event data, a stack the data lake guide covers.

Many lakehouse teams split storage into medallion tiers, which run from raw bronze copies to business-ready gold tables. Integration data that feeds other systems can share the raw tier with analytics. Once operations depend on the feed, the integration data needs its own cleaned tables, so an analytics change can’t break it.

Transformation Layer

The transformation layer turns raw copies into clean, joined tables that match how the business talks. It renames fields, removes duplicates, applies business rules and builds the models reports read from. Most modern teams write these steps as SQL in dbt, which runs inside the warehouse.

A semantic layer on top of the models defines each metric once, such as revenue, so every dashboard and AI tool calculates it the same way.

Serving Layer

The serving layer delivers integrated data to the people and systems that use it. Dashboards in Power BI, Tableau or Looker are the usual consumers. Reverse ETL tools push modeled data back into the CRM and support tools, so frontline teams see it where they work.

Orchestration, Monitoring and Governance

Orchestration, monitoring and governance run across every other layer instead of sitting in one place. An orchestrator such as Airflow or Dagster runs each job in the right order. Monitoring checks freshness, row counts and failed loads. Governance records who owns each dataset, who may see it and where it came from.

Governance is also the layer most tied to one vendor, as the Purview retirement shows. Keeping ownership and access rules as part of wider data management turns the move into a migration instead of a rebuild.

Data Integration Architecture Patterns

Data integration architecture patterns describe how the layers connect, and six of them cover almost every stack. Each pattern trades build effort against flexibility and cost. The patterns sit one level above the integration techniques, such as ETL and CDC, that actually move the data.

The table compares the six patterns by fit, main cost driver and the point where each stops fitting.

Pattern

Best Fit

Main Cost Driver

Relative Cost

Stops Fitting When

Point to Point

A few systems sharing data directly

Engineer time per new connection

Low, then rising

Nobody can track every connection

Hub and Spoke

Many sources feeding shared reports

The hub platform and its team

Medium

One team becomes the bottleneck

Warehouse-Centric ELT

Analytics on cloud data

Warehouse compute

Low to medium

Data must act within seconds

Event-Driven and Streaming

Decisions made within seconds

Always-on clusters and scarce skills

High

Daily reports are enough

Data Virtualization

Querying data where it lives

Query load on source systems

Low to medium

Sources are slow or busy

Data Mesh and Data Fabric

Business units with their own data teams

Platform team and governance overhead

High

One central team still copes

Cost rises with the freshness and independence a pattern promises, so the cheapest pattern that answers the business questions usually wins.

Point to Point

Point to point connects each system directly to every system that needs its data. It is the fastest way to start and works while a company runs a few sources. Every new system adds connections to the others, so maintenance grows faster than the stack. Most teams only notice its cost after they outgrow it.

The usual signal to move on is two reports that disagree on the same number, because point to point has no single place to reconcile them.

Hub and Spoke: The Data Integration Hub

Hub and spoke routes every source through one central data integration hub, which then feeds every destination. Each source connects once, to the hub, so a new system means one new connection. The risk is a bottleneck when one small team owns the whole hub.

Hub and spoke diagram with CRM, ERP, billing, ad, and product database sources feeding one cloud warehouse hub.

Today the hub is usually a cloud warehouse, which is the modern form of the pattern.

Warehouse-Centric ELT

Warehouse-centric ELT loads raw data into a cloud warehouse first and transforms it there. ELT stands for extract, load, transform. The order matters because raw history stays available when business rules change. It is the default stack for most growing US businesses.

The main cost is warehouse compute, which grows with every model rebuild and query. Classic ETL, which transforms data before loading it, still suits cases where raw data must never land in the warehouse.

Event-Driven and Streaming

Event-driven integration moves each change as an event the moment it happens, usually through Apache Kafka or a managed service built on it. The pattern suits fraud checks, inventory and pricing, where a delay of minutes costs money. It is overkill for a report someone reads once a day.

Streaming also needs skills many teams lack, so most teams stream only the few sources where speed changes a decision.

Data Virtualization and Federation

Data virtualization queries data where it lives, without copying it into central storage first. A virtual layer accepts one query and splits it across the source systems, a method also called federation. It suits quick access to data that changes often or can’t be copied for legal reasons.

Virtualization is a good bridge while a warehouse is being built, though heavy dashboards load production systems.

Data Mesh and Data Fabric

Data mesh and data fabric are operating models layered on top of the other patterns. In a data mesh, each business unit owns and publishes its own data products under shared standards. A data fabric uses metadata and automation to connect data across many platforms.

A company with one central data team gains little from either model, because the governance overhead arrives long before the benefits.

Data Integration Architecture Diagrams by Company Size

The right data integration architecture diagram depends mostly on how many sources a company runs and how many teams use the data. The three reference diagrams here work as data integration architecture examples for a small team, a mid-market company and an enterprise. Each diagram adds layers only when the company size calls for them.

Decision diagram sending under 10 sources to warehouse ELT, 10 to 50 to hub plus CDC, and many units to a federated platform.

The table lists common tool choices for each layer at each company size, with no vendor favored.

Layer

Small Team

Mid-Market

Enterprise

Ingestion

Fivetran or Airbyte

Fivetran or Airbyte, plus Debezium

Managed connectors, Debezium and Kafka

Storage

BigQuery or Snowflake

Snowflake, BigQuery or Databricks

Lakehouse on Databricks or Snowflake

Transformation

dbt

dbt with tests and CI

dbt projects per business unit

Orchestration

Schedulers built into each tool

Airflow or Dagster

Airflow or Dagster per business unit

Serving

One BI tool

BI tools plus reverse ETL

BI tools, reverse ETL and AI agents

Governance

A shared ownership sheet

A catalog and access roles

An enterprise catalog and data contracts

Each step up adds control and team cost, so moving up too early wastes money just as moving up too late does.

Small Team, Under 10 Sources

A small team with fewer than 10 sources needs a simple warehouse-centric ELT diagram. Managed connectors handle data warehouse integration for CRM, billing and ad data. A dbt project models the tables, and one BI tool reads them. One engineer can run this stack.

The diagram is a single left-to-right line from sources to dashboards, and every job runs on a schedule.

Mid-Market, 10 to 50 Sources

A mid-market company with 10 to 50 sources needs the same warehouse hub plus CDC and an orchestrator. Database sources move through CDC, so the warehouse stays current without heavy extracts. Airflow or Dagster runs the jobs in order and alerts on failures.

The diagram becomes hub and spoke, with the warehouse at the center and two-way arrows to operational tools. This step is where most data architecture design work happens, because every source needs a settled owner.

Enterprise, Multiple Business Units

An enterprise with several business units needs a federated diagram, with each unit running its own pipelines on a shared platform. A central team runs data lake integration into a shared lakehouse and owns the catalog. Domain teams own their data products and publish them for others.

Enterprise diagrams also have to show legacy systems and the path off them. Planning the migration path is a large part of architecture consulting at this size.

How to Design Your Data Integration Architecture

Good design starts with the business questions and works backward to the tools. Five decisions shape most designs. Each one is cheaper to get right early than to fix later.

The same five decisions shape Data Prism’s data integration strategy and architecture work, which spans more than 100 projects shipped over more than six years.

Start From the Questions the Business Asks

Start from the questions the business asks, then list the data each question needs. A question such as “which channels bring customers who stay a year” needs ad, CRM and billing data joined on one customer ID. The list of questions sets the sources, the join keys and the freshness the architecture must deliver.

Choose Batch or Real Time per Source

Choose batch or real time for each source separately, never for the whole stack at once. Most sources only need a daily or hourly load. Real time pays off for the few sources where a delay changes a decision, such as stock levels or fraud signals. Mixing both in one architecture is normal.

Plan for Schema Change

Plan for schema change, because every source will rename, add or drop fields. Managed connectors handle many changes on their own, while custom pipelines often break without warning. Data contracts, where a source owner agrees to the fields and types it will send, catch breaking changes before they reach reports.

Build In Data Quality Checks

Build in data quality checks at ingestion and after every transformation. Simple tests catch most problems, such as missing IDs, duplicate rows and totals that don’t match the source. Tests in dbt, paired with alerts in the orchestrator, turn failed checks into tickets before a dashboard shows bad numbers.

Decide Build or Buy per Layer

Decide build or buy for each layer, not for the whole architecture. Buying managed connectors makes sense for common SaaS sources, where the vendor keeps up with API changes. Building makes sense for proprietary systems, unusual volumes or logic no tool supports. The team’s skill usually matters more than the license price.

Buying one integrated data platform makes the choice once for every layer, which is faster at first and harder to undo.

Architecture Mistakes That Raise Cost Later

The architecture mistakes that raise cost later are usually choices that worked at a smaller size. A database that handled the first million rows struggles at a hundred million. Each shortcut becomes rework.

The Scout data platform shows the pattern. Its analytics ran on PostgreSQL, an operational database, while the data grew past 200 million consumer records. Data Prism moved the platform to Snowflake for analytics and OpenSearch for search, with Amazon S3 staging. Processing time fell from 84 hours to 2 hours, and monthly infrastructure cost fell from about $4,000 to $40.

Scout results graphic showing processing time cut from 84 hours to 2 hours on 200M+ records, with cost down to about $40.

Custom one-off integrations carry a similar hidden cost. Fivetran’s survey of 500 IT, data and AI leaders found that 53% of companies relying on custom integrations lose at least a quarter of their AI spending to avoidable work, while 40% of those on standardized platforms do.

Is a Formal Architecture Worth It?

A formal data integration architecture is worth it once a company has more than a handful of sources or more than one team reading the data. Below that point, a few managed connectors and a warehouse cover the need without a design document. Above it, the cost of reports that disagree grows faster than the cost of planning.

So, is a full architecture overkill for a startup? Usually it is, as long as the startup keeps one warehouse as the single place every report reads from.

Everything above assumes a team can see its own architecture clearly, and most teams can’t until someone draws it. Data Prism reviews a current architecture diagram for free and flags its cost and reliability risks, and a strategy call is where the review starts.

Book a Free 30-Minute Meeting

Discover how our services can support your goals — no strings attached. Schedule your free 30-minute consultation today and let's explore the possibilities.

Book a Free Call

Frequently Asked Questions

No. ETL is one technique for moving data, where data is extracted, transformed and then loaded. Data integration is the wider goal of combining data from many systems, and it can use ETL, ELT, CDC, streaming or virtualization.

A data integration layer is the part of the stack that moves and prepares data between sources and storage. It usually covers ingestion and transformation, and it hides each source’s quirks from the reports and models downstream.

Data architecture covers every data store and data model a company runs. Data integration architecture is the part of it that decides how data moves between those stores, in what order and on what schedule.

An integrated data platform is a single product that bundles several layers, such as ingestion, storage, transformation and dashboards. It is quicker to adopt than a layered stack, though switching away later is harder than swapping one tool.

Data pipeline architecture describes how one pipeline is built, from its source through each processing step to its destination. Integration architecture sits a level higher and decides how the pipelines fit together across the business.

Book Consultation