Introduction
On November 12, 2026, Microsoft told partners it is retiring Purview Data Governance, with support ending on November 12, 2027. The retirement covers the Unified Catalog, Data Map and Workflows. Microsoft now points governance work to the OneLake catalog in Microsoft Fabric, so teams that built their governance around Purview have a year to move it.
So, how does a data integration architecture survive a change like this? It survives when each layer has one job and a clean boundary, so one tool can change without a rebuild. This article explains the layers of a data integration architecture, the six patterns teams build on, and reference diagrams for three company sizes.
What Is Data Integration Architecture?
Data integration architecture is the plan for how data moves from source systems, through ingestion and transformation, into storage where reports, applications and AI models can use it. The plan names each layer, the pattern that connects the layers, and the tools that run each step. A good plan lets a team replace one tool without rebuilding the others.
Most businesses need this plan once their tools stop agreeing. A CRM, an ERP and an ad platform each count customers in their own way, which creates data silos. The architecture decides where those counts get reconciled.
The Layers of a Data Integration Architecture
The architecture has six layers, and each one does a single job. Data enters through the source and ingestion layers. It becomes usable in the storage and transformation layers, and the serving layer hands it to people and systems. Orchestration, monitoring and governance run across all of them.
Many teams treat these six layers as their data integration architecture framework, since the layers stay fixed while the tools inside them change.
Source Layer
The source layer is every system that creates data the business wants to analyze. Typical sources are the CRM, ERP, billing tool, product database, ad platforms and spreadsheets. Each source exposes data through an API, a database connection or a file export.
Each source needs a named owner who knows when its fields change. SaaS sources are usually reached through API integration, and partners who ask for direct database access are better served by a governed API or a file drop.
Ingestion Layer
The ingestion layer copies data out of the sources and into storage, either on a schedule or as changes happen. Managed tools such as Fivetran and Airbyte cover common SaaS sources with ready-made connectors. Custom code covers the rest. Some teams call ingestion plus transformation the data integration layer.
For databases, change data capture (CDC) reads the transaction log instead of querying whole tables again. Debezium, an open-source CDC tool, records every row-level change as an event stream, in the order the changes happened.
Proprietary sources need custom pipeline development, and each pipeline follows its own data pipeline architecture.
Storage Layer: Data Warehouse and Data Lake Integration
The storage layer holds the integrated data, usually in a warehouse, a lake or a lakehouse. A warehouse such as Snowflake or BigQuery stores structured tables built for SQL. A lake stores raw files of any type at low cost. A lakehouse, such as Databricks, adds warehouse-style tables on top of lake storage.
Data warehouse integration suits teams whose questions center on sales, finance and operations. Data lake integration suits teams with logs, documents or event data, a stack the data lake guide covers.
Many lakehouse teams split storage into medallion tiers, which run from raw bronze copies to business-ready gold tables. Integration data that feeds other systems can share the raw tier with analytics. Once operations depend on the feed, the integration data needs its own cleaned tables, so an analytics change can’t break it.
Transformation Layer
The transformation layer turns raw copies into clean, joined tables that match how the business talks. It renames fields, removes duplicates, applies business rules and builds the models reports read from. Most modern teams write these steps as SQL in dbt, which runs inside the warehouse.
A semantic layer on top of the models defines each metric once, such as revenue, so every dashboard and AI tool calculates it the same way.
Serving Layer
The serving layer delivers integrated data to the people and systems that use it. Dashboards in Power BI, Tableau or Looker are the usual consumers. Reverse ETL tools push modeled data back into the CRM and support tools, so frontline teams see it where they work.
Orchestration, Monitoring and Governance
Orchestration, monitoring and governance run across every other layer instead of sitting in one place. An orchestrator such as Airflow or Dagster runs each job in the right order. Monitoring checks freshness, row counts and failed loads. Governance records who owns each dataset, who may see it and where it came from.
Governance is also the layer most tied to one vendor, as the Purview retirement shows. Keeping ownership and access rules as part of wider data management turns the move into a migration instead of a rebuild.
Data Integration Architecture Patterns
Data integration architecture patterns describe how the layers connect, and six of them cover almost every stack. Each pattern trades build effort against flexibility and cost. The patterns sit one level above the integration techniques, such as ETL and CDC, that actually move the data.
The table compares the six patterns by fit, main cost driver and the point where each stops fitting.
|
Pattern |
Best Fit |
Main Cost Driver |
Relative Cost |
Stops Fitting When |
|---|---|---|---|---|
|
Point to Point |
A few systems sharing data directly |
Engineer time per new connection |
Low, then rising |
Nobody can track every connection |
|
Hub and Spoke |
Many sources feeding shared reports |
The hub platform and its team |
Medium |
One team becomes the bottleneck |
|
Warehouse-Centric ELT |
Analytics on cloud data |
Warehouse compute |
Low to medium |
Data must act within seconds |
|
Event-Driven and Streaming |
Decisions made within seconds |
Always-on clusters and scarce skills |
High |
Daily reports are enough |
|
Data Virtualization |
Querying data where it lives |
Query load on source systems |
Low to medium |
Sources are slow or busy |
|
Data Mesh and Data Fabric |
Business units with their own data teams |
Platform team and governance overhead |
High |
One central team still copes |
Cost rises with the freshness and independence a pattern promises, so the cheapest pattern that answers the business questions usually wins.
Point to Point
Point to point connects each system directly to every system that needs its data. It is the fastest way to start and works while a company runs a few sources. Every new system adds connections to the others, so maintenance grows faster than the stack. Most teams only notice its cost after they outgrow it.
The usual signal to move on is two reports that disagree on the same number, because point to point has no single place to reconcile them.
Hub and Spoke: The Data Integration Hub
Hub and spoke routes every source through one central data integration hub, which then feeds every destination. Each source connects once, to the hub, so a new system means one new connection. The risk is a bottleneck when one small team owns the whole hub.

Today the hub is usually a cloud warehouse, which is the modern form of the pattern.
Warehouse-Centric ELT
Warehouse-centric ELT loads raw data into a cloud warehouse first and transforms it there. ELT stands for extract, load, transform. The order matters because raw history stays available when business rules change. It is the default stack for most growing US businesses.
The main cost is warehouse compute, which grows with every model rebuild and query. Classic ETL, which transforms data before loading it, still suits cases where raw data must never land in the warehouse.
Event-Driven and Streaming
Event-driven integration moves each change as an event the moment it happens, usually through Apache Kafka or a managed service built on it. The pattern suits fraud checks, inventory and pricing, where a delay of minutes costs money. It is overkill for a report someone reads once a day.
Streaming also needs skills many teams lack, so most teams stream only the few sources where speed changes a decision.
Data Virtualization and Federation
Data virtualization queries data where it lives, without copying it into central storage first. A virtual layer accepts one query and splits it across the source systems, a method also called federation. It suits quick access to data that changes often or can’t be copied for legal reasons.
Virtualization is a good bridge while a warehouse is being built, though heavy dashboards load production systems.
Data Mesh and Data Fabric
Data mesh and data fabric are operating models layered on top of the other patterns. In a data mesh, each business unit owns and publishes its own data products under shared standards. A data fabric uses metadata and automation to connect data across many platforms.
A company with one central data team gains little from either model, because the governance overhead arrives long before the benefits.
Data Integration Architecture Diagrams by Company Size
The right data integration architecture diagram depends mostly on how many sources a company runs and how many teams use the data. The three reference diagrams here work as data integration architecture examples for a small team, a mid-market company and an enterprise. Each diagram adds layers only when the company size calls for them.

The table lists common tool choices for each layer at each company size, with no vendor favored.
|
Layer |
Small Team |
Mid-Market |
Enterprise |
|---|---|---|---|
|
Ingestion |
Fivetran or Airbyte |
Fivetran or Airbyte, plus Debezium |
Managed connectors, Debezium and Kafka |
|
Storage |
BigQuery or Snowflake |
Snowflake, BigQuery or Databricks |
Lakehouse on Databricks or Snowflake |
|
Transformation |
dbt |
dbt with tests and CI |
dbt projects per business unit |
|
Orchestration |
Schedulers built into each tool |
Airflow or Dagster |
Airflow or Dagster per business unit |
|
Serving |
One BI tool |
BI tools plus reverse ETL |
BI tools, reverse ETL and AI agents |
|
Governance |
A shared ownership sheet |
A catalog and access roles |
An enterprise catalog and data contracts |
Each step up adds control and team cost, so moving up too early wastes money just as moving up too late does.
Small Team, Under 10 Sources
A small team with fewer than 10 sources needs a simple warehouse-centric ELT diagram. Managed connectors handle data warehouse integration for CRM, billing and ad data. A dbt project models the tables, and one BI tool reads them. One engineer can run this stack.
The diagram is a single left-to-right line from sources to dashboards, and every job runs on a schedule.
Mid-Market, 10 to 50 Sources
A mid-market company with 10 to 50 sources needs the same warehouse hub plus CDC and an orchestrator. Database sources move through CDC, so the warehouse stays current without heavy extracts. Airflow or Dagster runs the jobs in order and alerts on failures.
The diagram becomes hub and spoke, with the warehouse at the center and two-way arrows to operational tools. This step is where most data architecture design work happens, because every source needs a settled owner.
Enterprise, Multiple Business Units
An enterprise with several business units needs a federated diagram, with each unit running its own pipelines on a shared platform. A central team runs data lake integration into a shared lakehouse and owns the catalog. Domain teams own their data products and publish them for others.
Enterprise diagrams also have to show legacy systems and the path off them. Planning the migration path is a large part of architecture consulting at this size.
How to Design Your Data Integration Architecture
Good design starts with the business questions and works backward to the tools. Five decisions shape most designs. Each one is cheaper to get right early than to fix later.
The same five decisions shape Data Prism’s data integration strategy and architecture work, which spans more than 100 projects shipped over more than six years.
Start From the Questions the Business Asks
Start from the questions the business asks, then list the data each question needs. A question such as “which channels bring customers who stay a year” needs ad, CRM and billing data joined on one customer ID. The list of questions sets the sources, the join keys and the freshness the architecture must deliver.
Choose Batch or Real Time per Source
Choose batch or real time for each source separately, never for the whole stack at once. Most sources only need a daily or hourly load. Real time pays off for the few sources where a delay changes a decision, such as stock levels or fraud signals. Mixing both in one architecture is normal.
Plan for Schema Change
Plan for schema change, because every source will rename, add or drop fields. Managed connectors handle many changes on their own, while custom pipelines often break without warning. Data contracts, where a source owner agrees to the fields and types it will send, catch breaking changes before they reach reports.
Build In Data Quality Checks
Build in data quality checks at ingestion and after every transformation. Simple tests catch most problems, such as missing IDs, duplicate rows and totals that don’t match the source. Tests in dbt, paired with alerts in the orchestrator, turn failed checks into tickets before a dashboard shows bad numbers.
Decide Build or Buy per Layer
Decide build or buy for each layer, not for the whole architecture. Buying managed connectors makes sense for common SaaS sources, where the vendor keeps up with API changes. Building makes sense for proprietary systems, unusual volumes or logic no tool supports. The team’s skill usually matters more than the license price.
Buying one integrated data platform makes the choice once for every layer, which is faster at first and harder to undo.
Architecture Mistakes That Raise Cost Later
The architecture mistakes that raise cost later are usually choices that worked at a smaller size. A database that handled the first million rows struggles at a hundred million. Each shortcut becomes rework.
The Scout data platform shows the pattern. Its analytics ran on PostgreSQL, an operational database, while the data grew past 200 million consumer records. Data Prism moved the platform to Snowflake for analytics and OpenSearch for search, with Amazon S3 staging. Processing time fell from 84 hours to 2 hours, and monthly infrastructure cost fell from about $4,000 to $40.

Custom one-off integrations carry a similar hidden cost. Fivetran’s survey of 500 IT, data and AI leaders found that 53% of companies relying on custom integrations lose at least a quarter of their AI spending to avoidable work, while 40% of those on standardized platforms do.
Is a Formal Architecture Worth It?
A formal data integration architecture is worth it once a company has more than a handful of sources or more than one team reading the data. Below that point, a few managed connectors and a warehouse cover the need without a design document. Above it, the cost of reports that disagree grows faster than the cost of planning.
So, is a full architecture overkill for a startup? Usually it is, as long as the startup keeps one warehouse as the single place every report reads from.
Everything above assumes a team can see its own architecture clearly, and most teams can’t until someone draws it. Data Prism reviews a current architecture diagram for free and flags its cost and reliability risks, and a strategy call is where the review starts.
Book a Free 30-Minute Meeting
Discover how our services can support your goals — no strings attached. Schedule your free 30-minute consultation today and let's explore the possibilities.
Book a Free Call

