What should the future platform be responsible for?
Clarify where Databricks, Azure storage, Unity Catalog, BI, ML, and data product ownership start and stop.
Advisory Playbook 01 | Lakehouse Modernization
Modernize legacy analytics into a governed lakehouse.
A working advisory method for assessing a legacy estate, comparing modernization choices, recommending a target direction, and sequencing delivery without guessing the path.
Client Decision
The modernization question is not only "can we move to Databricks?"
The real question is how to move without rebuilding legacy confusion on a newer platform. This playbook turns that decision into evidence, trade-offs, a recommendation, and a sequence leaders and delivery teams can evaluate together.
Which workloads move first, and which should not move yet?
Prioritize by business value, dependency risk, quality issues, report usage, platform readiness, and retirement opportunity.
Who owns quality, access, cost, releases, and reliability?
Modernization only holds if governance, CI/CD, observability, and domain ownership are designed before scale.
Modernization Thesis
The business case should be explicit before architecture gets detailed.
A good lakehouse decision connects platform change to business trust, delivery speed, governance maturity, and the ability to support analytics, ML, and AI workloads.
Fragmented data estate
Legacy warehouses, point-to-point ETL, duplicated reports, and unclear ownership create slow change cycles and inconsistent metrics.
Governed lakehouse platform
Data is organized into medallion layers, governed through Unity Catalog, delivered as reusable data products, and promoted through CI/CD.
Business-ready data products
Teams gain reliable analytics, clearer ownership, faster onboarding of new data sources, and a stronger base for AI and ML workloads.
What I Would Assess
What must be understood before the target state is credible.
A modernization plan should make legacy risk visible: source complexity, report dependencies, data quality issues, ownership gaps, and operational constraints.
| Assessment Area | Signals To Capture | Architecture Output |
|---|---|---|
| Source landscape | ERP, CRM, SaaS, files, APIs, streams, refresh frequencies, volumes, and criticality. | Source inventory and ingestion prioritization map. |
| Pipeline estate | Legacy ETL jobs, failure rates, dependencies, schedule conflicts, manual interventions. | Migration dependency graph and wave plan. |
| Reporting usage | High-value reports, duplicate metrics, owner conflicts, consumers, SLA expectations. | Report rationalization and semantic model backlog. |
| Governance gaps | Access exceptions, unclear data ownership, missing lineage, data classifications. | Unity Catalog ownership and access model. |
Example Client Deliverable 01
Target-state decision map.
An engagement-ready view for aligning leaders and teams on how business domains, source systems, the lakehouse platform, governance, and consumers should interact.
Producers
Business applicationsERP, CRM, operational databases, SaaS platforms
Event sourcesApplication events, IoT signals, transactional streams
External dataPartner files, APIs, market data, manual uploads
Lakehouse Platform
Azure landing zoneIdentity, networking, monitoring, policies, DevOps
Databricks lakehouseIngestion, Delta Lake, Spark, workflows, SQL warehouses
Unity CatalogCatalogs, schemas, tables, lineage, permissions, discovery
Consumers
BI and analyticsPower BI, semantic models, operational dashboards
ML and AIFeature pipelines, model training, RAG knowledge feeds
Data productsDomain-owned datasets, APIs, exports, governed sharing
Decision purpose: align executives, platform teams, and data consumers on the modernization boundary before detailed solution design.
Example Client Deliverable 02
Target-state responsibility model.
A platform-level output that clarifies capabilities, boundaries, and ownership before detailed solution design begins.
Ingestion
Azure Data FactoryBatch orchestration, source extraction, dependency control
Auto LoaderIncremental cloud file ingestion into Delta
Structured StreamingNear real-time processing for event workloads
Landing storageADLS Gen2 raw landing and retention policy
Processing
Databricks WorkflowsPipeline scheduling, orchestration, dependencies
Spark jobsTransformations, joins, enrichment, validation
Delta Live TablesDeclarative pipelines and quality expectations
Data qualityRules, quarantine, alerts, and release gates
Governance
Unity CatalogCatalogs, schemas, permissions, lineage
Data classificationPII, sensitivity labels, retention requirements
Policy modelRBAC, ownership, approvals, audit evidence
ObservabilityFreshness, quality, cost, failure, and SLA metrics
Serving
SQL WarehouseCurated serving for BI and analytics
Power BICertified semantic models and reporting
ML / AIFeature sets, model training, knowledge retrieval
Data sharingAPIs, exports, partner feeds, Delta Sharing
Decision purpose: clarify platform responsibilities across ingestion, processing, governance, and serving layers.
Example Client Deliverable 03
Data-product readiness model.
This example explains how quality, business meaning, ownership, and consumption readiness should improve through each layer.
Bronze
Raw, replayable data aligned closely to source systems.
- Preserve source fidelity
- Capture ingestion metadata
- Support reprocessing and audit
- Apply schema capture and quarantine
Silver
Validated, conformed, deduplicated, and governed enterprise data.
- Apply quality rules
- Standardize identifiers and dates
- Conform dimensions and facts
- Publish lineage and ownership
Gold
Business-ready data products and curated marts for consumption.
- Domain-owned data products
- Certified semantic models
- SLA and refresh guarantees
- BI, ML, AI, and sharing outputs
Decision Area
Unity Catalog ownership and control model.
The governance model makes ownership, access, classification, and auditability visible. It should answer who owns each data product, who can access it, how lineage is tracked, and what controls apply.
| Governance Area | Design Decision | Evidence Produced |
|---|---|---|
| Catalog strategy | Separate catalogs by environment and domain where governance requires clear ownership boundaries. | Catalog/schema naming standard and workspace mapping. |
| Access control | Use groups, roles, and object-level permissions aligned to data classification. | RBAC matrix and access request workflow. |
| Lineage | Track lineage from source to data product and connect it to report and ML consumers. | Lineage review dashboard and impact-analysis process. |
| Quality ownership | Assign data product owner, technical owner, quality rules, and SLA for each product. | Data product contract and operational scorecard. |
Operating Model
What every reusable data product should define.
This template turns the lakehouse from a storage architecture into an operating model for governed data products.
Product NameCustomer 360 Gold Dataset
Business OwnerSales Operations Director
Technical OwnerData Engineering Customer Domain Team
ConsumersSales dashboards, churn model, account planning, customer support analytics
Source DependenciesCRM accounts, billing, support tickets, product usage events
Quality RulesUnique customer ID, valid account status, no orphan invoices, event freshness under 24h
Security ClassificationConfidential with PII fields masked for non-approved roles
SLADaily refresh by 07:00, 99% successful monthly runs, quality incidents triaged same day
Decision purpose: establish ownership, expectations, quality, access, and usage boundaries for each data product.
Delivery Model
How changes move safely to production.
Modernization needs an engineering delivery model, not only a target architecture. This view shows code, config, tests, and deployment flow.
01
Develop
Feature branch, notebooks, jobs, DLT definitions, tests, and bundle configuration.
02
Validate
Static checks, unit tests, data contract checks, sample pipeline execution.
03
Deploy Dev
Automated deployment to dev workspace with isolated catalog and test data.
04
Promote Stage
Integration testing, quality checks, permission review, and performance baseline.
05
Release Prod
Controlled production deployment, monitoring, rollback path, and release evidence.
Decision purpose: make environment strategy, testing, deployment, and rollback visible to engineering and platform teams.
Migration Wave Plan
A realistic modernization sequence.
The wave plan avoids a big-bang migration and gives stakeholders a controlled path from legacy estate to product-oriented lakehouse.
Wave 01
Assess
Inventory sources, pipelines, reports, business domains, quality gaps, and platform constraints.
Wave 02
Foundation
Set up Azure landing pattern, Databricks workspaces, Unity Catalog, CI/CD, and observability.
Wave 03
Pilot Domain
Implement one high-value domain through bronze, silver, gold, semantic model, and operations dashboard.
Wave 04
Scale
Migrate domains by priority, retire duplicate reports, enforce data product ownership, and optimize costs.
Recommendation Evidence
Example decisions that make the architecture explainable.
ADRs show the reasoning behind platform choices, helping teams understand tradeoffs instead of inheriting unexplained rules.
Use medallion architecture
Decision: organize data into bronze, silver, and gold layers to separate raw capture, validation, and business-ready consumption.
Govern with Unity Catalog
Decision: centralize discovery, permissions, lineage, and ownership through Unity Catalog rather than workspace-local controls.
Deploy through CI/CD
Decision: promote code and configuration through dev, stage, and production with automated validation and release evidence.
Apply this playbook to a real modernization decision.
Start with the legacy estate, decision pressure, constraints, and target-state questions. The assessment and outputs can be adapted from there.