From BI to Databricks: Simplifying Architecture Layers
Over the past few weeks, we’ve been exploring the journey from traditional Business Intelligence (BI) to Databricks. As part of this transition, it’s essential to address a key aspect: architecture. While the terminology might seem daunting at first—Bronze, Silver, Gold, these layers aren’t so different from what you’re already familiar with. Let’s break it down and show how you can adapt this framework to suit your organization.
Layers Are Layers—Let’s Keep It Simple
When it comes to data architectures, we all think in layers. They bring structure and clarity to an otherwise complex ecosystem. So, if you’re transitioning to the medallion architecture with its Bronze, Silver, and Gold layers, don’t let the terminology overwhelm you. We’ve even seen customers add Platinum and Diamond to their layers—why not? If it works for your organization, it works! Remember, a framework is just a starting point; tailor it to fit your needs.
Mapping Staging to the Bronze Layer
The key is to focus on the characteristics of each layer. For example, in the MSBI world, a staging layer is where raw source data lands. It’s still structured around the source, with minimal transformation. The Bronze layer in Databricks serves the same purpose: it’s the raw, unprocessed representation of the source data. Once you see this connection, the transition becomes less intimidating.
Mapping the Data Warehouse to the Silver Layer
The Data Warehouse layer in MSBI aligns closely with the Silver layer in the medallion architecture. In this stage, you introduce organizational standards, naming conventions, and other structures while keeping data at its lowest granularity. This layer is your backbone, designed to remain stable over time.
One key difference in Databricks is the flexibility around traditional data modeling approaches like Kimball or Inmon (star-schema), Anchor modeling, or Data Vault. Here, you can choose how strictly to adhere to these techniques based on your organizational needs. However, it’s critical to ensure this layer is resilient. Changes to data sources or organizational structures should have minimal impact on your models. To achieve this, consider domain-driven design, bounded contexts, and data mesh principles—these sociotechnical concepts help keep your architecture flexible and future-proof.
The Data Mart Layer: Gold (or Platinum, or Diamond)
The final layer—often referred to as the Gold layer in Databricks—is where you optimize data for consumption. Whether it’s a one-big-table design, 3NF, or star-schema, this layer is about delivering business value. Because of its direct impact on the end user, this is where companies tend to allocate the most investment. However, it’s vital not to overlook the upstream layers. A stable foundation is the only way to ensure a reliable and effective Gold layer.
At RevoData, we’ve learned that a logical and user-friendly structure for your Data Catalog is key. Instead of naming catalogs “Bronze,” “Silver,” or “Gold,” we use descriptive labels like “sources,” “domains,” or “data products” and apply the familiar terms as metadata tags. This approach provides a clear path to data for all users while keeping the architecture intuitive and scalable.
Make your Architecture Work for You
Transitioning to Databricks doesn’t mean starting from scratch. By mapping your existing architecture to the medallion framework and customizing it for your organization, you can create a system that’s both familiar and future-ready.
Ready to Take the Next Step?
At RevoData, we specialize in helping organizations make the most of Databricks. Whether you’re starting your journey or looking to refine your approach, we’re here to support you. Let us show you how Databricks can transform your data strategy and deliver real business impact. Reach out to us today to get started!
Rafal Frydrych
Senior Consultant at RevoData, sharing with you his knowledge in the opinionated series: Migrating from MSBI to Databricks.