BI Developer - what does migration to Databricks mean for you?
When transitioning from MSBI to Databricks, the hardest part often isn’t the tools or the technology—it’s the people and their skills. That’s why, even though it’s listed last on our leaflet, we’re tackling this topic first. Let’s talk about what this migration means for your team and how to align their expertise with the Databricks ecosystem.
Expanding horizons - from Dashboards to Analysis and beyond
In the MSBI world, BI developers hold a central role. They’re highly skilled in SQL, possess extensive domain knowledge, and excel at creating dashboards, reports, and even complex cubes using MDX or DAX. Traditionally, this expertise has been closely tied to the classic Data Warehouse (DWH) environment, where structured data models and ETL processes form the backbone of the work. However, in the Databricks landscape, the BI Developer role evolves significantly, adapting to new paradigms and technologies that emphasize scalability, agility, and advanced data analytics.
With Databricks, SQL remains a vital skill, forming a strong foundation for exploring the platform’s capabilities. However, Databricks also introduces the world of Spark, with PySpark emerging as a favored tool among organizations. For BI developers, this shift offers an exciting opportunity to expand their skill set and evolve their role. Rather than a departure from strengths (SQL), this transition represents a chance to adapt and thrive in a rapidly changing environment and to become a more complete data professional.
The Data Engineer - why software skills matter?
As organizations venture into modern platforms like Databricks, the role of the Data Engineer emerges as critical for unlocking its full potential. To set the stage, it’s important to understand why Databricks excels—it’s a platform designed for flexibility, scalability, and advanced processing. However, it truly shines when operated by individuals with strong software engineering skills, particularly if PySpark is a key component of the data processing strategy.
For teams missing this expertise, our advice is clear: stick to SQL-based workloads in the beginning. This approach minimizes migration risks and ensures your team isn’t overwhelmed by the demands of Spark. After all, you don’t want to leave anyone behind at the station as the data train rolls forward.
The Platform Engineer - bringing infrastructure in-house
In an MSBI environment, platform support often comes from external teams, such as platform, infrastructure, or cloud operations. With Databricks, embedding a Platform Engineer, etc, within your team—even temporarily—can make all the difference.
This person ensures your team owns and optimizes the Azure Subscription and/or Resource Group. They help leverage Databricks’ robust security, isolate data storage and workloads, and manage dependencies effectively. Without this role integrated into your team, you risk missing out on these critical capabilities.
Building a future data team
Migrating to Databricks is more than just a technological shift; it’s a transformation of roles, skills, and team dynamics. This change brings challenges but also opportunities to build a robust, future-proof data team.
- Leverage existing SQL expertise as the starting point for migration to reduce risk and maintain momentum.
- Invest in upskilling your team to embrace new tools and workflows, positioning them for long-term growth.
- Embed platform engineering expertise, whether internally or through temporary support, to fully optimize Databricks’ capabilities.
Ultimately, the success of your Databricks implementation hinges on aligning your team’s skills with the platform’s strengths. By empowering your people and providing the right resources, you’ll not only navigate the migration smoothly but also unlock the full potential of a modern, agile data ecosystem. If you’re ready to make the leap, let’s start the journey together—reach out, and we’ll help you chart the course.
Rafal Frydrych
Senior Consultant at RevoData, sharing with you his knowledge in the opinionated series: Migrating from MSBI to Databricks.