Microsoft Fabric End-to-End: From Raw Data to Business Decisions with Amit Chandak [MVP]
Microsoft Fabric brings data engineering, analytics, business intelligence, governance and increasingly AI together in one platform. But what does an end-to-end Fabric architecture actually look like when you move beyond individual features and start connecting everything?In this episode of the M365 FM Podcast, Mirko Peters is joined by Amit Chandak [Microsoft Data Platform MVP] for a practical journey through Microsoft Fabric — starting with raw organizational data and ending with trusted information that business users can use to make decisions.
WHY MICROSOFT FABRIC?
Before Fabric, organizations could already build sophisticated analytics architectures using Azure, Power BI and other platforms. The problem wasn't a lack of technology. In many cases, it was the opposite: organizations had too many choices, separate storage technologies, different compute models and multiple copies of essentially the same data.Amit explains how Microsoft Fabric attempts to simplify this architecture by bringing workloads together around shared foundations such as OneLake, common Fabric capacity and the Delta format. Lakehouses, warehouses, Power BI and other Fabric experiences can therefore operate as parts of a broader platform instead of completely isolated services.
ONELAKE AS THE FOUNDATION
OneLake is one of the central concepts behind Fabric. Amit compares it conceptually to OneDrive: instead of every analytics workload creating completely independent storage environments, OneLake provides a virtualized storage foundation across the Fabric tenant.Organizations can still separate data through workspaces, Lakehouses, Warehouses and domains, but those resources exist within a common Fabric storage architecture. This becomes particularly important when organizations want to reduce unnecessary duplication while maintaining security and organizational boundaries.
CENTRALIZED DATA OR DATA MESH?
Fabric doesn't automatically mean putting everything into one giant centralized analytics environment.For smaller organizations, a centralized architecture may still work well. As organizations become larger, Amit sees increasing value in domain-oriented architectures where areas such as sales, finance and purchasing can have their own workspaces and responsibilities.IT can remain responsible for availability, governance and the technical foundation while business domains increasingly take ownership of how their data is analyzed and consumed.
SHORTCUTS INSTEAD OF COPYING DATA
One of the recurring themes throughout the conversation is avoiding unnecessary copies of data.Fabric Shortcuts allow teams to reference data stored elsewhere rather than physically copying it into every environment that needs it. That can apply both inside Fabric and to supported external storage.Amit also explains an interesting architectural benefit of shortcuts: they can help separate workloads across capacities. This can become important when organizations want Power BI consumption workloads isolated from intensive data engineering workloads while still working with the same underlying information.
LAKEHOUSE VS. WAREHOUSE
One of the biggest Fabric architecture questions remains: Should you use a Lakehouse or a Warehouse?A Lakehouse can work with structured and unstructured data and is naturally aligned with Spark. A Fabric Warehouse focuses on structured data and provides the familiar T-SQL experience.Both ultimately use Delta for structured data inside Fabric, which means the decision increasingly comes down to the type of data, preferred technologies and workloads.Organizations with strong SQL teams don't necessarily need to abandon their existing skills. Teams working with very large datasets, advanced engineering scenarios, unstructured information or extensive data science workloads may find the Lakehouse and Spark approach more attractive.
GETTING DATA INTO FABRIC
Once the architecture is defined, organizations still need to bring data into Fabric.Amit walks through several approaches, including Shortcuts, Mirroring, Pipelines, Copy Activity, Copy Jobs, Dataflow Gen2 and notebooks.The right option depends heavily on the source and use case. Dataflow Gen2 remains particularly useful because of its broad connector support and familiar Power Query experience. For Power BI professionals entering Fabric, this can provide a natural starting point before moving toward more engineering-oriented approaches.
WHEN PYSPARK BECOMES IMPORTANT
Power Query and Dataflow Gen2 can work very well for small and medium-sized workloads, but scale changes the equation.For larger transformation workloads, Amit sees significant advantages in Spark-based processing. PySpark and Spark notebooks provide greater flexibility and are designed for distributed processing at scale.SQL and Power BI professionals don't necessarily have to make that transition immediately. Spark SQL can provide a familiar entry point for SQL developers, while many PySpark operations have conceptual similarities to transformations Power Query users already understand.
DO YOU REALLY NEED BRONZE, SILVER AND GOLD?
The Medallion Architecture has become almost synonymous with modern data engineering: Bronze for raw data, Silver for cleaned and transformed data, and Gold for business-ready information.But Amit argues that organizations shouldn't create layers simply because an architecture diagram says they should.If an organization already has excellent master data management and high-quality source data, every intermediate layer may not provide enough value to justify another copy and another transformation step. Where source data quality is inconsistent, however, the traditional Bronze-Silver-Gold structure remains highly valuable.
WHERE SHOULD BUSINESS LOGIC LIVE?
Another important architecture decision is determining where calculations and business rules belong.Amit prefers keeping measures and KPIs in the semantic model where possible because they remain dynamic and easier to change. Heavy row-level calculations across millions of records, however, are generally better handled earlier in the Gold layer.The result is not an either-or decision. A mature Fabric architecture distributes business logic deliberately between the transformation layer and semantic model depending on the type and cost of the calculation.
SEMANTIC MODELS ARE BECOMING MORE IMPORTANT
The semantic model was traditionally viewed primarily as the foundation for Power BI reports. AI is changing that role.Relationships, measures, KPIs and business definitions encoded within a semantic model represent organizational knowledge. That knowledge can increasingly be consumed by experiences beyond traditional dashboards.Amit discusses how Data Agents, ontology-driven solutions and Fabric Apps can build on semantic models. Instead of being merely a Power BI component, the semantic model can become a reusable business layer between organizational data and multiple human or AI-driven experiences.
DOES DAX STILL MATTER IN THE AI ERA?
AI can already generate DAX, SQL, PySpark and other code remarkably quickly. Does that mean professionals no longer need to learn these languages?Amit argues that expertise still matters, particularly when generated solutions don't work correctly or need optimization. AI can dramatically reduce the amount of syntax professionals need to write manually, but understanding the underlying logic remains valuable for debugging and improving what AI creates.Over time, the skill may shift from remembering syntax toward understanding architecture, algorithms, business logic and how to describe requirements precisely.
SECURITY FROM WORKSPACE TO DATA
Putting more organizational data into a unified platform makes security increasingly important.The discussion covers security across multiple layers, beginning with Fabric workspace roles and continuing through item-level access, OneLake security and semantic-model security.For larger organizations, Amit recommends using security groups rather than managing individual users wherever possible. Microsoft Entra provides the identity and group foundation, while Fabric applies those identities and groups to workspaces, items and data access.
FROM DATA ENGINEERING TO BUSINESS DECISIONS
Ultimately, most business users don't care whether their information came through PySpark, a Lakehouse, Delta tables or Dataflow Gen2. They care about getting reliable answers.This is where Amit sees Data Agents, ontology and AI-driven analytics becoming increasingly important.Instead of every user receiving the same predefined dashboard, future analytics experiences could become far more dynamic. Users may interact conversationally with organizational data and eventually have personalized reports or applications generated around their specific questions and responsibilities.
RAPID FIRE
The episode closes with a rapid-fire round.SQL or PySpark? PySpark.Dataflow Gen2 or Notebook? Notebook.DAX or SQL? DAX.Most underrated Fabric feature? Fabric Apps.Biggest Power BI modeling mistake? Many-to-many relationships.And perhaps the most provocative answer of the episode:Are Fabric Apps the new Power BI?Amit's answer: Yes.
THE BIG PICTURE
Microsoft Fabric isn't simply another analytics product to add to the Microsoft stack.Its larger opportunity is connecting ingestion, engineering, storage, Lakehouse and Warehouse architectures, semantic models, security, Power BI and emerging AI experiences into one coherent analytics platform.The technical architecture matters, but the endpoint isn't OneLake, PySpark or even Power BI.The endpoint is trusted information that people — and increasingly AI agents — can turn into better business decisions.
Become a supporter of this podcast: https://www.spreaker.com/podcast/m365-fm-modern-work-security-and-productivity-with-microsoft-365--6704921/support.
WHY MICROSOFT FABRIC?
Before Fabric, organizations could already build sophisticated analytics architectures using Azure, Power BI and other platforms. The problem wasn't a lack of technology. In many cases, it was the opposite: organizations had too many choices, separate storage technologies, different compute models and multiple copies of essentially the same data.Amit explains how Microsoft Fabric attempts to simplify this architecture by bringing workloads together around shared foundations such as OneLake, common Fabric capacity and the Delta format. Lakehouses, warehouses, Power BI and other Fabric experiences can therefore operate as parts of a broader platform instead of completely isolated services.
ONELAKE AS THE FOUNDATION
OneLake is one of the central concepts behind Fabric. Amit compares it conceptually to OneDrive: instead of every analytics workload creating completely independent storage environments, OneLake provides a virtualized storage foundation across the Fabric tenant.Organizations can still separate data through workspaces, Lakehouses, Warehouses and domains, but those resources exist within a common Fabric storage architecture. This becomes particularly important when organizations want to reduce unnecessary duplication while maintaining security and organizational boundaries.
CENTRALIZED DATA OR DATA MESH?
Fabric doesn't automatically mean putting everything into one giant centralized analytics environment.For smaller organizations, a centralized architecture may still work well. As organizations become larger, Amit sees increasing value in domain-oriented architectures where areas such as sales, finance and purchasing can have their own workspaces and responsibilities.IT can remain responsible for availability, governance and the technical foundation while business domains increasingly take ownership of how their data is analyzed and consumed.
SHORTCUTS INSTEAD OF COPYING DATA
One of the recurring themes throughout the conversation is avoiding unnecessary copies of data.Fabric Shortcuts allow teams to reference data stored elsewhere rather than physically copying it into every environment that needs it. That can apply both inside Fabric and to supported external storage.Amit also explains an interesting architectural benefit of shortcuts: they can help separate workloads across capacities. This can become important when organizations want Power BI consumption workloads isolated from intensive data engineering workloads while still working with the same underlying information.
LAKEHOUSE VS. WAREHOUSE
One of the biggest Fabric architecture questions remains: Should you use a Lakehouse or a Warehouse?A Lakehouse can work with structured and unstructured data and is naturally aligned with Spark. A Fabric Warehouse focuses on structured data and provides the familiar T-SQL experience.Both ultimately use Delta for structured data inside Fabric, which means the decision increasingly comes down to the type of data, preferred technologies and workloads.Organizations with strong SQL teams don't necessarily need to abandon their existing skills. Teams working with very large datasets, advanced engineering scenarios, unstructured information or extensive data science workloads may find the Lakehouse and Spark approach more attractive.
GETTING DATA INTO FABRIC
Once the architecture is defined, organizations still need to bring data into Fabric.Amit walks through several approaches, including Shortcuts, Mirroring, Pipelines, Copy Activity, Copy Jobs, Dataflow Gen2 and notebooks.The right option depends heavily on the source and use case. Dataflow Gen2 remains particularly useful because of its broad connector support and familiar Power Query experience. For Power BI professionals entering Fabric, this can provide a natural starting point before moving toward more engineering-oriented approaches.
WHEN PYSPARK BECOMES IMPORTANT
Power Query and Dataflow Gen2 can work very well for small and medium-sized workloads, but scale changes the equation.For larger transformation workloads, Amit sees significant advantages in Spark-based processing. PySpark and Spark notebooks provide greater flexibility and are designed for distributed processing at scale.SQL and Power BI professionals don't necessarily have to make that transition immediately. Spark SQL can provide a familiar entry point for SQL developers, while many PySpark operations have conceptual similarities to transformations Power Query users already understand.
DO YOU REALLY NEED BRONZE, SILVER AND GOLD?
The Medallion Architecture has become almost synonymous with modern data engineering: Bronze for raw data, Silver for cleaned and transformed data, and Gold for business-ready information.But Amit argues that organizations shouldn't create layers simply because an architecture diagram says they should.If an organization already has excellent master data management and high-quality source data, every intermediate layer may not provide enough value to justify another copy and another transformation step. Where source data quality is inconsistent, however, the traditional Bronze-Silver-Gold structure remains highly valuable.
WHERE SHOULD BUSINESS LOGIC LIVE?
Another important architecture decision is determining where calculations and business rules belong.Amit prefers keeping measures and KPIs in the semantic model where possible because they remain dynamic and easier to change. Heavy row-level calculations across millions of records, however, are generally better handled earlier in the Gold layer.The result is not an either-or decision. A mature Fabric architecture distributes business logic deliberately between the transformation layer and semantic model depending on the type and cost of the calculation.
SEMANTIC MODELS ARE BECOMING MORE IMPORTANT
The semantic model was traditionally viewed primarily as the foundation for Power BI reports. AI is changing that role.Relationships, measures, KPIs and business definitions encoded within a semantic model represent organizational knowledge. That knowledge can increasingly be consumed by experiences beyond traditional dashboards.Amit discusses how Data Agents, ontology-driven solutions and Fabric Apps can build on semantic models. Instead of being merely a Power BI component, the semantic model can become a reusable business layer between organizational data and multiple human or AI-driven experiences.
DOES DAX STILL MATTER IN THE AI ERA?
AI can already generate DAX, SQL, PySpark and other code remarkably quickly. Does that mean professionals no longer need to learn these languages?Amit argues that expertise still matters, particularly when generated solutions don't work correctly or need optimization. AI can dramatically reduce the amount of syntax professionals need to write manually, but understanding the underlying logic remains valuable for debugging and improving what AI creates.Over time, the skill may shift from remembering syntax toward understanding architecture, algorithms, business logic and how to describe requirements precisely.
SECURITY FROM WORKSPACE TO DATA
Putting more organizational data into a unified platform makes security increasingly important.The discussion covers security across multiple layers, beginning with Fabric workspace roles and continuing through item-level access, OneLake security and semantic-model security.For larger organizations, Amit recommends using security groups rather than managing individual users wherever possible. Microsoft Entra provides the identity and group foundation, while Fabric applies those identities and groups to workspaces, items and data access.
FROM DATA ENGINEERING TO BUSINESS DECISIONS
Ultimately, most business users don't care whether their information came through PySpark, a Lakehouse, Delta tables or Dataflow Gen2. They care about getting reliable answers.This is where Amit sees Data Agents, ontology and AI-driven analytics becoming increasingly important.Instead of every user receiving the same predefined dashboard, future analytics experiences could become far more dynamic. Users may interact conversationally with organizational data and eventually have personalized reports or applications generated around their specific questions and responsibilities.
RAPID FIRE
The episode closes with a rapid-fire round.SQL or PySpark? PySpark.Dataflow Gen2 or Notebook? Notebook.DAX or SQL? DAX.Most underrated Fabric feature? Fabric Apps.Biggest Power BI modeling mistake? Many-to-many relationships.And perhaps the most provocative answer of the episode:Are Fabric Apps the new Power BI?Amit's answer: Yes.
THE BIG PICTURE
Microsoft Fabric isn't simply another analytics product to add to the Microsoft stack.Its larger opportunity is connecting ingestion, engineering, storage, Lakehouse and Warehouse architectures, semantic models, security, Power BI and emerging AI experiences into one coherent analytics platform.The technical architecture matters, but the endpoint isn't OneLake, PySpark or even Power BI.The endpoint is trusted information that people — and increasingly AI agents — can turn into better business decisions.
Become a supporter of this podcast: https://www.spreaker.com/podcast/m365-fm-modern-work-security-and-productivity-with-microsoft-365--6704921/support.