A few of Dufrain’s team members took a deep dive into Microsoft Fabric through the data architecture lens.
The tech world is extremely excited about Microsoft Fabric and its ability to revolutionise how we process and handle data.
From a data architecture perspective, there are several key aspects to watch out for. First and foremost, it is an all-in-one analytics solution that covers everything from data movement to data science, Real-Time Analytics, and business intelligence.
It also contains a widespread suite of services in one place, including data engineering, data lake, and data integration that can also handle massive volumes of data in real-time (which is a game-changer!).
As we delve deeper into the world of Microsoft Fabric here are some key pieces of information to summarise our findings and highlights.
Let’s get started.
Getting started just got so much easier
Fabric allows an organisation to easily start on their data journey. Previous concerns about scaling, networking and infrastructure are reduced as Fabric’s SaaS solution caters for these architectural components out of the box. This means there is no need to create separate PaaS components in the Azure Portal like Azure Data Factory, Data Lake or Synapse because they already exist and are integrated in Fabric’s SaaS solution.
Democratisation of data through OneLake
OneLake is the OneDrive for data – this is a great idea, and really does improve the life for citizen developers. What we will need to consider, is how IT based Data Engineers work alongside business-based developers and analysts alike to really empower all in a governed manner. It provides one data lake for the entire organisation at scale and one copy of data for use across multiple analytical engines.
- Shortcuts allow teams to connect data across business domains without data movement which reduces ETL complexity, data duplication and data transfer costs. It is a symbolic link and functions as metadata that points from one data location to another
- Tables in a warehouse can be made available to another lakehouse without copying the data from the warehouse to the lakehouse. Again, this reduces complexity in the ETL
- Whilst data is shared the ownership does not, the original owner remains responsible for loading and managing it which ensures security and consistency across the platform
- Ability to create shortcuts to existing Azure Data Lake Storage (ADLS) gen2 accounts, enables the data to be virtualised as if it physically exists in the workspace
- Ability to add a shortcut to S3, potentially others will get added later. The data will be mapped and can be accessed using the same APIs including the ADLS gen2 APIs. The user will be able to use multiple clouds to write SQL queries or generate PBI reports effortlessly.
Out of the box Continuous Integration/Continuous Delivery (CI/CD) and pipelines
We’re excited about out of the box CI/CD because developers can integrate their development tools and processes straight into the Fabric platform. This allows for critical CI/CD task like:
- Integration of GitHub and Deployment Pipelines
- Fabric provides Git integration to provide developers with key tasks, including:
- Connect a Fabric Workspace to a git branch and sync the content into a repository
- Backup and version control their development work
- Revert to previous versions if required
- Collaborate with other developers using Git branching strategies.
There are still some outstanding tasks for CI/CD in Fabric though, but we are confident Microsoft will address them:
- As Fabric is still in Preview, only Reports and Datasets are supported
- Any unsupported items can still be connected to DevOps, but they won’t get saved or synched. They cannot also be committed or updated
- Only Git in Azure Repos is supported.
For Deployment Pipelines, Fabric uses REST APIs to achieve CI/CD within Azure DevOps. The Fabric deployment process seems very similar to the current PBI deployment pipeline process, with some limitations listed below that are not supported.
- Datasets that don’t originate from a .pbix
- PUSH datasets
- Streaming dataflows
- Reports based on unsupported datasets
- Template app workspaces
- Workbooks
- Metrics.
Governance within Microsoft Fabric
So many of the businesses we work with do not have control of their data governance. Fabric seems like it could be the perfect answer to this. We need to remove the noise from data governance and answer the simple questions:
- What is it?
- Who owns it?
- Who has access to it?
Microsoft Fabric pulls lineage and classification of data into one interface with Purview as a centralised page in Fabric for admins and data owners to manage their Fabric data estate.
Data lakehouses and warehouses
Both Synapse Data Engineering and Synapse Data Warehousing offerings serve the same purpose within Microsoft Fabric, storing data in Parquet files using the Delta format.
- Both fundamentally store data in a data lake
- As both are stored in Delta format, both support ACID (Atomicity, Consistency, Isolation and Durability) transactions
- Synapse Data Engineering is based on Apache Spark, supporting multiple programming languages such as Python, R, Scala, and Spark SQL
- Synapse Data Warehouse follows a traditional approach with SQL queries and stored procedures as the primary means of interaction. Full T-SQL capabilities
- Fabric allows mixing and matching workloads, enabling python code to interact with Lakehouse and Data warehouse tables while T-SQL queries can interact with Data warehouse and Lakehouse tables.
- Enables flexible architecture design. For instance, you can create a build utilising Medallion architecture, combining Spark-drive data ingestion, whilst cleaning the data with existing (migrated) T-SQL logic for a curated data warehouse
- Read-only. Unable to INSERT/UPDATE/DELETE the data from the other workload
- Able to combine multiple data lake tables with multiple data warehouse tables in a single workflow to generate an output.
- The interoperability allows the user to leverage the strengths of each workload type based on the project nature, language flexibility, data complexity, and existing or desired team skills.
- Data Engineering (Spark) is more suitable for metadata-driven frameworks and complex transformations
- Data Warehousing is more suitable if the primary work is done with T-SQL and requires full T-SQL support
- To further explore properties of each workload type and assist in the decision-making process between building a lakehouse or a warehouse, you can refer to Microsoft’s Fabric decision guide.
Game-changing compute
The serverless POLARIS engine gives the ability to bring unlimited compute capability to a query. For example, if 100 nodes ran – 1 node will run. If this is like Synapse and will be pay per query, this is game changing as the queries will be blisteringly fast at a much cheaper cost.
True realtime capability
Features such as PowerBI new Direct Lake storage mode, Event Streaming and Realtime Analytics through KQL will allow for true real-time data. For those working in manufacturing, credit cards or say IT Log files, this capability will be of great interest.
Power BI
A big question is “What does Microsoft Fabric mean for Power BI Specialists?” To answer this important question, one of our Power BI specialists Megan Livadas wrote a separate blog through the lens of Power BI.
The role of a data team
The move from PaaS to SaaS, and the shift in AI Capabilities inevitably raises questions around how job roles will be impacted and change over the coming years. The best advice our people give is embrace the change – don’t run from it, run to it. If we can adopt Microsoft Fabric to create data teams who have the reliability of an IT Based Data Engineer, but with the speed of insight of a Business Based Citizen Developer, we will see a great shift in capability.
Pricing
Microsoft Fabric (preview) SKU capacities are available to purchase in the Azure portal. The bill is calculated by the amount of storage used and overall compute provisioned as opposed to getting billed for separate Azure components.
Here is an example, the following table represents the estimated monthly cost pricing for the different SKU levels in UK South:
Fabric capacities are similar to existing Power BI Premium capacities, so exiting PowerBI Premium users will not get confused with the pricing structure.
Microsoft have provided a table that compares the Fabric vs Power BI SKU level to help customers gauge the equivalent SKU levels between the two products.
As an example, a P1 Power BI Premium per capacity has the same power as an F64 Fabric capacity.
Initial thoughts are the lowest Fabric tier of F2 provides a very low entry point into Fabric at around £250 per month (PAYG) or £9 a day, which is very affordable for small business users that have light workloads.
It will be interesting to see if Microsoft introduce a Reservations strategy similar to Synapse that provide discounts to customers that commit to using Fabric over a longer time-period (i.e. 1 year or 3 year reserved reservation).
Additional note is that OneLake storage pricing is comparable to Azure Data Lake Storage pricing, which is very cheap! For example, the price at US West 2 for OneLake is $23 per TB per month.
In summary
We are extremely excited by Microsoft Fabric, and Dufrain are actively developing Proof of Concepts and Demo solutions whilst it’s in Preview mode. These are our favourite top three Fabric concepts:
- A single unified data engineering, data science, real-time analytics and business intelligence platform in a SaaS solution. This has never been done before!
- The democratisation of data using OneLake which provides a data lake for the entire organisation at scale. That’s one central repository for all of your data!
- Real-time capabilities, especially around Power BI news Direct Lake storage mode. This is could dramatically reduce improve Power BI reporting performance.
[km-cta-block padding=20 block-classes=”has-dark-teal-background-colour has-white-colour” label=”Contact us for a free data health check” ]
Take control of your data
Take a look at our recent blog on Microsoft Fabric through the lens of Power BI or contact Dufrain today to discuss how data architecture can add real value to your business.
[km_button link=”https://www.dufrain.co.uk/contact/” classes=”cta-2″]Contact us[/km_button] or [km_button link=”tel:08001303656″ classes=”cta-2″]Call us on 0800 130 3656[/km_button][/km-cta-block]