Differences between Data Factory in Microsoft Fabric and Azure Data Factory
A comparison of the features and functionalities of the two data integration services.
Data Factory is a cloud-based service that enables data integration, orchestration, and transformation across various sources and destinations. It allows users to create data pipelines that can perform various tasks such as copying, cleansing, transforming, and loading data.
Microsoft Fabric is a new cloud platform that aims to provide a unified data experience for customers. It offers a range of services and capabilities that enable data ingestion, storage, processing, analysis, and visualization. One of the services offered by Fabric is Data Factory, which is an enhanced version of Azure Data Factory.
Below we compare the features and functionalities of the two services and highlight the advantages of using Fabric Data Factory.
- Unified Data Platform Integration:
Data Factory in Fabric is better integrated with the unified data platform, allowing seamless data movement and transformation across various services. For example, users can easily connect to the Fabric Lakehouse and Datawarehouse services and use the Fabric SQL engine to perform high-scale data transformations. Users can also leverage the Fabric Catalog service to discover and manage data assets across the platform.
Azure Data Factory, on the other hand, has a more limited integration with the unified data platform. Users have to use different connectors and services to access and transform data from various sources and destinations
- Dataflow Gen2:
Fabric introduces Dataflow Gen2, which provides an easier experience for building transformations compared to mapping dataflows in Azure Data Factory. It supports high-scale data transformations using the Fabric Lakehouse SQL engine, which offers better performance and scalability than the Azure Data Factory Spark engine. It also supports more data sources and destinations than mapping dataflows
- Dataset Concept:
Unlike Azure Data Factory, Data Factory in Fabric doesn’t have the dataset concept. Instead, connections are used to connect to data sources and pull data. Connections are similar to linked services, but they offer a more intuitive way to create them. Connections can also be parameterized to allow dynamic values for the data.
- Publishing and Saving:
Unlike Azure Data factory in Fabric there is no need to publish to save pipeline content. Use the Save button directly. Clicking the Run button will save content before executing the pipeline.
- Autoresolve and Azure Integration Runtime:
Unlike Azure Data Factory, Fabric doesn’t have the concept of Integration runtime, it leverages the compute infrastructure that provides the data movement and transformation capabilities from the capacity provisioned for Fabric Workspaces. And as for Data Movement it relies on the On-premises Data Gateway for accessing on-premises data.
- Triggers
Data Factory in Fabric currently only supports schedules as triggers, allowing users to set the frequency and interval of the data pipeline execution. More triggers supported by Azure Data Factory, such as tumbling window triggers, event-based triggers, and on-demand triggers, are being added to Fabric Data Factory
- MVNet and Private End Point:
MVNet and Private End Point are features that enable users to secure and isolate their data pipeline network traffic. These features are currently not supported in Fabric Data Pipeline and only available in Azure Data factory.
In summary, Data Factory in Microsoft Fabric aims to provide an easy-to-use, powerful, and truly enterprise-grade solution for complex ETL scenarios. Its integration with the unified data platform and new features make it a compelling choice for data pipelines.
