Understanding Data Flows in Azure Data Factory

For instance, imagine a scenario where you need to clean, enrich, and aggregate sales data from multiple regions. Instead of writing extensive SQL or Python scripts, you can use a Data Flow to visually map these transformations and execute them seamlessly within ADF.

Key Components of Data Flows

• Source Transformation: defines where the data originates, such as Blob Storage or a SQL Database;
• Transformations: include tools like filtering, joining, aggregating, or deriving new columns to manipulate the data;
• Sink Transformation: specifies the destination for the processed data, such as another SQL Database, a data lake, or a file storage.

We will start our work with creating simple dataflow with source and sink transformations.

How to Set Up a Source Transformation

1. Add a new Data Flow in the Author section of Azure Data Factory Studio;
2. Drag a Source Transformation from the toolbox onto the Data Flow canvas;
3. In the Source Transformation settings, select a Linked Service, such as Azure SQL Database or Azure Blob Storage, to connect to your data source;
4. Choose an existing Dataset or create a new Dataset that represents the data to be ingested;
5. Configure file format options if connecting to Blob Storage, or provide a SQL query to filter or structure the incoming data for databases;
6. Validate the configuration and preview the data to ensure the source is correctly set up.

Sink Transformation for Processed Data

After defining transformations, use a Sink Transformation to specify where the transformed data will be stored. For example, you might save aggregated data back to the SQL database or export it as a CSV file to Blob Storage.