What is ADF Pipeline?

In Azure Data Factory, pipelines are the backbone of data integration workflows. They act as containers for activities, defining the sequence of tasks needed to move, transform, and process data. Pipelines streamline the process of working with diverse data sources, enabling seamless integration and automation of tasks. Think of it as a recipe for data movement and transformation, where each activity represents a step in the process.

In the next chapters, we will consider the most important pipeline activities and now let's start with the most commonly used - Copy Activity.

What is Copy Activity?

The Copy Activity is like a delivery truck for data. It extracts data from a source (e.g., Azure Blob Storage, Amazon S3, or an on-premises database) and loads it into a destination (e.g., Azure SQL Database or Data Lake). It handles the heavy lifting of data transfer while ensuring accuracy and efficiency.

How to Create a Copy Activity

1. Create a new pipeline in the Author section of Azure Data Factory Studio;
2. Drag the Copy Data Activity from the Activities pane onto the pipeline canvas;
3. Configure the source by selecting an existing Linked Service and Dataset for the data you want to move. Optionally, apply filters or specify queries to extract specific data using Query option;
4. Configure the destination (Sink) by choosing a Linked Service and Dataset where the data will be stored;
5. Align source and destination fields in the Mapping tab if necessary;
6. Validate the pipeline by hitting Debug button.