Introduction to Data Engineering with Azure

Introduction to Data Engineering with Azure

0%

1. Getting Started with Azure and Core Tools

Have a deep dive into the foundational concepts and tools required to begin your journey with Microsoft Azure. You'll explore cloud computing basics, learn how to organize and manage resources efficiently using Azure Resource Groups, and set up core services like Blob Storage and Azure SQL Database. Additionally, you’ll secure sensitive data with Azure Key Vault and gain an overview of Azure Data Factory (ADF) to understand its role in modern data engineering workflows.

Getting Started

Understanding Azure Subscriptions

Organizing Resources with Resource Groups

Using Azure Blob Storage for Data Storage

Setting Up Azure SQL Database

Managing Credentials with Azure Key Vault

Introduction to Azure Data Factory

2. Foundations of Azure Data Factory

Explore the core functionalities of Azure Data Factory (ADF), a powerful tool for creating and automating ETL/ELT workflows. In this section, you'll learn how to configure services, datasets, and pipelines while exploring key activities such as Copy, Lookup, and Script. You'll also work with advanced features like the ForEach activity for iteration and pipeline parameters for flexibility.

Linked Services

What is ADF Pipeline?

Script Activity

Lookup Activity

Dataset Parameters

ForEach Activity

Execute Pipeline Activity

3. Data Flows and Transformations in ADF

This section focuses on leveraging Azure Data Factory's data flow capabilities to transform and enrich your data. You'll learn to connect various data sources, create derived columns, and merge datasets using joins. Practical examples will guide you through filtering, sorting, conditional splits, and aggregating data to generate meaningful insights. By mastering these transformations, you'll be equipped to build robust, efficient ETL workflows tailored to real-world data engineering challenges.

Understanding Data Flows in Azure Data Factory

Derived Column Transformation

Combining Data with Joins

Using Conditional Split and Filtering in ADF

Grouping, Select, and Sort Transformations in ADF

4. Practical Problem Solving with ADF

Apply your knowledge to real-world data engineering scenarios in this hands-on section. You'll tackle practical challenges like loading and analyzing data for different categories, identifying patterns, and assessing financial health using advanced ADF workflows. By solving these problems, you'll gain experience in designing end-to-end data solutions, working with diverse datasets, and generating actionable insights.

Challenge: Loading Data to Several Tables

Solving Loading Data Challenge

Challenge: Conducting Transaction Analysis

Solving Conducting Analysis Challenge

Course Content

Introduction to Data Engineering with Azure

Introduction to Data Engineering with Azure

1. Getting Started with Azure and Core Tools

Getting Started Understanding Azure Subscriptions Organizing Resources with Resource Groups Using Azure Blob Storage for Data Storage Setting Up Azure SQL Database Managing Credentials with Azure Key Vault Introduction to Azure Data Factory

2. Foundations of Azure Data Factory

Linked Services Datasets What is ADF Pipeline?Script Activity Lookup Activity Dataset Parameters ForEach Activity Execute Pipeline Activity

3. Data Flows and Transformations in ADF

Understanding Data Flows in Azure Data Factory Derived Column Transformation Combining Data with Joins Using Conditional Split and Filtering in ADF Grouping, Select, and Sort Transformations in ADF

4. Practical Problem Solving with ADF

Challenge: Loading Data to Several Tables Solving Loading Data Challenge Challenge: Conducting Transaction Analysis Solving Conducting Analysis Challenge

Combining Data with Joins

Data often resides in multiple tables or sources, making it essential to merge information to derive meaningful insights. In this chapter, you'll learn how to use joins in Azure Data Factory (ADF) Data Flows to merge datasets efficiently

ADF supports several types of joins:

Inner Join: combines rows where keys match in both datasets;
Left Outer Join: includes all rows from the left dataset and matching rows from the right;
Right Outer Join: includes all rows from the right dataset and matching rows from the left;
Full Outer Join: includes all rows from both datasets, with nulls for unmatched rows;
Cross Join: produces a Cartesian product of both datasets.

How to Combine Data with Joins in ADF

Create new Data Flow or use existing one;
Drag two Source Transformations onto the Data Flow canvas and connect them to the respective SQL tables;
Drag a Join Transformation from the toolbox onto the canvas and connect the two sources to it;
In the Join Transformation settings, select the Join Type and set the Join Condition;
Add a Derived Column Transformation or any other transformation after the join to get some insights;
Add a Sink Transformation to store the output;
Validate the Data Flow configuration to ensure everything is correct.

Everything was clear?

Thanks for your feedback!

Section 3. Chapter 3

some-alt