Azure Data Engineering Course — Basic to Advanced | Grow Data SKills

Light
Dark

Complete Data Engineering With Azure - Basic To Advance

Data Engineering With AWS

By Shashank In Grow Data Skills

Self Paced
English
Completion Certificate

Demo Class Video

Data Engineering With AWS

1 Year

Validity (From the date of Enrollment)

180 Hours

Duration

40

Sessions

15

Projects

INR 9000

This course includes

Content Duration180 Hours
Total Sessions40
Total Industry Projects15
AZURE & GCP Cloud
Quality Assignments & quizzes after each module
Interview preparation guide
Dedicated placement assistance
Resume & Linkedin profile making
Doubt support on private discord community
Certificate of completion

Show More

Tech stack you'll learn

Data Engineering With Azure

Course Content

✅ Class - 1

What is Database?
Difference between Transactional Databases and NoSQL databases
What is DBMS & RDBMS?
Transactions & ACID Properties
Setup MySQL Workbench
DDL, DML, DQL, DCL
CREATE Command
INSERT Command
Integrity Constraints

✅ Class - 2

Alter Command
Drop, Truncate and Delete
Primary Key vs Foreign Key
Referential Integrity
Select Query, In-Built Functions, Aliases
UPDATE Command
Auto Increment in create table
Limit
Order By Clause
Conditional Operators
Logical Operators
Like Operation
User Defined Functions (UDFs)

✅ Class - 3

IS NULL, IS NOT NULL
Group By, Having Clause
Group Concat, Group RollUP
Sub Queries, IN and NOT IN
CASE-When
SQL Joins

✅ Class - 4

Exists and Not Exists
Window Functions
Frame Clause
Coalesce Function
Common Table Expressions - Iterative and Recursive

✅ Class - 1

BigData Fundamentals
5 V’s of BigData
Distributed Computation
Distributed Storage
Cluster, Commodity Hardware
File Formats
Types of Data
History of Hadoop
Hadoop Architecture & Components

✅ Class - 2

Map-Reduce Architecture
YARN Architecture

✅ Class - 3

Hive Complete Architecture
Hadoop Cluster Setup on GCP (Dataproc)

✅ Class - 4

Data Types in Hive
Create Database
Create Table
Load Data From Local
Load Data From HDFS
Internal Table
External Table
Array & Map Data Types
SerDe in Hive
File Formats in Hive - ORC, Parquet, Avro

✅ Class - 5

CSV SerDe
JSON SerDe
Parquet SerDe
ORC SerDe
Static Partitioning
Dynamic Partitioning
Bucketing
Map-Side Join, Bucket Map Join, Sorted Merge Join, Skew Join

✅ Class - 1

Kafka Cluster Architecture
Brokers
Topics
Partitions
Producer-Consumer, Consumer Group
Offset Management
Replicas
Commits
Sync & Async Commits

✅ Class - 2

Confluent Kafka Setup
Topic Creation
Schema Registry
Key, Value Message
Message in Kafka Topics based on Random and Constant Keys
Kafka Producer Code with Serialisation
Kafka Consumer Code with De-Serialization
Consumer Groups
Working with JSON, CSV Data
GCP Pub-Sub Setup
Producer & Consumer for GCP Pub-Sub Setup

✅ Class - 1

CAP Theorem
What is MongoDB and MongoDB Atlas?
MongoDB vs Relational Database
MongoDB features
MongoDB use cases and applications
MongoDB architecture
Node
Data Centre
Cluster
Data replication
Write operation
Read operation
Indexing

✅ Class - 2

MongoDB Atlas Setup
MongoDB Cluster Creation
MongoDB Compass Setup
Database & Collection in MongoDB
Connect with MongoDB Cluster from MongoDB Compass
Import JSON data in MongoDB Collection
Queries on MongoDB Collection from Python Application
KSQLdb in Confluent Kafka
Streams in KSQLdb
Tables in KSQLdb
Persistent Queries in KSQLdb
JOIN queries on streams in KSQLdb
McDonald's Payments Stream data ingestion from Kafka to MongoDB
Setup Orders & Payments Streams using KSQLdb
Setup windowed JOIN streams using KSQLdb
Setup MongoDB Sink Connector

✅ Class - 3

CAP Theorem
What is Apache Cassandra?
Cassandra Database vs Relational Database
Apache Cassandra features
Cassandra use cases and applications
Cassandra architecture
Node
Data Centre
Cluster
Commit log
Mem-table
SSTable
Data replication
Read operation

✅ Class - 4

Data Partitioning and Token
VNodes in Cassandra
Read Operation in Cassandra
Compaction in Cassandra
Gossip Protocol in Cassandra
Write consistency in Cassandra
Read consistency in Cassandra
Partition Key, Cluster Key, Row Key Declaration
Cassandra Setup Using Docker
CQL in Cassandra
Cassandra Free Tier Setup On DataStax
Queries in Cassandra using Python

✅ Class - 1

Problems with Hadoop Map-Reduce
What is Apache Spark?
Features of Spark
Spark ecosystem
RDD in Spark
Properties of RDD
How Spark perform data partitioning?
Transformation in Spark
Narrow Transformation vs Wide Transformation
Action in Spark
Read & Write operation in Spark are transformation or action?
Lazy evaluation in Spark
Lineage graph or DAG in Spark
How DAG looks on Spark Web UI?
Job, Stage and Task in Spark
What if Spark cluster capacity is less than the size of data to be processed?
Spark in-depth architecture and it's components
Spark with Standalone Cluster Manager Type
Spark with YARN Cluster Manager Type
Deployment modes of Spark Application
How DAG looks on Spark Web UI?
Internals of Spark Job over the cluster

✅ Class - 2

Persist and Caching in Spark
Storage Levels in Persist
How does data skewness occur in Spark?
Techniques to deal with data skewness
Repartition vs Coalesce
Example of Key Salting technique
RDD vs Dataframe vs Dataset
How to use Spark-Submit utility?
Memory management in Spark
Memory components in Executor Container
Dynamic occupancy mechanism
How to process 1 TB of data in Spark?
Resource allocation case study - 1 : 6 Nodes and each node have 16 cores & 64 GB RAM
Resource allocation case study - 2 : 6 Nodes and each node have 32 cores & 64 GB RAM
Resource allocation case study - 3 : When more memory isn't required for the executors
Broadcast and Accumulators in Spark
Different type of failures in Spark and how to resolve them
Out Of Memory failures
Code and Resource level optimizations in Spark
Best practices to design Spark Applications

✅ Class - 3

Spark Cluster Setup On GCP Dataproc
Spark Session creation
Create dataframe with custom schema
Read csv data from HDFS
Partitions and Partition size in Spark job
Select operation
withColumn operation
withColumnRenamed operation
Filter operation
Drop column operation
Drop Duplicates operation
Order By operation
Group By operation
Accumlator
Case-When operation
Window functions
Join & Broadcast join operation
Spark SQL, Register dataframe as table
Write CSV data in HDFS without partition key
Write Parquet data in HDFS
Write CSV data in HDFS with partition key
Write CSV data in HDFS with Coalesce
Read JSON Data in Spark and explode columns

✅ Class - 4

Execution of Spark application using Spark-Submit Utility
Monitor, Debug & Understand Spark Dag on Spark Web UI - Practical Example
What is Stream Processing?
Spark structured streaming
Spark streaming with word count example
Output modes in writeStream in Spark structured streaming
What if memory due to state management is full?
DStream vs Spark Structured Streaming
Spark structured streaming with File as source
Triggers in Spark structured streaming

✅ Class - 5

Checkpointing
Exactly once in spark structured streaming
Stateless and Stateful Processing
Global aggregation and Windowed aggregation
Windowing
Sliding window
Tumbling window vs Sliding window
When and why we should use windowing?
Windowed aggregations example
Arbitrary stateful transformations
Watermarking
Working example of handling delayed events using watermarking
Code implementation for Stateless spark structured streaming with source as Confluent Kafka Topic
Code implementation for Stateful spark structured streaming with source as Confluent Kafka Topic - Global aggregation and Windowed aggregation
Spark structure streaming pipeline implementation where Source is Confluent Kafka Topic and Destination is MongoDB

✅ Class - 1

What is orchestration in BigData?
Need of dependency management in Data Pipeline design
What is Airflow?
Architecture & Different Components of Airflow
Operators in Airflow
How to write Airflow DAG Scripts?
Attribute description
How to execute parallel tasks?

✅ Class - 2

Setup Airflow on GCP using Composer
Create and schedule Airflow Dag with sequential tasks using BashOperator and PythonOperator
Create and schedule Airflow dag with parallel tasks using BashOperator and PythonOperator
Airflow Exercise - 1 : End-To-End Airflow Dag to Create Dataproc Cluster, Run PySpark Job on cluster and Delete GCP Dataproc cluster
Airflow Exercise - 2 : Airflow Dag to support data backfilling via parameterized date inputs and use if Variables in Airflow
Project 1 - Flight Booking Data Pipeline with Airflow & CICD (Industrial Project)

Tech Stack - GitHub, GitHub Actions, Google Storage, PySpark, Dataproc Serverless, Airflow, BigQuery

✅ Class - 1

What is Databricks?
Unity Catalog
Delta Lake & Delta Tables
Databricks Account Setup on GCP
Workspace Setup
Metastore Setup
Managed & External Catalog Setup
Volumes In Databricks
Databricks Cluster Setup
PySpark Notebook Setup
Read/Write from Databricks Volume in PySpark Notebook
Create Delta table and Write data in PySpark using DeltaTable Python API
Write partitioned data in Delta table
Read from Delta table in PySpark
Time travel in Delta table (Read from specific version or timestamp)
How to connect Databricks Notebook with Github and take care code versioning
Delta Sharing in Databricks
Create Share & Recipients in Databricks for sharing data assets via Delta Sharing
Access data assets in local machine shared via Delta Sharing

✅ Class - 2

Project 1 - E-commerce Event-Driven Data Pipeline (Industrial Project)

Tech Stack: Databricks, PySpark, Delta Lake, Databricks Volumes, Databricks Workflows, GitHub

✅ Class - 3

Project 2: Travel Booking SCD2 Data Warehouse(Industrial Project)

Tech Stack: Databricks, PySpark, Delta Lake, Unity Catalog, PyDeequ (For Data Quality Checks), Databricks Volumes, Databricks Workflows
Project 3: Healthcare DLT Medallion Pipeline (Industrial Project)

Tech Stack - Databricks DLT, Delta Lake, SQL, Unity Catalog, Expectations, Databricks Workflows
Project 4: UPI Transactions CDC Streaming Analytics(Industrial Project)

Tech Stack: Databricks, PySpark Structured Streaming, Delta Lake (Change Data Feed), Unity Catalog

✅ Class - 1

OLAP vs OLTP
What is a Data Warehouse?
Difference between Data Warehouse, Data Lake and Data Mart
Fact Tables
Dimension Tables
Slowly changing Dimensions
Types of SCDs
Star Schema Design
Snowflake Schema Design
Galaxy Schema Design

✅ Class - 2

Case Study - 1: Expedia Advanced Data Warehousing & Modeling
Designed a galaxy schema for the travel domain covering hotels, flights, cars, payments, and customer dimensions.
Defined fact tables (search sessions, impressions, clicks, bookings, cancellations, payments, refunds) and their granularity.
Mapped out conformed dimensions like date, time, geo, device, channel, customer, property, room, flight, car, promo, and payment method.
Discussed business metrics such as gross booking value (GBV), take rate, cancellations, refund rates, average booking value, occupancy, etc.
Wrote SQL queries to compute KPIs and insights from the warehouse model.
Case Study - 2: Swiggy – Advanced Data Warehousing & Modeling
Built a food delivery domain model with facts for sessions, impressions, add-to-cart, order header, order items, delivery trips, cancellations, refunds, payments, and rider shifts.
Designed conformed dimensions for customers, restaurants, menu items, delivery partners, promotions, payment methods, fee policies, geo, device, channel, and date/time.
Covered SCD2 handling for slowly changing entities like customers, restaurants, menu items, and riders.
Explained key business metrics such as order-to-delivery conversion, AOV, take rate, on-time delivery %, cancellation rate, refund %, repeat customers, restaurant acceptance %, rider utilization, and SLA adherence.
Demonstrated SQL-based insights on customer behavior, operational efficiency, and revenue levers.

✅ Class - 1

Snowflake free tier account setup
Snowflake UI walkthrough
Load data from UI and create snowflake table
Hands On - Event driven data ingestion in snowflake table using SnowPipe
Tech Stack Used : Google Storage Bucket, GCP Pub-Sub, Snowflake
Create storage integration in snowflake to read data from external Google Storage Bucket
Create GCP Pub-Sub topic to send notification into it as soon as file is uploaded in bucket
Setup notification subscription between Bucket & Pub-Sub
Create notification integration in snowflake
Create external stage in snowflake
Create SnowPipe to load data from external stage to snowflake target table
How to create and schedule task in snowflake

✅ Class - 2

Project - 1: News Data Analysis(Industrial Project)

Tech Stack: NewsAPI, Airflow, GCS (Parquet), Snowflake, Python (Requests, Pandas), SQL
Project - 2: Car Rental Batch Ingestion (Industrial Project)

Tech Stack: Airflow, Google Cloud Storage (GCS), Dataproc (Spark), Snowflake, Python, SQL
Project - 3: Movies CDC Dynamic Tables (Industrial Project)

Tech Stack: Snowflake (Streams, Tasks, Dynamic Tables), SQL, Streamlit (Snowpark)

✅ Class - 3

BigQuery Fundamentals
BigQuery Architecture : Capacitor, Colossus Storage, Dremel Execution Engine, Borg Cluster Manager, Jupiter Network
Difference Between Internal & External Tables
Create Dataset in BigQuery From UI
Create Managed Table in BigQuery from UI on top of CSV file stored in Google Storage
Managed Table SQL Operations:
Create Schema
Create Table With Partition & Cluster Keys
Load CSV Data From Google Storage into BigQuery Table
Insert Command To Ingest Data in BigQuery Table
Update Command
Where Clause
Query table data with partition column
Show table lists in Schema
Show table definition
List distinct partitions
Time travel select query
Create External Table in BigQuery On Top of Semi-Structured Data (Multiline JSON) stored in Hive Style Partitioned Google Storage Bucket
Craft Aggregate Queries on Telecom Data and Schedule queries for frequency based execution
Starbucks Data Analysis & Visualization using BigQuery Data Canvas (Gemini AI assisted Query Generation)
Telecom Data Cleaning using BigQuery's AI Oriented Data Preparation Feature
Silver & Gold Layer Data Preparation with BigQuery Python Notebook using BigQuery Dataframe
Telecom Data Pipeline (Medallion Architecture) in BigQuery
Task 1 - Bronze Layer Data Preparation Execution
Task 2 - Silver Layer Notebook Execution
Task 3 - Gold Layer Notebook Execution
Loyality Program Real Time Data Ingestion in BigQuery Table From GCP Pub-Sub using Dataflow Pipeline
Project - Weather Forecast Data Processing (Industrial Project)
Tech Stack: Python, OpenWeather API, GCP Composer (Airflow), PySpark, Dataproc Serverless, BigQuery, Google Cloud Storage (GCS), GitHub, GitHub Actions

Challenges with traditional data lake storages?
What is open table format?
Challenges solved by open table formats?
What is small file problem?
How open table formats solved small file problem?
Apache Iceberg Overview
Apache Iceberg Architecture Overview
How Iceberg catalog works?
Metadata layer - Metadata files, Manifest lists and Manifest files
Data Layer - Data Files
Backend representation of Iceberg tables after CRUD operations
After Create Table Command
After Insert Command
After MERGE INTO / UPSERT Command
Copy-On-Write Approach (CoW)
Delete files
Positional Delete Files
Equality Delete Files
Merge-on-Read Approach (MoR)
How to choose between COW and MOR?
Behind the scene process while running Select commands on Iceberg tables
Create Table Command in Iceberg
Insert Command in Iceberg
Delete & Update Commands in Iceberg
Alter Command in Iceberg
Merge Query in Iceberg
Time travel Query (Read) in Iceberg
Compaction in Iceberg
Data pipeline with Iceberg on AWS & Snowflake (Case Study)
Medallion Architecture with Iceberg (Case Study)
Apache Hudi Overview
Apache Hudi Architecture
Data Sources
Hudi Core - ACID Guarantees, Incremental Pipelines, Multimodal Indexes, Managed Tables
Lakehouse Platform
Metadata
Data Sinks
Apache Hudi Storage Layout - Base Path, Meta Path, Partition Paths, Data Files
Apache Hudi Query Types
Snapshot Queries
Time Travel Queries
Read Optimized Queries (Only MoR tables)
Incremental Queries (Latest State)
Incremental Queries(CDC)
Create Table Command in Hudi
Alter Command in Hudi
Insert, Update & Delete Command in Hudi
Merge Query in Hudi
Time Travel QUERIES in Hudi
Case Study of Data pipeline of Apna with Hudi
Develop and execute PySpark application in GCP Dataproc environment for Write
Configure Glue catalog for Iceberg tables in Spark session along with GCS warehouse path
Build dummy dataframe and use createOrReplace() dataframe v2 api to create Iceberg table with partitions & other table properties
Perform append() operation on Iceberg table
Perform overwritePartitions() operation on Iceberg table
Perform merge query on Iceberg table
Understanding of iceberg metadata file after each write operation
Develop and execute PySpark application in GCP Dataproc environment for Read
Read iceberg table data for latest snapshot
Query snapshot information of Iceberg table
Read iceberg table for specific snapshot_id
Read iceberg table for specific committed timestamp

✅ Class - 1

What is Trino?
Features of Trino
Trino Architecture
Responsibilities of the coordinator
Responsibilities of workers
Process of logical plan making in Trino & optimizations
Process of Physical plan making in Trino & optimizations
Stages, Shuffles & Tasks
What is data split and how does Trino creates data splits for different connectors
Stage scheduling in Trino : All-at-Once Scheduling , Phased Scheduling
Picking Nodes for Task Scheduling
Leaf Tasks (Reading Data from Source)
Intermediate Tasks (Processing Stage Outputs)
Lazy Split Assignment in Trino
Worker Task Scheduling in Trino
Shuffle Buffers in Trino
Writes to External Files in Trino
Memory Limitations in Trino
Operating on Files in Trino
Why use Trino if we already have Spark?
Setup Trino on GCP Dataproc Cluster
Trino UI Walkthrough
Execute Trino queries using Trino CLI
Trino Queries
SHOW CATALOGS
HOW SCHEMA
SHOW TABLES
Create tables query in Trino to query CSV data from Google Cloud Storage path - 2 Million Rows
How to register new connector in Trino setup to query data from different data sources?
MongoDB Connector Setup to query MongoDB collections from Trino

✅ Class - 2

Project 1: Unified Stock Trading Analytics Platform (Production-Grade Lakehouse)(Industrial Project)

Tech Stack: MongoDB Atlas, BigQuery, GCS, PySpark, Airflow, Dataproc, Hive Metastore, Trino, Python (Faker, Pymongo, Pandas), SQL

Azure Services Covered

Azure Blob Storage, Azure Functions, Azure Role-Based Access Control (RBAC), Azure Virtual Machines (VMs), Azure Logic Apps, Azure Service Bus (Queue, Topics), Azure Key Vault

Azure Stream Data Analytics, Azure Event Hubs, Azure Cosmos DB, Azure Data Factory (ADF), Azure SQL Database, Azure Synapse, Azure Delta Lake, Azure DevOPS

✅ Class - 1

Setup Free Trial Azure Cloud Account
Setup VSCode IDE With Azure Extensions
Connect VSCode IDE With Azure Account
Azure Blob Storage Fundamentals
Azure Data Lake Storage Fundamentals
Setup Storage Account in Azure
Create Blob Container in Storage Account
Create ADLS with hierarchical namespace
Azure CLI Setup in local machine and Blob/ADLS service related commands
Read csv file from Azure Blob using Python
Azure Function & Azure Function App Fundamentals
Create Azure Function App from VSCode
Create & Setup HTTP Trigger based Azure function
Local testing of Azure function in VSCode
Deploy HTTP Trigger based Azure function in Function App through VSCode
Create & Setup Blob Trigger based Azure function
Local testing of Azure function in VSCode
Deploy Blob Trigger based Azure function in Function App through VSCode

✅ Class - 2

Azure Service Bus Fundamentals - Topics & Queues
Create Service Bus Namespace
Create Topic in Service Bus & It's detailed configurations
Write messages in Topic from Azure Console
Subscriber setup for Topic and Read (Peek and Receive Mode) messages from Azure Console
Azure Function With Service Bus Topic Trigger
Python script to publish mock data in topic
Create Azure Function from VSCode which will get triggered as soon as message is published in Topic
Azure Function will consume message through subscription (With filter enabled for High priority records) and process it
Deploy Azure Function in Azure Function App through VSCode
Create Queue in Service Bus & It's detailed configurations
Write messages in Queue from Azure Console
Azure Function With Service Bus Queue Trigger
Python script to publish mock data in Queue
Create Azure Function from VSCode which will get triggered as soon as message is published in Queue
Azure Function will consume message from Queue and process it
Deploy Azure Function in Azure Function App through VSCode
Azure Logic App Fundamentals
Create & Setup Logic App Service from Azure Console
Logic App to send automated Email notification upon file arrival in Azure Blob Container
Create Azure Storage Account & Blob Container
Create Azure Function With Blob Trigger from VSCode
Create Topic in Azure Service Bus
Azure Function will publish object create notification of blob into Topic
Create Logic App from Visual Builder with "Message in Topic As Trigger" & "Send Email From Outlook As Action"

✅ Class - 3

Azure Virtual Machine Setup
SSH based login from terminal in VM
Azure SQL Database fundamentals
Setup Azure SQL database from Azure Portal
SQL commands for Azure SQL database using Query Editor
Azure Key-Vault basics
How to create and store secrets in Azure Key-Vault
Access & Manipulate data in SQL database using Python script (Access secrets from Key-Vault)
Azure Event Hub Fundamentals
Difference between Azure Event Hub & Kafka
Setup Azure event hub namespace from Azure Portal
Create event hub in namespace
Publish data in event hub using Python script
Create consumer group in event hub
Consume data from event hub using Python script
Azure Stream Analytics job basics
Create Inputs, Outputs & Query in Stream Analytics Job
Setup Stream Analytics job to read data from event hub in real time, apply window based aggregations and ingest data in SQL database

✅ Class - 4

Azure CosmosDB Fundamentals
Setup CosmosDB Account from Azure Portal
Create dataset and containers in CosmosDB
Connect & Manipulate data in CosmosDB using Python
Azure Data Factory Fundamentals
Create first Azure Data Factory Namespace
Connect ADF with Github
Setup Linked Services for ADLS & Snowflake in ADF
Create CSV file and Snowflake table datasets
Create DataFlow in ADF
Perform source, filter, aggregate & sink transformations in ADF
Debug Dataflow
Create Pipeline in ADF with DataFlow activity
Manual trigger for ADF pipeline
Azure Synapse Analytics fundamentals and Architecture
Setup first Azure Synpase Analytics workspace
Different features of Synapse Studio
Work with in-built and dedicated SQL pools
SQL Database under workspace
Read external files using in-built SQL pool queries
Create external tables using in-built sql pool
Work with SQL worksheet and create Schema, Internal & External tables in dedicated sql pool
Setup Spark SQL Pool
Create PySpark notebook in Synapse and write data in delta tables in Lake Database
Create linked services for ADLS and CosmosDB in Synapse
Data ingestion pipeline in Synapse Analytics

✅ Project - 1: Flight Booking Data Pipeline with Airflow & CICD (Covered In Module 6)

Tech Stack - GitHub, GitHub Actions, Google Storage, PySpark, Dataproc Serverless, Airflow, BigQuery
✅ Project - 2: E-commerce Event-Driven Data Pipeline (Covered In Module 7)

Tech Stack - Databricks, PySpark, Delta Lake, Databricks Volumes, Databricks Workflows, GitHub
✅ Project - 3: Travel Booking SCD2 Data Warehouse (Covered In Module 7)

Tech Stack - Databricks, PySpark, Delta Lake, Unity Catalog, PyDeequ (For Data Quality Checks), Databricks Volumes, Databricks Workflows
✅ Project - 4: Healthcare DLT Medallion Pipeline (Covered In Module 7)

Tech Stack - Databricks DLT, Delta Lake, SQL, Unity Catalog, Expectations, Databricks Workflows
✅ Project - 5: UPI Transactions CDC Streaming Analytics (Covered In Module 7)

Tech Stack - Databricks, PySpark Structured Streaming, Delta Lake (Change Data Feed), Unity Catalog
✅ Project - 6: News Data Analysis with Event-Driven Incremental Load in Snowflake Table (Covered In Module 9)

Tech Stack: Airflow, Google Cloud Storage, Python, Snowflake
✅ Project - 7: Car Rental Data Batch Ingestion with SCD2 Merge in Snowflake Table (Covered In Module 9)

Tech Stack: Python, PySpark, GCP Dataproc, Airflow, Snowflake
✅ Project - 8: Movie Booking CDC data real time aggregation in Snowflake Dynamic Table (Covered In Module 9)

Tech Stack: Python, Snowflake Dynamic Table, Snowflake Stream, Snowflake Tasks, Streamlit
✅ Project - 9: Weather Forecast Data Processing (Covered In Module 9)

Tech Stack: Python, OpenWeather API, GCP Composer (Airflow), PySpark, Dataproc Serverless, BigQuery, Google Cloud Storage (GCS), GitHub, GitHub Actions
✅ Project - 10: Unified Stock Trading Analytics Platform (Production-Grade Lakehouse) (Covered In Module 11)

Tech Stack: MongoDB Atlas, BigQuery, GCS, PySpark, Airflow, Dataproc, Hive Metastore, Trino, Python (Faker, Pymongo, Pandas), SQL
✅ Project - 11: Fintech SQL Data Migration into Azure Portal

Tech Stack: Python, Azure SQL Database, SQL, Azure Synapse Analytics, ADLS, PySpark, Delta Tables, Azure Logic Apps
✅ Project - 12: AirBnB CDC Ingestion Pipeline

Tech Stack: Python, Azure ADLS, Azure Cosmos DB, Azure Data Factory (ADF), Azure Synapse Analytics, SQL, n8n, Agentic AI, LLM (OpenAI)
✅ Project - 13: BookMyShow Real-Time Data Processing Pipeline

Tech Stack: Python, Azure Event Hub, Azure Stream Analytics, SQL, Azure Synapse Analytics, n8n, Agentic AI, LLM (OpenAI), JavaScript
✅ Project - 14: Airlines Incremental Data Processing Pipeline with CI/CD on Azure DevOps

Tech Stack: Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), GitHub, Azure DevOps
✅ Project - 15: Ride Analytics Data Pipeline (Industrial Project – Inspired by Uber, Ola, Rapido)

Tech Stack: Python, GCP Dataproc, Airflow, GCS, BigQuery, FastAPI, Docker, GCP Cloud Run, GCP Artifact Registry, GitHub, GitHub Actions, PyTest, n8n, Telegram, JavaScript

Attention Seeking Resume Preparation and Interview Strategies
Strategies To Crack Tech Interviews
Linkedin Profile Making
How To Expand Your Professional Network On Linkedin
How To Use Various Job Portals
How To Approach For Referrals

Course Schedule

Mode of the Course:

Self Paced

Course Duration:

180 Hours

Total Sessions:

40

Total Projects:

15

Validity:

1 Year (Starting From The Date Of Enrollment)

Programming Language Used:

Python

Prerequisite:

Basic Python

⚠️ Important Notice :

The video may not work on Linux due to DRM restrictions. It is only accessible on Chrome when using Windows or macOS.

Workaround: To access the video on Linux, you can create a Windows virtual machine (VM) and watch the video through the VM. Alternatively, you can use our Android or iOS application to view the video on your mobile device.

Instructor

Data Engineering With AWS

Shashank Mishra

AI Data Engineer Ex-Amazon, Expedia, Paytm, Prophecy & McKinsey & Company

Shashank Mishra is a AI Data Engineer with 10+ years of experience building scalable data platforms, real-time data pipelines, cloud-native architectures, and AI-powered data solutions for global organizations.

An MCA graduate from NIT Allahabad, he specializes in modern multi-cloud Data Engineering and its growing intersection with AI. His areas of focus include RAG pipelines, Agentic AI platforms, AI-powered analytics, intelligent data workflows, and production-grade AI applications.

Beyond industry work, Shashank is the educator behind GrowDataSkills, where he creates practical, project-driven content for Data & AI professionals. His community includes 185K+ YouTube subscribers, 25K+ Instagram followers, and 190K+ LinkedIn professionals.

Having mentored 32,000+ learners, he focuses on helping professionals build industry-ready skills, transition into Data Engineering from diverse backgrounds, and prepare for careers at leading technology companies.

Complete Data Engineering With Azure - Basic To Advance (Self Paced)

INR 9000

Copyright © 2026 Regex Data Learning Pvt Ltd. All Rights Reserved.