AWS Data Engineering Course — Basic to Advanced

Light
Dark

Complete AI Data Engineering With AWS ( 6.0 )
Basic To Advance

By Shashank In Grow Data Skills

Live Classes (In Progress)
English
Completion Certificate

Demo Class Video

1 Year

Validity (From the date of Enrollment)

180 Hours

Duration

40

Sessions

15

Projects

INR 13000

📢 Admissions Open

Buy Now

This course includes

🎁 FREE Python Crash Course
Content Duration180 Hours
Total Video Sessions40
Total Industry Projects15
AWS Cloud
Quality Assignments & quizzes after each module
Interview preparation guide
Dedicated placement assistance
Resume & Linkedin profile making
Doubt solving in live classes and offline doubt support on private discord group
Certificate of completion

Tech stack you'll learn

Course Content

✅ Class - 1

What is Database?
What is DBMS?
Transactional Databases vs NoSQL Databases
What is RDBMS?
What is SQL?
MySQL Workbench Setup and Connection
DDL, DML, DQL, DCL Commands
SHOW Databases
CREATE Database
DROP Database
CREATE Table
MySQL DataTypes
INSERT Command Syntaxes
Integrity Constraints
NOT NULL
DEFAULT
UNIQUE
CHECK
PRIMARY Key
Foreign Key
Apply NOT NULL, DEFAULT, UNIQUE, CHECK constraints and test violation
Customized name for Integrity Constraints

✅ Class - 2

ALTER Command
Add Column
Modify Column
Drop Column
Rename Column
Add Constraint
Drop Constraint
Primary Key vs Foreign Key
Create Primary Key
Referential Integrity With Foreign Key Setup
SELECT Command
Count Function
Distinct Function
Aliases and Derived Columns in Select Command
WHERE Clause
Comparison Operators
Logical Operators
BETWEEN Operator
UPDATE Command
Conditional UPDATE
Multicolumn UPDATE
DELETE Command
Conditional DELETE
DROP Command
TRUNCATE Command
DELETE vs DROP vs TRUNCATE
LIMIT
ORDER BY Clause
Multicolumn ORDER BY
LIKE Operation

✅ Class - 3

IS NULL
IS NOT NULL
Uses of IS NULL and IS NOT NULL with SUM Aggregate for Data Quality Checks
GROUP BY Clause
GROUP BY with Single Grouping Key
GROUP BY with Multiple Grouping Keys
GROUP BY with WHERE Clause
HAVING Clause
GROUP_CONCAT Application
GROUP ROLLUP Implementation and Practical
Scalar SubQuery
Correlated SubQuery
IN and NOT IN Application
CASE WHEN Clause
Conditional Aggregates with CASE WHEN and GROUP BY
Uber SQL Interview Question
SQL Joins
Inner Join
Left Join
Right Join
Cross Join

✅ Class - 4

Self Join Practical
EXISTS Operation
NOT EXISTS Operation
ANY Operation
ALL Operation
UNION
UNION ALL
Uber SQL Interview Question
Window Functions Fundamentals
Running SUM
ROW_NUMBER()
RANK()
DENSE_RANK()
LAG()
LEAD()
NTILE()
Frame Clause Fundamentals
ROWS BETWEEN
RANGE BETWEEN
Running Weekly SUM Problem
FIRST_VALUE()
LAST_VALUE()
Iterative CTE
CTE Chaining
Recursive CTE
Views
User Defined Functions (UDF)

✅ Class - 1

Big Data Fundamentals
5 V’s of Big Data
Distributed Computation
Distributed Storage
Cluster and Commodity Hardware
File Formats - CSV, JSON, Parquet, ORC, Avro
Types of Data - Structured, Semi Structured, Unstructured
History of Hadoop
Hadoop Architecture and Components

✅ Class - 2

Map-Reduce Architecture
YARN Architecture

✅ Class - 1

Kafka Architecture
Brokers, Topics, Partitions
Producer, Consumer
Offset Management
Replicas
Commits
Sync and Async Commits
How Does Consumer Group Work?
Rebalancing

✅ Class - 2

Where Kafka Fits in Data Pipeline Architecture?
Confluent Kafka Setup
Kafka Cluster Setup
Confluent Kafka UI Walkthrough
Kafka Topic Creation and Configuration
Data Contract Creation
Schema Registry Setup
API Key Creation for Kafka Cluster
API Key Creation for Schema Registry
Avro Schema Setup for Key
Avro Schema Setup for Value
Schema Evolution and Schema Versioning
Python Producer Code Setup
Publishing Avro Serialized Data into Kafka Topic
Python Consumer Code Setup
Consuming Avro Deserialized Data from Kafka Topic with Sync Commit
Consumer Group Functioning

✅ Class - 3

Data Lake vs Data Warehouse
What is an Open Table Format?
Overview of Apache Iceberg in Lakehouse Architectures
Tableflow Feature in Confluent Cloud
Overview of Kafka Connect (Source & Sink)
Understanding KStream and Its Deployment Architecture
Introduction to ksqlDB
ksqlDB Setup in Confluent Cloud
Creating Streams on Top of Kafka Topics
Executing Push Queries in ksqlDB
Creating KTables in ksqlDB
Windowing Concepts – Sliding & Tumbling Windows
Building Window-Based Aggregated KTables
Handling Late Events in Window-Based Aggregations
Real-Time E-commerce Streaming Data Ingestion
Window-Based Stream Joins in ksqlDB
Building Aggregated KTables for Real-Time Reporting

✅ Class - 1

CAP Theorem
MongoDB and MongoDB Atlas
MongoDB vs Relational Database
MongoDB Features, Use Cases and Architecture
Node, Data Center, Cluster
Data Replication
Write and Read Operations
Indexing

✅ Class - 2

Where MongoDB Fits in Data Pipelines
When to Use MongoDB
When Not to Use MongoDB
MongoDB Atlas Free Tier Setup
MongoDB Atlas Cluster Creation on AWS
MongoDB Atlas Components Overview
MongoDB Atlas UI Walkthrough
Create Database in MongoDB
Create Collection in MongoDB
MongoDB Aggregation Framework
Schema Analyzer
Indexing in MongoDB
Compound Indexing
Collection Validation Rules
MongoDB Compass Setup
Connect MongoDB Atlas with MongoDB Compass
Import Airbnb Reviews JSON Dataset into MongoDB
AI-Assisted Query Generation in MongoDB
Find Queries in MongoDB
Aggregation Queries in MongoDB
Connect MongoDB with Python
Insert, Find & Aggregate Data Using Python
E-Commerce Checkout System Design
Frontend Development (HTML, CSS & JavaScript)
Backend Development with FastAPI & MongoDB
Checkout, Payment & Cancellation Event Handling
FastAPI Checkout API Implementation
MongoDB Atlas Source Connector with Confluent Kafka
MongoDB CDC (Change Data Capture) to Kafka
Real-Time Streaming of Insert, Update & Delete Events to Kafka

✅ Class 1 - Spark Fundamentals & Internals

Problems With Hadoop MapReduce
What is Apache Spark?
Spark Features and Ecosystem
RDD and RDD Properties
Data Partitioning in Spark
Transformations and Actions
Narrow vs Wide Transformations
Read/Write Operations in Spark
Lazy Evaluation
Lineage Graph (DAG)
Spark Web UI Walkthrough & DAG Understanding
Creation Of Job, Stage and Tasks
Spark Architecture And Components
Standalone and YARN Cluster Manager
Deployment Modes - Cluster & Client
Spark Job Internals

✅ Class 2 - Optimization & Resource Planning

Persist and Caching
Storage Levels
Data Skewness
Techniques To Handle Data Skewness
Repartition vs Coalesce
Key Salting
RDD vs DataFrame vs Dataset
Spark-Submit Utility
Spark Memory Management
Executor Memory Components
Dynamic Occupancy Mechanism
Processing 1 TB Data with Spark
Resource Allocation Case Studies & Infra Capacity Planning
Broadcast Variables and Accumulators
Spark Planning Breakdown - Analyzed Plan, Logical Plan, Physical Plan
Spark Failure Scenarios and Resolutions
Out-of-Memory Failures & Fixes
Code and Resource-Level Optimizations
Best Practices For Spark Applications

✅ Class 3 - PySpark Practical

AWS Cloud Free Tier Setup
Setup Billing Budget Alert in AWS
Difference Between AWS EMR Dedicated Cluster and EMR Serverless
Primary Node, Core Node, and Task Node in EMR Cluster
Instance Types in EMR
AWS EC2 Overview
Networking Setup for EMR Cluster
What Are VPC, Subnet, and Security Group?
Security Configuration and EC2 Key Pair Setup
IAM Users, IAM Roles, and Policies in AWS
Setup Amazon EMR Service Role
Setup EC2 Instance Profile for Amazon EMR
SSH Connectivity to EMR Primary Node
SSH Tunnel Setup to Access EMR and Application UIs
HDFS NameNode UI Walkthrough
YARN ResourceManager UI Walkthrough
Spark History Server UI Walkthrough
JupyterHub Access
AWS S3 Bucket and Object Creation
Spark Notebook Setup in JupyterHub
SparkSession Creation
Create a Dummy DataFrame in Spark with Columns
Print DataFrame Schema
Show Action on Spark DataFrame
Define Schema in Spark Using StructType, StructField, and Native Data Types
Read a CSV File from Amazon S3 Using Spark CSV Properties
Select Transformation on Spark DataFrame
withColumn Transformation on Spark DataFrame

✅ Class 4 - PySpark Practical

Blinkit Dataset Setup on Amazon S3
Creating Spark DataFrames from CSV Files in S3
withColumn Transformation
withColumnRenamed Transformation
Filter Transformation
Drop Transformation
DropDuplicates Transformation
Full Row vs Column-Based Deduplication
How DropDuplicates Works & Data Shuffling
OrderBy Transformation
NULL Ordering in Spark (NULLS FIRST/LAST)
GroupBy & Aggregation Transformations
Shuffle & Partial Aggregation in GroupBy
Understanding DataFrame Partitions
spark.sql.shuffle.partitions Configuration
Adaptive Query Execution (AQE)
AQE Optimizations & Physical Plan Improvements
AQE Impact on Shuffle Partitions
Case When Transformation
Join Transformation
Window Functions
Spark SQL Fundamentals
Creating Temporary Views & Running Spark SQL Queries
Parquet File Format Fundamentals
Parquet File Internals & Architecture
Row Groups & Row Group Sizing
Controlling Records per Parquet File & Row Groups
Impact of Row Groups on Spark Partitioning
Column Chunks, Pages & Compression
Footer Metadata, Column Pruning & Predicate Pushdown
When Not to Use Parquet
Splittability & Spark Parallelism
Spark File Partitioning & maxPartitionBytes
Join Strategies in Spark
Broadcast Hash Join (BHJ)
Broadcasting, Hash Tables & autoBroadcastJoinThreshold
Why Broadcast Hash Join Avoids Data Shuffling
Using explain() to Analyze Spark Physical Plans
Impact of AQE on Spark Physical Execution Plans
Spark-submit Execution

✅ Class 5 - Spark Structured Streaming

Broadcast Hash Join
How Broadcast Hash Join Works in Spark
Broadcast Hash Join Configuration Properties
Shuffle Hash Join
How Shuffle Hash Join Works in Spark
Shuffle Hash Join Configuration Properties
Implementing Shuffle Hash Join
Physical Plan Analysis for Shuffle Hash Join
Spark Execution Plan With and Without AQE
Understanding Shuffle Partitions After Wide Transformations
How Adaptive Query Execution (AQE) Reduces Shuffle Partitions
How Shuffle Partitions Are Created
Configuring Shuffle Partitions in Spark
AQE Coalesce Configuration Properties
Concept of Merging Two Sorted Arrays
Sort Merge Join in Spark
How Sort Merge Join Works
Properties to Enable Sort Merge Join
When Spark Uses Sort Merge Join
Writing Spark DataFrames as CSV to Amazon S3
Write Modes in Spark
Why Multiple Output Files Are Generated
Writing Spark DataFrames as Parquet to Amazon S3
Partitioning Parquet Files While Writing to Amazon S3
Controlling Parquet Row Group Size and File Records
Repartition vs Coalesce
How Repartition Works
Shuffle Behavior During Repartition
How Coalesce Works
Understanding the Coalesce(1) Performance Trap
Writing a Single CSV File Using Coalesce
When to Use Repartition vs Coalesce
User Defined Functions (UDFs) in Spark
How UDFs Work in Spark
Creating and Registering UDFs
Why UDFs Impact Spark Performance
Why UDFs Are Slow
Evaluating Parquet Files
Analyzing Parquet Row Groups and Metadata
Calculating Optimal Spark Partitions Based on Data Size
Formula for Partition Calculation
Understanding the Small File Problem
Partition Creation with Small Input Files
Understanding OpenCostInBytes
Spark Web UI Walkthrough Using Real Jobs
Executing an Unoptimized Spark Job
Mental Model of an Unoptimized Spark Job
Spark Submit Utility and Resource Configuration
Designing Spark DAGs from Code
Understanding Jobs, Stages and Tasks
Job Creation and DAG Visualization in Spark Web UI
Stage Creation and Analysis in Spark Web UI
Task Creation and Analysis in Spark Web UI
Dynamic Resource Allocation in Spark
Breaking Down the Spark Physical Plan
Input, Output and Shuffle Metrics
Behavior of an Unoptimized Spark Job
Stage-Level Metrics in Spark Web UI
Understanding WholeStageCodeGen
Debugging Spark Tasks Using Spark Web UI
Task-Level Analysis in Spark Web UI
Task Locality Levels: PROCESS_LOCAL, NODE_LOCAL, RACK_LOCAL & ANY
Important Task Metrics: GC Time, Input Size, Shuffle Read & Write Metrics
Spark Job Resource Allocation and Timeline Analysis

✅ Class 6 - Stateful Streaming

Spark Web UI Deep Dive for an Unoptimized Job
Input, Output, Shuffle Read and Shuffle Write Metrics
When Input, Output and Shuffle Metrics Appear
Stage-Level Metrics in Spark Web UI
Spark Job and Executor Timelines
Scheduler Delay Metric
Task Deserialization Time
Shuffle Remote Reads
What Is Garbage Collection and How Does It Work?
Rule of Thumb for Garbage Collection
Memory Pressure Due to Garbage Collection
Data Spilling Due to Garbage Collection
Impact of Garbage Collection on Spark Jobs and Resources
How to Optimize Garbage Collection
Executor Fine-Tuning for Garbage Collection
Peak Execution Memory Metric
Deep Dive into Spark Job Planning
Parsed, Analyzed and Optimized Logical Plans
Spark Physical Plan Deep Dive
Spark Optimized Job Execution
Changes in Jobs, Stages and Tasks After Optimization
Changes in Spark Physical Plan After Optimization
Spark Web UI Deep Dive for an Optimized Job
Performance Improvements After Spark Optimizations
Introduction to Stream Processing
Stream Processing Architecture
Word Count with Spark Structured Streaming
How to Read Streaming Data in Spark
Micro-Batches in Spark Structured Streaming
WriteStream and awaitTermination in Structured Streaming
Output Modes: Complete, Append and Update
Triggers in Spark Structured Streaming
Types of Structured Streaming Triggers
Stateless Processing in Spark Structured Streaming
Stateful Processing in Spark Structured Streaming
How Stateful Processing Works
How Spark Manages In-Memory State
Custom State Management in Spark
RocksDB for State Management in Spark
Configuring and Enabling RocksDB State Store
On-Heap Memory vs Off-Heap Memory
Spark Memory Overhead Configuration
When to Configure Memory Overhead
Checkpointing in Spark Structured Streaming
Why Checkpointing Is Required and How to Configure It
Offsets, Commits and State Store
At-Least-Once Delivery Semantics
Exactly-Once Delivery Semantics
End-to-End Exactly-Once Processing in Structured Streaming
Idempotent Sinks
Transactional Sinks
Windowing in Spark Structured Streaming
Tumbling and Sliding Windows
Window-Based Aggregations and Joins
Watermarking in Spark Structured Streaming
Handling Late-Arriving Events
Practical Watermarking for Late-Arriving Events

✅ Class 7 - PySpark

Stateless Spark Structured Streaming Pipeline
Kafka Topic Setup & Mock Data Generation
Reading Kafka Streams Using Spark ReadStream
Stateless Data Transformations in Structured Streaming
WriteStream with Update Mode & Console Sink
Checkpointing in Spark Structured Streaming
Spark Web UI for Structured Streaming Jobs
YARN Log Analysis for Streaming Applications
Executing Structured Streaming via Spark Submit
Stateful GroupBy Structured Streaming Pipeline
GroupBy Aggregation with Complete Output Mode
Checkpoint Recovery & Stateful Processing
Stateful GroupBy Output Validation Using YARN Logs
Impact Analysis After Deleting Checkpoint Data
Window & Watermark Based Structured Streaming
Tumbling Window Aggregation with Watermarking
Update Mode for Window Aggregations
Validating Window & Watermark Aggregations
Custom Stateful Streaming with applyInPandasWithState
Kafka Topics for Orders, Payments & DLQs
Managing In-Memory State Using Key-Value Store
State Expiration (TTL) & Dead Letter Queue (DLQ)
Order & Payment Matching Using Stateful Processing
Publishing Matched Events & Cleaning State
Validating Output Using Confluent Cloud UI

✅ Class - 1

Orchestration in Big Data
Dependency Management in Data Pipeline Design
Airflow Fundamentals
Airflow Architecture and Components
Operators in Airflow
Writing Airflow DAG Scripts
Attribute Description

✅ Class - 2

Complete setup of Airflow on AWS (AWS Managed Airflow)
Create S3 bucket to setup Dag folder and configure it in Airflow environment
Plugins in Airflow and how to configure it in Airflow environment
Setup external dependencies in requirements.txt and configure it in Airflow environment
VPC Setup through AWS Cloudformation for Airflow
Environment class for Airflow
IAM Role setup for Airflow environment
Attach S3 access policy
Complete overview of Airflow Web UI
Dag run history & observability
Airflow providers
Reatil Data Pipeline Orchestration via Airflow
Pipeline Architecture overview
Create Databse and Collection in MongoDB Atlas
Generate mock json data through script
Ingest json data into MongoDB collection through Python script
PySpark job overview to read the data from S3 bucket, transform the data, aggregation to prepare summary and load aggregated data back into S3 bucket
Create IAM user with admin permissions, genaret Access and Secret keys
Setup AWS CLI to connect local machine to AWS Account
Install boto3 package locally
Purpose of Xcom in Airflow
How airflow operators exchange data using Xcom
Push & Pull data from Xcom with airflow task
How to create Key-Value based Variables from Json file in Airflow Web UI
How to access variables in Airflow Dag
PythonOperator task execution to fetch data from MongoDB collection
PythonOperator task to get extracted data from Xcom and write it in json form in S3 bucket using S3Hook
Create TaskGroup (Which combines 3 parallel tasks) for data quality checks
Submit batch spark job on EMR Serverless Aplication using EMRServerlessStartJobOperator task
Spark job driver configuration and settings
Jinja expressions in Ariflow Dag
Create TaskGroup (Which combines 2 parallel tasks) for data quality checks on processed data
Sequential and Parallel dependencies of tasks in Airflow Dag
Upload Airflow Dag in Dag folder
Overview of imported Dag on Airflow Web UI
Whitelist Airflow Environment IP Addresses on MongoDB Network
Manual trigger of Airflow Dag from UI with custom operators
Backfilling options and parameters for Airflow Dag
Airflow Dag execution and task log analysis
Airflow task failures, log analysis and fixes
UI navigation for Xcom, Code, Run History
Fundamental of Airflow Assets and How does it work?
Payment Settlement Data Pipeline Orchestration via Airflow
Create Producer Airflow Dag to generate mock data for Payments and Merchants
Create and configure Assests in Airflow Dag
PythonOperator task to write payments data on S3 path using S3Hook and update RAW_TRANSACTIONS_ASSET
PythonOperator task to write merchants data on S3 path using S3Hook and update MERCHANT_REFERENCE_ASSET
PythonOperator task to write summary marker file on S3 path using S3Hook and update PAYMENT_DATA_READY_ASSET
Create sequential and parallel task dependencies
Create Consumer Dag which will get triggered as soon as PAYMENT_DATA_READY_ASSET is updated
Configure trigger of consumer dag as schedule=[PAYMENT_DATA_READY_ASSET]
PythonOperator task to read params and Airflow Variables
PythonOperator tasl to validate the raw transaction file before business checks
TaskGroup to group independent monitoring checks that can run in parallel
BranchPythonOperator task to choose the report path based on monitoring results
PythonOperator with outlet to write fraud alerts and updates the fraud report Asset
PythonOperator to write a normal-activity report if no thresholds are breached
PythonOperator with outlet to write settlement totals and updates the settlement Asset
PythonOperator with outlet to final summary must run even when one branch is skipped
Setup sequential and parallel task dependencies
Execute Producer Dag manually
One asset is updated Consumer Dag gets triggered automatically

✅ Class - 1

AI as a Productivity Multiplier for Data Engineers
Practical Use Cases
What AI Can Generate?
What Data Engineers Still Validate
Common AI Tools
How Data Engineers Create & Enable Production AI Systems
What is AI?
What is Machine Learning?
Traditional Programming vs Machine Learning
Features, Label, Model, Weights, Training, Testing & Overfitting in Machine Learning
Types of Machine Learning
Supervised Learning – Regression & Classification
Unsupervised Learning
Reinforcement Learning
What is a Neural Network?
How Does a Neural Network Work?
Neuron, Hidden Layer & Neural Network
Weights, Bias & Activation Function
What is Deep Learning?
Where Deep Learning Became Important
What is NLP (Natural Language Processing)?
How NLP Has Evolved
What is Generative AI?
What Exactly is an LLM?
Practical LLM Example
Why "Large" in Large Language Models?
Industry-Wide Popular LLMs
Closed / Proprietary LLMs
Open / Open-Weight LLMs
Token
Embedding
Vector & Vector Database
Semantic Search
Prompting
System Prompt
Context
User Prompt
Temperature
Context Window
How an LLM Works When You Send a Prompt?
Tokenization
Embeddings
Transformer Layers
Token Prediction
Complete LLM Architecture & Workflow
Input Prompt
Tokenization
Embeddings + Position Information
Transformer Blocks – Masked Self-Attention & Feed Forward Network
Next-Token Prediction

✅ Class - 2

What is RAG (Retrieval-Augmented Generation)?
Problem with a Standalone LLM
RAG Approach
RAG Pipeline Main Components – Knowledge / Indexing Pipeline & Query / Retrieval Pipeline
Knowledge / Indexing Pipeline Components – Data Sources, Document Processing, Chunking, Embedding Model & Vector Store / Search Index
Query / Retrieval Pipeline Components – Retriever, Reranker, Prompt Builder & LLM
RAG Pipeline Flow Representation
Chunking Strategy in RAG
Fixed-Size Chunking
Recursive Chunking
Semantic Chunking
Overlap Chunking
Chunking Design Trade-Offs
Reranking in RAG
Why Do We Need Reranking?
Reranking Example
RAG Architectures
Naive / Basic RAG
Advanced RAG
Conversational RAG
Agentic RAG
Tooling in LLM Applications
Why Tools Are Needed
How LLM Tool Calling Works
What are AI Agents?
AI Agents in Daily Life
AI Agents in Data Engineering Use Cases
AI Agents Flow & Processes
What is Agent Architecture?
Agent Architecture Components – User Request, LLM, Planner, Tool Selector, Tool Execution Layer, Observation, State / Memory, Guardrails & Final Response
What is an Agent Framework?
What Agent Frameworks Help With
Frequently Used Frameworks – LangGraph, OpenAI Agents SDK & CrewAI
What is LangChain?
What is LangGraph?
What is LangSmith?
What is MCP (Model Context Protocol)?
How Things Work Without MCP vs With MCP
What Can an MCP Server Expose?
MCP Flow
MCP-Based Agent Flow for Data Engineering Use Case
How to Configure Swiggy MCP Server in ChatGPT
Utilize Swiggy MCP Server & Its Tooling for Personalized Conversations and Actions
Weather-Based Daily Planning Agent Implementation Using LangGraph
LangGraph to Define the Workflow as a Graph
Ollama to Run a Local LLM
Open-Meteo to Fetch Free Weather Data Without an API Key
LangGraph Studio to Visually Run, Inspect & Debug the Graph

✅ Class - 1

OLAP vs OLTP
What is a Data Warehouse?
Data Warehouse vs Data Lake vs Data Mart
Fact Tables
Dimension Tables
Slowly Changing Dimensions
Types of SCDs
Star Schema
Snowflake Schema
Galaxy Schema

✅ Class - 2

Case Study 1: Expedia Advanced Data Warehousing & Modeling
Galaxy schema for travel domain
Hotels, flights, cars, payments and customer dimensions
Fact tables: search sessions, impressions, clicks, bookings, cancellations, payments, refunds
Conformed dimensions
Business metrics: GBV, take rate, cancellations, refund rates, average booking value, occupancy
SQL queries for KPIs
Case Study 2: Swiggy Advanced Data Warehousing & Modeling
Food delivery domain model
Facts: sessions, impressions, add-to-cart, order header, order items, delivery trips, cancellations, refunds, payments, rider shifts
Conformed dimensions
SCD2 handling for customers, restaurants, menu items and riders
Metrics: AOV, take rate, on-time delivery, cancellation, refund, repeat customers, rider utilization, SLA adherence

✅ Class - 1

Challenges with Traditional Data Lake Storage
Open Table Formats
Small File Problem
Iceberg Overview and Architecture
Iceberg Catalog
Metadata Files, Manifest Lists, Manifest Files
Data Files
Backend Representation after CRUD Operations
Create Table, Insert, Merge/Upsert
Copy-on-Write
Delete Files
Positional Delete Files
Equality Delete Files
Merge-on-Read
Choosing between CoW and MoR
Select Query Internals
Create, Insert, Delete, Update, Alter, Merge
Time Travel
Compaction
Iceberg on AWS and Snowflake Case Study
Medallion Architecture with Iceberg

✅ Class - 1

What is Databricks ?
Databricks On Cloud
Databricks Free Edition Complete Setup
What is Workspace ?
Serverless Workspace Setup
Serverless Compute and Default Storage In Serverless Workspace
How Databricks Manages Serverless Workspace
Classic Workspace Setup On Cloud
Classic Compute, Serverless Compute and Default Storage In Classic Workspace
Classic vs Serverless Compute : Cost Perspective
When Should We Choose Serverless?
When Should We Choose Classic Compute?
Production Decision Rule for Classic Compute vs Serverless
Databricks Controle Plane Walkthrough
Catalog Explorer in Databricks Controle Plane
Jobs & Pipelines in Databricks Controle Plane
Difference Between Serverless Compute and SQL Warehouse Compute
What is AI Search in Databricks Controle Plane ?
What is APPS in Databricks Controle Plane ?
What is Lakebase in Databricks Controle Plane ?
Basics of Genie Agent
Alerts in Databricks Controle Plane
Data Ingestion in Databricks Controle Plane
Lakeflow designer in Databricks Controle Plane
Playground in Databricks Controle Plane
MCPs and Agents in Databricks Controle Plane
Unity AI Gateway in Databricks Controle Plane
Models in Databricks Controle Plane
Serving in Databricks Controle Plane
Datalake vs Data Warehouse
What is Lakehouse ?
Lakehouse architecture
What is Unity Catalog ?
Why do we need Unity Catalog ?
What Unity Catalog provides ?
Core Architecture of Unity Catalog
Metastore
Catalog
Schema
Data & AI Objects
How many Unity Catalog Metastores can an account have?
How many Metastores can one Workspace use?
How many Catalogs can we create?
How many Catalogs can a Workspace see?
Create Managed Catalog In Databricks
Create External Location and Credential In Databricks to connect with AWS S3
Create External Catalog In Databricks
Create Schema in External Catalog
Create External Volume In Databricks
What is Delta Lake ?
What is Delta Table ?
Delta Format Architecture
Parquet Data Files
Delta Transaction Log — _delta_log
Checkpoint Files
Table Snapshot
PySpark Notebook Setup in Databricks
Read Parquet Data In PySpark from Databricks Volume
Use Genie Code to generated aggregated transformation
Write aggregated data in Delta Table
Pyhsical analysis of Delta format parquet data and _delta_log stored in AWS S3
Insert and Delete operation on Delta Table Data
What are Deletion Vectors?
Pyhsical analysis of Deletion Vectors
Time Travel Queries in Delta Tables

✅ Class - 2

Project 1: Agentic Uber Eats Marketplace Lakehouse Data Platform(Industrial Project)

Tech Stack: Databricks, Databricks Serverless Compute, Databricks Asset Bundles, Databricks Workflows, Unity Catalog, Unity Catalog Volumes, Delta Lake, Auto Loader, Medallion Architecture, PySpark, Spark SQL, Python, Delta MERGE, SCD Type 2, Star Schema Data Modeling, Fact Tables, Dimension Tables, Incremental Batch Processing, Native Data Quality Checks, Quarantine Tables, Liquid Clustering, Predictive Optimization, Delta UniForm, Databricks Genie, Conversational Analytics, GitHub, GitHub Actions, Databricks CLI, CI/CD, pytest, YAML, TOML, JSONL, CSV

Description: Built a production-style Uber Eats marketplace lakehouse on Databricks with incremental ingestion, governed medallion architecture, SCD Type 2 dimensions, optimized gold data models, native data quality checks, and CI/CD-based deployment. The platform also includes Genie-powered agentic conversational analytics for business KPI exploration and failed-record investigation across marketplace orders, payments, merchants, couriers, refunds, delivery SLA, and support operations.

✅ Class - 3

Project 2: ProductPulse AI - Retail Product Intelligence & Support Copilot(Industrial Project)

Tech Stack: Databricks, Unity Catalog, Delta Lake, Unity Catalog Volume, PySpark, Databricks Workflows, Lakebase Postgres, pgvector, Databricks SQL Warehouse, Databricks Apps, Streamlit, Databricks Asset Bundles, Databricks Foundation Model APIs, Qwen Embedding Model, Llama Foundation Model, LLM-Based Tool Planning, RAG Pipeline, PDF Processing

Description: Built an end-to-end production-style retail AI copilot that processes structured retail data and unstructured PDFs, generates embeddings using a Qwen embedding model, stores vectors and conversation memory in Lakebase, and uses a Llama foundation model with LLM-based tool planning to answer product, return, support, and metric questions through a Streamlit-powered Databricks App.

✅ Class - 1

Snowflake Free-Tier Account Setup
Snowflake UI Walkthrough
Load Data and Create Table
Event-Driven Ingestion using Snowpipe
S3 Bucket + AWS SQS + Snowflake Integration
Storage Integration
Notification Integration
External Stage
Snowpipe
Snowflake Tasks
Introduction to Snowflake Cortex AI
Cortex AI Functions Overview
Using Cortex AI Functions with SQL
Text Summarization, Sentiment Analysis, Classification and Entity Extraction using Cortex
Introduction to Cortex Search for RAG-style Retrieval
Introduction to Cortex Analyst for Natural Language Analytics

✅ Class - 2

Project - 1: Event-Driven Incremental News Data Analysis(Industrial Project)

Tech Stack: NewsAPI, AWS Airflow, S3, Snowflake, Python (Requests, Pandas), SQL
Project - 2: Car Rental Data Platform (Industrial Project)

Tech Stack: Python, PySpark, AWS EMR, AWS Airflow, Snowflake (SCD2)
Project - 3: Realtime Entertainment Data Analysis (Industrial Project)

Tech Stack: Python, Snowflake Dynamic Table (CDC), Snowflake Stream, Snowflake Tasks, Streamlit
Project - 4: Customer Support Intelligence using Snowflake Cortex AI (Industrial Project)

Tech Stack: Snowflake, Cortex AI Functions, Cortex Search, Cortex Analyst, Snowflake Tasks, SQL, Streamlit

AWS Services Covered

AWS IAM , AWS S3, AWS Lambda, AWS CodeBuild, AWS CloudFormation, AWS SNS, AWS SQS

AWS EC2, AWS EventBridge, AWS CloudWatch, AWS RDS, AWS Aurora, AWS Secrets Manager

AWS Glue, AWS Athena, AWS Redshift, Amazon MWAA (Managed Airflow), AWS EMR, AWS Kinesis Streams, AWS DynamoDB, AWS Kinesis Firehose, AWS Step Functions, Amazon Bedrock, Amazon Q , AWS OpenSearch

✅ Class - 1

AWS Free Tier Account Setup
Setup for Billing Alerts
IAM Users, Roles, Policies and Access Credentials
Amazon Q AI Assistant Usage
AWS S3 — Simple Storage Service Fundamentals
AWS S3 Storage Classes
Role of S3 in Data Engineering and Pipeline Design
General Purpose Buckets
Directory Buckets
Table Buckets — Apache Iceberg and its Fundamentals
Vector Buckets — Vector Embedding and RAG Fundamentals
Create S3 Bucket from AWS Console
Properties and Configuration of S3 Bucket
Create Folder Object in S3 Bucket from AWS Console
Upload, Download and Delete Files from S3 Bucket from AWS Console
AWS CLI Setup
AWS CLI Secrets Configuration
Interact with AWS S3 from Command Line — Create, Delete, Copy and Make Bucket
Access AWS S3 from Python using boto3 Library
AWS Lambda Architecture and Fundamentals
Where Lambda Fits in Data Engineering and Pipeline Architecture
Create First Lambda Function from AWS Console
Lambda Configurations and IAM Role Permissions
Testing Lambda Function with Test Event
Versions and Aliases for AWS Lambda
Weighted Aliases for Production Deployment
Setup S3 Notification Event Trigger for Lambda
Process S3 Notification Event Trigger Event in Lambda
Lambda Logs in CloudWatch
Packaging and Deployment of Lambda Function in ZIP Format
Concept of Layers in AWS Lambda
Creation and Deployment of Custom Layer
Lambda Function Execution with Custom Layer Package
Fundamentals of AWS Lambda Durable Function Pattern
Setup AWS Lambda Durable Function to Trigger Another Lambda as Worker
Execution of AWS Lambda Durable Function
Setup Lambda Function with External Python Dependency
Package Lambda Function and External Python Dependency for Deployment
CI/CD Fundamentals for Data Engineering
Create GitHub Repository for Code Versioning
Design AWS Lambda Function to Read CSV File from S3 based on S3 Create Event Trigger
Setup Project in GitHub Repository with Branching
Create AWS CodeBuild Project for CI/CD of AWS Lambda Function Deployment
AWS CodeBuild Environment Setup for Deployment using buildspec.yml, AWS CLI and Shell Script
Testing CI/CD Job in AWS CodeBuild through Feature Branch and Main Branch of GitHub

✅ Class - 2

AWS SNS Fundamentals
Real-world Pipeline Example with SNS : E-commerce Data Platform
Standard SNS vs FIFO SNS
SNS Configuration Properties
Setup AWS SNS from Console
Create SNS Topic
Publish Message on SNS Topic from Console
Subscription Protocols in SNS
Email Subscriber Setup for SNS Topic
Create AWS Lambda to Send Customized Messages on SNS Topic using boto3
AWS EC2 Fundamentals
Setup EC2 Instance from AWS Console
SSH Access of EC2 Instance
AWS SQS Fundamentals
Standard Queue vs FIFO Queue
Core SQS Parameters and Configurations — Visibility Timeout, Message Retention Period, Maximum Message Size, Delivery Delay, Receive Message Wait Time
Dead Letter Queue Fundamentals
Difference between AWS SQS and Kafka
AWS SQS Setup from Console
Publish and Consume Messages from SQS via Console
Create Lambda Function for Mock Data Generation
Create EventBridge Scheduler to Invoke Lambda Every Minute
Create Consumer Lambda Function and Set SQS as Trigger to Showcase Auto Polling / Event Consumption
Create Consumer Lambda Function to Programmatically Poll Messages from SQS and Showcase Long Polling
Dead Letter Queue Redrive Task
AWS EventBridge Pipe Fundamentals
Setup SQS to Store Logistics-domain Event Data
Create AWS Lambda to Generate Mock Logistics Data and Publish into SQS
Setup AWS EventBridge Pipe with SQS as Source
Apply Filter and Enrichment in EventBridge Pipe
Write Lambda Function for Enrichment and Transformation Logic
Add Target Lambda in EventBridge Pipe

✅ Class - 3

AWS Glue fundamentals
Glue Data Catalog fundamentals
How Glue Data Catalog works with Glue jobs, Crawlers and Athena
Glue Crawler fundamentals
Schema detection and schema evolution using Glue Crawler
Create Glue database and crawl CSV, JSON and partitioned data from S3
Register S3 datasets as Glue Catalog tables
Create Glue jobs from scratch to read data using Glue Catalog tables
Working with Glue DynamicFrames and DataFrames
Glue job configurations and properties
Data quality checks in Glue jobs
Publish Glue Data Quality results
Job Bookmarking for incremental data processing
Visual ETL job with Job Bookmarking to process incremental S3 data and write partitioned output
AWS RDS fundamentals
AWS Aurora architecture
Setup Aurora MySQL
Writer instance and read replica concepts
Aurora VPC, subnet and security group configuration
Connect Aurora writer and read replicas from DBeaver
Store Aurora credentials in AWS Secrets Manager
Create S3, Glue and STS VPC endpoints for Glue-Aurora connectivity
Create JDBC connection for Aurora in AWS Glue
Publish mock IoT data into Aurora MySQL table
Crawl Aurora tables using Glue Crawler and register them in Glue Data Catalog
Create Glue job with Job Bookmarking to process incremental Aurora data
Build medallion architecture using AWS Glue, Triggers and Workflows — Telecom data domain
Generate and upload mock telecom CSV data into S3
Create Bronze layer by reading raw data from Glue Catalog, applying filters and data quality checks, and writing Parquet to S3
Create Silver layer by transforming Bronze data and writing curated Parquet to S3
Create Gold layer by transforming Silver data for analytics-ready output
Create conditional Glue Triggers based on crawler and job success status
Create Glue Workflow with on-demand trigger, conditional triggers and Glue jobs
AWS Athena fundamentals and architecture
Where Athena fits in data pipelines
Athena Query Editor walkthrough
Glue Data Catalog and Athena integration
Configure Athena query results in S3
Create external databases and external tables in Athena
Query CSV, JSON and partitioned S3 data using Athena
Use MSCK REPAIR TABLE to refresh latest partitions
S3 Table Buckets and Apache Iceberg fundamentals
Create, ingest and query Iceberg tables in Athena
Event-driven Shopkart data pipeline using S3, Lambda, Glue and Athena
Trigger Lambda on S3 object creation
Invoke Glue job from Lambda
Transform, validate and write Parquet output to S3 using Glue
Query processed S3 data using Athena for analytics and reporting

✅ Class - 4

AWS Redshift fundamentals and architecture
MPP architecture, leader node and compute nodes
Columnar storage, compression encoding, sort keys and distribution styles — KEY, ALL, EVEN
Setup provisioned Redshift cluster and Redshift Serverless
Redshift Serverless network, security, namespace, workgroup and Query Editor walkthrough
Create database, schema and internal tables with encoding and distribution strategy
Load CSV data into Redshift using COPY command
UNLOAD data from Redshift into CSV and Parquet with manifest file
Load Parquet data into Redshift using manifest file
Create external database, external schema and external tables using Redshift Spectrum
Create and refresh materialized views
AWS EMR fundamentals and architecture
Setup EMR cluster, SSH access and SSH tunneling for application endpoints
Enable Apache Iceberg on EMR
AWS Managed Airflow fundamentals and architecture
Setup Airflow environment and UI walkthrough
Create and deploy Airflow DAGs with Bash operators
Insurance data pipeline orchestrated via Airflow and EMR
Build parameterized Airflow DAG using S3KeySensor, EMRAddStepOperator and EMRStepSensor
Submit Spark job from Airflow to EMR to process daily JSON files from S3 and write into Iceberg tables
EMR Serverless fundamentals
EMR Studio and EMR Serverless runtime application setup
Run Airflow-orchestrated Spark batch job on EMR Serverless
AWS DynamoDB fundamentals
Create DynamoDB table and perform insert, update and delete operations from Console and Python
DynamoDB Streams fundamentals for CDC
AWS Kinesis Data Streams fundamentals and setup
AWS Data Firehose fundamentals
Ride-hailing CDC pipeline using DynamoDB Streams, Kinesis Data Streams, Firehose, S3 and Athena
Create Athena table on CDC data and run analytical queries
AWS Step Functions fundamentals
Core components and state types in Step Functions
Where Step Functions fit in data pipelines
FreshCart quick-commerce data pipeline orchestrated using AWS Step Functions

✅ Project - 1: Agentic Uber Eats Marketplace Lakehouse Data Platform (Covered In Module 10)

Tech Stack - Databricks, Databricks Serverless Compute, Databricks Asset Bundles, Databricks Workflows, Unity Catalog, Unity Catalog Volumes, Delta Lake, Auto Loader, Medallion Architecture, PySpark, Spark SQL, Python, Delta MERGE, SCD Type 2, Star Schema Data Modeling, Fact Tables, Dimension Tables, Incremental Batch Processing, Native Data Quality Checks, Quarantine Tables, Liquid Clustering, Predictive Optimization, Delta UniForm, Databricks Genie, Conversational Analytics, GitHub, GitHub Actions, Databricks CLI, CI/CD, pytest, YAML, TOML, JSONL, CSV
✅ Project - 2: ProductPulse AI - Retail Product Intelligence & Support Copilot (Covered In Module 10)

Tech Stack - Databricks, Unity Catalog, Delta Lake, Unity Catalog Volume, PySpark, Databricks Workflows, Lakebase Postgres, pgvector, Databricks SQL Warehouse, Databricks Apps, Streamlit, Databricks Asset Bundles, Databricks Foundation Model APIs, Qwen Embedding Model, Llama Foundation Model, LLM-Based Tool Planning, RAG Pipeline, PDF Processing
✅ Project - 3: DLT Pipeline for Healthcare Domain (Covered In Module 10)

Tech Stack - Medallion Architecture, Databricks DLT, Delta Lake, SQL, Unity Catalog, Expectations, Databricks Workflows
✅ Project - 4: UPI Transactions CDC Streaming Analytics (Covered In Module 10)

Tech Stack - Databricks, PySpark Structured Streaming, Delta Lake (Change Data Feed), Unity Catalog
✅ Project - 5: Event-Driven Incremental News Data Analysis (Covered In Module 11)

Tech Stack: NewsAPI, AWS Airflow, S3, Snowflake, Python (Requests, Pandas), SQL
✅ Project - 6: Car Rental Data Platform (Covered In Module 11)

Tech Stack: Python, PySpark, AWS EMR, AWS Airflow, Snowflake (SCD2)
✅ Project - 7: Realtime Entertainment Data Analysis (Covered In Module 11)

Tech Stack: Python, Snowflake Dynamic Table (CDC), Snowflake Stream, Snowflake Tasks, Streamlit
✅ Project - 8: Weather Forecast Data Processing

Tech Stack: Python, OpenWeather API, AWS Airflow, PySpark, EMR Serverless, Redshift, S3, GitHub, GitHub Actions
✅ Project - 9: Aviation Data Processing Pipeline

Tech Stack - GitHub, GitHub Actions, S3, PySpark, EMR Serverless, AWS Airflow, Redshift, CI/CD setup
✅ Project - 10: Unified Stock Trading Analytics Platform

Tech Stack: MongoDB Atlas, Redshift, S3, PySpark, AWS Airflow, EMR, Hive Metastore, Python, SQL
✅ Project - 11: Industrial Maintenance RAG Assistant

Tech Stack: Python, AWS S3, AWS S3 Vectors, AWS Bedrock, AWS Lambda, AWS IAM, AWS DynamoDB, Static Frontend (HTML/CSS/JS), AWS CLI, Text LLM, Embedding LLM
✅ Project - 12: Retail Brokerage Data Processing Pipeline

Tech Stack: Python, AWS Glue (PySpark), AWS Step Functions, AWS EventBridge, AWS S3, AWS Redshift Serverless, AWS CloudFormation, AWS CodeBuild, Amazon SNS, PyTest, GitHub
✅ Project - 13: Retail Banking Analytics Data Platform — Transaction Processing, Risk Scoring & Portfolio Intelligence

Tech Stack: Python, PySpark, EMR Serverless, Apache Airflow (MWAA), AWS S3, Apache Iceberg, AWS Glue Data Catalog, AWS Athena, AWS IAM
✅ Project - 14: Real-Time Ticket Booking & Payment Analytics Platform

Tech Stack: Python, AWS Kinesis Data Streams, AWS Glue Streaming ETL, Spark Structured Streaming, AWS SQS, AWS Redshift Serverless, AWS S3, AWS CloudFormation, AWS Secrets Manager, Amazon VPC, n8n, OpenRouter (Frontier LLM), JavaScript
✅ Project - 15: Ride-Sharing Operations & Driver Performance Analytics Platform

Tech Stack: Python, AWS S3, EMR Serverless (PySpark), AWS Step Functions, AWS EventBridge, AWS Glue Data Catalog, AWS Athena, FastAPI, AWS Lambda Function URL, AWS CloudFormation, AWS CodeBuild, AWS SNS, n8n, Telegram Bot, JavaScript, GitHUb
✅ New Exclusive Project of LLM Application, RAG Pipelines, MCP & Agents for Data Engineering

Attention Seeking Resume Preparation and Interview Strategies
Strategies To Crack Tech Interviews
Linkedin Profile Making
How To Expand Your Professional Network On Linkedin
How To Use Various Job Portals
How To Approach For Referrals

Course Schedule

Mode Of The Course:

Online Live Classes

Started On:

20-June-2026 (In Progress)

Course Duration:

180 Hours

Total Sessions:

Total Projects:

Validity:

1 Year (Starting From The Date Of Enrollment)

Class Timing:

Saturday & Sunday [9:00 AM - 12:00 PM Live Teaching, 12:00 PM to 1:00 PM Live Doubt Session] (IST)

Class Duration:

3 Hours Live Teaching, 60 minutes Doubt Solving

Class Recording Provided:

Yes

Programming Language Used:

Python

Prerequisite:

No Prerequisite & No Prior Knowledge

⚠️ Important Notice :

The video may not work on Linux due to DRM restrictions. It is only accessible on Chrome when using Windows or macOS.

Workaround: To access the video on Linux, you can create a Windows virtual machine (VM) and watch the video through the VM. Alternatively, you can use our Android or iOS application to view the video on your mobile device.

Instructor

Shashank Mishra

Staff AI Data Engineer Ex-Expedia, Amazon, PayTM, Prophecy & McKinsey & Company

Shashank Mishra is a Staff AI Data Engineer with 9+ years of experience building scalable data platforms, real-time pipelines, cloud-native systems, and AI-powered data solutions for leading global organizations.

An MCA graduate from NIT Allahabad, he specializes in modern Data Engineering and its evolving intersection with AI, including RAG pipelines, LLM-ready data systems, intelligent workflows, and production-grade AI applications.

He is also the educator behind "E-Learning Bridge" YouTube channel, with 184K+ YouTube subscribers and a LinkedIn community of 189K+ professionals. Having mentored 25,000+ learners, Shashank is known for practical, project-driven teaching that helps professionals build industry-ready skills and prepare for top tech careers.

Complete AI Data Engineering With AWS ( 6.0 ) - Basic To Advance (Live Classes)

INR 13000

Buy Now

Complete AI Data Engineering With AWS ( 6.0 ) Basic To Advance

1 Year

180 Hours

40

15

Tech stack you'll learn

Course Content

Module 1 - SQL

Module 2 - Big Data Fundamentals & Hadoop

Module 3 - Confluent Kafka

Module 4 - NoSQL Database: MongoDB

Module 5 - Apache Spark (PySpark)

Module 6 - Apache Airflow

Module 7 - AI Fundamentals for Data Engineering

Module 8 - Data Warehousing & Data Modelling

Module 9 - Apache Iceberg

Module 10 - Databricks & AI

Module 11 - Snowflake & AI

Module 12 - Complete AWS Cloud + Practical AI Data Engineering

Module 13 - Industrial Projects (15 Projects)

Module 14 - Resume, LinkedIn & Interview Strategies

🎁 FREE Python Crash Course

Course Schedule

Instructor

Shashank Mishra

Complete AI Data Engineering With AWS ( 6.0 ) - Basic To Advance (Live Classes)

Namaste 🙏

Complete AI Data Engineering With AWS ( 6.0 )
Basic To Advance