GCP Data Engineering Course With Projects

Light
Dark

GCP Data Engineering Mastery

AWS Services For Data Engineering With Projects

By Shashank In Grow Data Skills

Self Paced
English
Completion Certificate

Demo Class Video

1 Year

Validity (From the date of Enrollment)

50 Hours

Duration

12

Sessions

6

Projects

INR 4000

Buy Now

This course includes

Content Duration50 Hours
Total Sessions12
Total Industry Projects6
GCP Cloud
Quality Assignments & quizzes after each module
Interview preparation guide
Dedicated placement assistance
Resume & Linkedin profile making
Doubt support on private discord community
Certificate of completion

Tech stack you'll learn

GCP Services For Data Engineering With Projects

Course Content

Get credit card of Visa/Mastercard type (Don't try with Debit Card, Net Banking or UPI etc).
Enable Online & International Transactions - Enable this via your bank app or customer service and make sure to enable AutoPay as well.
Use the same name on the card and GCP account to avoid verification issues.
While setting up Billing profile in GCP or at any step if it asks for address then make sure the address matches what’s registered in your bank credit card.
Avoid Using VPN - VPNs can trigger fraud detection and block your account.
GCP does not charge you during the Free Tier or 90-day trial unless you upgrade. Set budget alerts to monitor spending.

GCP Services Covered - GCS, IAM, GCP Compute Engine, GCP Cloud Monitoring & Cloud Logging, GCP Cloud Run Function, GCP Cloud Build, GCP Pub-Sub, GCP Cloud Scheduler

✅ Class - 1

Overview of GCP Free Tier account setup
Gcloud CLI setup
Google Cloud Storage (GCS)
Create & Access Buckets, File in GCS from CLI
Create & Access Buckets, File in GCS from Python Script
IAM (Identity and Access Management) & Role Based Access
Setup & Access Cloud compute machine (GCP Compute Engine)
Server health monitoring and alerting (GCP Cloud Monitoring & Cloud Logging)

✅ Class - 2

GCP Cloud Run Function Fundamentals
HTTP trigger based GCP Cloud Run Function implementation and execution with test event
Implementation & Execution of modular GCP Cloud Run Function with external Python library dependencies
Continuous deployment of GCP Cloud Run Function with GCP Cloud Build & GitHub
Implementation & Execution of GCP Cloud Run Function with Cloud Storage Object Create Event Trigger
Read CSV file data using Pandas in GCP Cloud Run Function
Create & Deploy GCP Cloud Run Function using gcloud command

✅ Class - 3

GCP Pub-Sub Fundamentals
Difference Between GCP Pub-Sub & Kafka
Create GCP Pub-Sub Topic & Subscription
Publisher and Subscriber (Single) code execution to publish & process JSON seriliazed data
Publisher and Subscriber (Multiple) code execution to publish & process JSON seriliazed data
Avro & ProtoBuf Schemas in GCP Pub-Sub
Avro Schema Setup & Integration with GCP Pub-Sub Topic
Publisher and Subscriber code execution to publish & process Avro seriliazed data
Cloud Run Function setup & execution with GCP Pub-Sub Message Trigger
HTTP tigger Cloud Run Function invocation with scheduled job via Cloud Scheduler

GCP Services Covered - GCP Cloud SQL, GCP Secret Manager, GCP BigTable, GCP Dataflow, Apache Beam, GCP BigQuery

✅ Class - 1

MySQL Setup setup on GCP using Cloud SQL
Connection setup with MySQL on GCP with MySQL Workbench
Maintaining Database credentials in GCP Secret Manager
Connection and Query execution on MySQL instance using Python application
GCP BigTable Fundamentals and Architecture
Create BigTable instance
Create Table with Column Families in BigTable instance
Setting up CDC stream on a Table in GCP BigTable
BigTable data manipulation using Python script
GCP Dataflow Fundamentals
Create Batch Dataflow job to offload data from BigTable into Google Storage in JSON format
Create Streaming Dataflow job capture CDC events from BigTable and send to GCP Pub-Sub Topic
Create Apache Beam application which can accept customized parameters from user and process current date csv file and load it in BigTable
Build image and Dataflow FlexTemplate of customized application
Create Dataflow Batch job using customized FlexTemplate
Create & Run Dataflow Batch job based on frequency using Cloud Scheduler and custom parameters as HTTP request Payload

✅ Class - 2

BigQuery Fundamentals
BigQuery Architecture : Capacitor, Colossus Storage, Dremel Execution Engine, Borg Cluster Manager, Jupiter Network
Difference Between Internal & External Tables
Create Dataset in BigQuery From UI
Create Managed Table in BigQuery from UI on top of CSV file stored in Google Storage
Managed Table SQL Operations:
Create Schema
Create Table With Partition & Cluster Keys
Load CSV Data From Google Storage into BigQuery Table
Insert Command To Ingest Data in BigQuery Table
Update Command
Where Clause
Query table data with partition column
Show table lists in Schema
Show table definition
List distinct partitions
Time travel select query
Create External Table in BigQuery On Top of Semi-Structured Data (Multiline JSON) stored in Hive Style Partitioned Google Storage Bucket
Craft Aggregate Queries on Telecom Data and Schedule queries for frequency based execution
Starbucks Data Analysis & Visualization using BigQuery Data Canvas (Gemini AI assisted Query Generation)
Telecom Data Cleaning using BigQuery's AI Oriented Data Preparation Feature
Silver & Gold Layer Data Preparation with BigQuery Python Notebook using BigQuery Dataframe
Telecom Data Pipeline (Medallion Architecture) in BigQuery
Task 1 - Bronze Layer Data Preparation Execution
Task 2 - Silver Layer Notebook Execution
Task 3 - Gold Layer Notebook Execution
Loyality Program Real Time Data Ingestion in BigQuery Table From GCP Pub-Sub using Dataflow Pipeline

GCP Services Covered - Managed Service for Apache Spark [Formerly GCP Dataproc], Managed Service for Apache Spark [Formerly GCP Dataproc] Serverless, Managed Service for Apache Airflow [Formerly GCP Composer]

✅ Class - 1

Complete Setup of Managed Service for Apache Spark [Formerly GCP Dataproc] Cluster
Managed Service for Apache Spark [Formerly GCP Dataproc] Cluster walkthrough
Hadoop, Yarn & Spark history server
SSH on Managed Service for Apache Spark [Formerly GCP Dataproc] Cluster
Execution of Hive & PySpark shell on Managed Service for Apache Spark [Formerly GCP Dataproc] cluster
Working with Jupyter Lab & Notebooks in Managed Service for Apache Spark [Formerly GCP Dataproc] Cluster
Earthquake Data Analysis on GCP Managed Service for Apache Spark [Formerly GCP Dataproc] Notebook
Project - Incremental Data Ingestion in SCD2 Type table in BigQuery with CICD
Tech Stack - Python, PySpark, GCP Managed Service for Apache Spark [Formerly GCP Dataproc], GCP BigQuery, GitHub, GitHub Actions, PyDeequ, Google Storage
PySpark job to process customers & orders data and then execution of it on Managed Service for Apache Spark [Formerly GCP Dataproc] Serverless via Gcloud command
Spark Structered Streaming job on Managed Service for Apache Spark [Formerly GCP Dataproc] cluster to read continuous File Stream of json format (Create via mock Python script) published in Google Storage, Process it and write data into Iceberg table built on top of Google Cloud Storage

✅ Class - 2

What is orchestration in BigData?
Need of dependency management in Data Pipeline design
What is Airflow?
Architecture & Different Components of Airflow
Operators in Airflow
How to write Airflow DAG Scripts?
Attribute description
How to execute parallel tasks?
Setup Airflow Environment on google cloud with Managed Service for Apache Airflow [Formerly GCP Composer]
Airflow UI Walkthrough
Create and schedule Airflow Dag with sequential tasks using BashOperator and PythonOperator
Create and schedule Airflow dag with parallel tasks using BashOperator and PythonOperator
Airflow Exercise - 1 : End-To-End Airflow Dag to Create Managed Service for Apache Spark [Formerly GCP Dataproc] Cluster, Run PySpark Job on cluster and Delete Managed Service for Apache Spark [Formerly GCP Dataproc] cluster
Airflow Exercise - 2 : Airflow Dag to support data backfilling via parameterized date inputs and use of Variables in Airflow
Industrial Project - Flight Booking Data Pipeline with CICD
Tech Stack - GitHub, GitHub Actions, Python, Google Storage, PySpark, Managed Service for Apache Spark [Formerly GCP Dataproc] Serverless, Airflow, BigQuery

GCP Services Covered - GCP Data Fusion, GCP Workflows

✅ Class - 1

GCP Data Fusion Instance Setup
GCP Data Fusion UI Walkthrough
Wrangler Transformations – Cleaning, parsing, and transforming raw data with no code from Google Storage & BigQuery
Custom Wrangler Transformations Recipe
Exercise - HackerNews Batch Data Processing Pipeline
Upload sample data of 100k records in Google Storage Bucket
Open pipeline canvas in GCP Data Fusion
Create GCS Source & Configure to read CSV file
Apply Wrangler Transformations
Add ErrorCollector to capture failed records
Use GCS as target to write ErrorCollector records in Google Storage
Connect BigQuery as target with Wrangler Transformations to write transformed data in BigQuery Table
Deploy & Execute pipeline in GCP DataFusion
Exercise - Multiple Orders File Processing
Place multiple csv files of orders data with same structure in GCS
Add Source node of GCS type in Data Fusion
Include file_name column in schema
Add GCS target to write transformed data
Run pipeline preview and validate result
Exercise - Fintech Data Batch Pipeline Processing
Read customers_dim table from BigQuery using BigQuery source node
Read transactions_raw csv data from GCS using GCS source node
Apply wrangler transformation on both
Perform Join operation
Perform group by operation to generate two separate aggregates - Txn_Date level aggregates, Currency level aggregates
Write these two aggregate results in two separate BigQuery tables
Deploy and execute pipeline

✅ Class - 2

GCP Workflows Fundamentals
How to create GCP Workflows and it's different settings
Create first Hello World GCP Workflow
Define main and sequential steps
Create step to assign a variable with dummy message
Create step to publish dummy message in log
Create step to return success message on console
Create GCP Workflow to pass input JSON payload and orchestrate Cloud Run Function
Create cloud run function which is http trigger base and accept input json payload, then return response with Success/Fail status
Create GCP Workflow with main, steps & params
Create step to trigger cloud run function with input param
Persist return response of cloud run function
Use switch action to check Success & Fail status
Create GCS create object event driven GCP Workflow which will execute PySpark job on Managed Service for Apache Spark [Formerly GCP Dataproc] Serverless & perform SCD2 Merge operation on BigQuery target table
Create GCS bucket to upload daily json file
Create staging and target BigQuery table
Create GCP Workflow with env variables, tigger, main and steps
Create step to parse input event and fetch bucket name & object name
Create step to validate if expected file pattern arrived
Create step to run PySpark job on Managed Service for Apache Spark [Formerly GCP Dataproc] serverless with input arguments
Create step to check job status in wait condition until succeed or fail
Create step to run Merge query from staging table to target table in BigQuery
Based on PySpark job status merge query step will get executed in wait condition until success or fail
Create step to insert data from staging table to target table with is_active flag as True

Industry Project - 1 : Flight Booking Data Pipeline with Airflow & CICD (Covered In Module 3)
Tech Stack - GitHub, GitHub Actions, Python, Google Storage, PySpark, Managed Service for Apache Spark [Formerly GCP Dataproc] Serverless, Airflow, BigQuery
Industry Project - 2 : Weather Forecast Data Processing
Tech Stack : Python, Open Weather API, Managed Service for Apache Airflow [Formerly GCP Composer], PySpark, Managed Service for Apache Spark [Formerly GCP Dataproc] Serverless, BigQuery, Google Storage, GitHub, GitHub Actions
Industry Project - 3 : Uber Car Idle Realtime Alerts
Tech Stack : Python, GCP Pub-Sub, GCP Dataflow, Apache Beam, GCP BigQuery, GCP Cloud Run Function
Industry Project - 4 : Credit Card Transactions Fraud Alert Pipeline
Tech Stack - Managed Service for Apache Airflow [Formerly GCP Composer], Managed Service for Apache Spark [Formerly GCP Dataproc] Serverless, Python, BigQuery, GitHub, GitHub Actions, PyTest, Looker Studio
Industry Project - 5 : Realtime Game Leaderboard Pipeline
Tech Stack - GCP Pub-Sub, GCP Dataflow, Apache Beam, Python, BigQuery, GCP Cloud Run Functions, GCP BigQuery Scheduled Query
Industry Project - 6 : YouTube Wide Trending Engagement Pipeline
Tech Stack - Google Storage, Managed Service for Apache Airflow [Formerly GCP Composer] (Airflow), PySpark, Python, Managed Service for Apache Spark [Formerly GCP Dataproc], Iceberg, GCP BigQuery

Attention Seeking Resume Preparation and Interview Strategies
Strategies To Crack Tech Interviews
Linkedin Profile Making
How To Expand Your Professional Network On Linkedin
How To Use Various Job Portals
How To Approach For Referrals

Course Schedule

Mode of the Course:

Recorded

Course Duration:

50 Hours

Total Sessions:

Total Projects:

Course Validity:

1 year (Starting From Day Of Enrollment)

Programming Language Used:

Python

Prerequisite:

Basics Python, SQL, Spark OR You Can Check 👉

Data Engineering & Python & SQL

⚠️ Important Notice :

The video may not work on Linux due to DRM restrictions. It is only accessible on Chrome when using Windows or macOS.

Workaround: To access the video on Linux, you can create a Windows virtual machine (VM) and watch the video through the VM. Alternatively, you can use our Android or iOS application to view the video on your mobile device.

Instructor

Shashank Mishra

Staff AI Data Engineer Ex-Expedia, Amazon, PayTM, Prophecy & McKinsey & Company

Shashank Mishra is a Staff AI Data Engineer with 9+ years of experience building scalable data platforms, real-time pipelines, cloud-native systems, and AI-powered data solutions for leading global organizations.

An MCA graduate from NIT Allahabad, he specializes in modern Data Engineering and its evolving intersection with AI, including RAG pipelines, LLM-ready data systems, intelligent workflows, and production-grade AI applications.

He is also the educator behind "E-Learning Bridge" YouTube channel, with 184K+ YouTube subscribers and a LinkedIn community of 189K+ professionals. Having mentored 25,000+ learners, Shashank is known for practical, project-driven teaching that helps professionals build industry-ready skills and prepare for top tech careers.