Codeless Data Engineering in GCP: Beginner to Advanced

Google Cloud Storage will be used in this course to make a data lake. We will also use Google BigQuery to bring data warehouse capabilities to the data lake to make the lakehouse architecture. Using services like DataStream, Dataflow, DataPrep, Pub/Sub and Cloud Storage as well as BigQuery, we will build four no-code data pipelines that will send and receive data.

Students will learn how to set up a data lake, build data pipelines for data ingestion, and transform data for analytics and reporting in a way that makes sense to them.

This is the first chapter of the course.

• We’ll start a project in Google Cloud.
• It’s time to learn about Google Cloud Storage.
• In this video, we’re going to show you how to use Google BigQuery

Data Pipeline 1

• Before we do any complicated ETL jobs, we will set up a cloud SQL database and add data to it.
• It is important for us to use DataStream Change Data Capture to stream data from our Cloud SQL Database into our data lake built with Cloud Storage.
• This is what we need to do: Add a notification to our bucket for people to see.
• Create a Dataflow Pipeline so that jobs can be streamed into BigQuery.

Data Pipeline 2

• Introduce Google’s Data Fusion tool.
• An ETL job is a way to change data and move it to a new place in our data lake. You write and watch the job to make sure it works.
• Data must be cleaned and normalized before it can be used in a study.
• Using metadata in Data Fusion, you can find and keep track of your data.

Data Pipeline 3

• In this video, we’ll show you how to use Google Pub/Sub.
• Then, I’ll make a .Net app that will send data to a Pub/Sub topic.
• Creating a real-time data pipeline to send messages to BigQuery as they come in

Data Pipeline 4

• Getting Started with Cloud Data Prep
• Profile, write and keep an eye on ETL jobs that use DataPrep to change our data.