Advanced Data Engineering with Snowflake

Overview

This is a technical, hands-on course that teaches you how to implement DevOps best practices to build data pipelines, and how to implement observability to maintain and monitor data pipeline health. The course focuses on the most practical Snowflake concepts, features, and tools to get you up and running quickly with these concepts.

You'll start by learning about DevOps, DevOps practices, and how DevOps fits into the context of data engineering. You'll incorporate source control, declarative management of database objects, continuous delivery, and use a command-line interface to implement DevOps best practices into a data pipeline.

You'll specifically learn how to:

- Use Snowflake's git integration to add source control to your data pipeline - Use GitHub for team-wide collaboration on your data pipeline - Use CREATE OR ALTER to declaratively manage database objects - Use GitHub Actions to implement continuous delivery for your pipeline - Use Snowflake CLI to deploy changes into dedicated data environments You'll also learn about observability, and how to implement it to maintain and monitor the health and performance of your data pipeline. You'll specifically learn how to:

- Use logs to keep a record of events that occur within your pipeline - Use traces to maintain a detailed journey of events for operations in your pipeline - Use alerts to monitor for specific conditions in your pipeline, and combine them with notifications to encourage action among team members if critical errors occur in the pipeline Throughout the course, you'll follow along with the instructor using a combination of Snowflake, Visual Studio Code, GitHub, and the command line.

The course is supplemented with readings containing resources to level up your understanding of specific concepts. You'll come away understanding how to incorporate DevOps best practices into data pipelines, and how to use observability to monitor the health and performance of pipelines.

Syllabus

• DevOps with Snowflake
In this module, you'll understand how DevOps helps software development teams iterate safely and efficiently, and understand how those practices can be applied in the field of data engineering. You'll learn how to implement a few key DevOps best practices for data pipelines. Namely, you'll learn how to implement source control for pipeline objects, how to declaratively manage database objects, and how to introduce changes to dedicated data development environments using continuous integration. By the end of the module, you'll understand how data pipelines can be built collaboratively by large teams, and how they can be evolved efficiently and reliably.
• Observability with Snowflake
In this module, you'll learn about observability, and how it can be implemented to monitor the health and performance of data pipelines. You'll specifically learn about Snowflake's observability framework, Snowflake Trail, and how to implement its core components. You'll use event tables, logs, and traces to implement detailed records of events occurring within your data pipeline. You'll also learn how to generate alerts to detect specific conditions in your data environment, and how to combine them with notifications to communicate information to key stakeholders, like a broader data engineering team.

Taught by

Snowflake Northstar

Subjects

Business