The Rise of the AI Data Engineer

Master all the skills needed to be a great AI data engineer on Databricks

Zach Wilson

Taught by Zach Wilson

Founder at DataExpert.io

What you'll learn

✓
How to build basic pipelines with Databricks
✓
How to use Databricks AI features
✓
An end-to-end capstone project
✓
3 Live classes with Zach

Learn directly from the experts

Zach Wilson

Zach Wilson

Founder at DataExpert.io

I have led teams of data engineers and software engineers at Airbnb, Facebook, and Netflix. My next goal is to upskill as many data knowledge workers as I can!

Course syllabus

45 lessons • 27+ hours of content • 8 assignments

Intro to Databricks
1
Getting Started with Databricks Free Edition
Data Engineering with Delta and Spark
1
Building your very first Data Lakehouse
Context Engineering and Vector Databases
1
Processing Unstructured Data with Vectors
End-to-end AI Agents
1
Creating your first Agent with AgentBricks

Also included

Boot Camp: Community Edition logo
IncludedBoot Camp: Community Edition
Join the free community boot camp that covers everything you need to know to go from junior data engineer to senior data engineer! 41 lessons

Bootcamp Orientation

1Bootcamp Kickoff
2Boot Camp Database Setup

Dimensional Data Modeling

1Dimensional Data Modeling Complex Data Type and Cumulation Day 1 Lecture
2Dimensional Data Modeling Complex Data Type and Cumulation Day 1 Lab
3Dimensional Data Modeling: Building Slowly Changing Dimensions Day 2 Lecture
4Dimensional Data Modeling: Building Slowly Changing Dimensions Day 2 Lab
5Dimensional Data Modeling: Graph Data Modeling Day 3 Lecture
6Dimensional Data Modeling: Graph Data Modeling Day 3 Lab

Fact Data Modeling

1Fact Data Modeling: Core Concepts, Deduplication Day 1 Lecture
2Fact Data Modeling: Practical Insights into Data Modeling Day 1 Lab
3Fact Data Modeling: Core Elements in Data Modeling Day 2 Lecture
4Fact Data Modeling: Compact Tables for Efficient Data Representation Day 2 Lab
5Fact Data Modeling: Minimizing Shuffle and Reducing Facts Day 3 Lecture
6Fact Data Modeling: Practical Guide to Formatting and Aggregating Data Day 3 Lab

Apache Spark Fundamentals

1Apache Spark: Architecture, Optimization, and Best Practices Day 1 Lecture
2Apache Spark: Hands-On for Broadcast and Hash Joins Day 1 Lab
3Apache Spark: Managing Spark Jobs and Notebooks Day 2 Lecture
4Apache Spark: User-Defined Functions and Broadcast Join Day 2 Lab
5Unit Testing Spark Jobs: Importance, Challenges, and Leadership Perspectives Lecture
6Unit Testing Spark Jobs: Mastering Spark and PySpark Testing Lab

Applying Analytical Patterns

1Applying Analytical Patterns: Exploring SQL, Scaling Projects and Aggregation Analysis Day 1 Lecture
2Applying Analytical Patterns: Mastering Growth Accounting and Retention Analysis Day 1 Lab
3Applying Analytical Patterns: Recursive CTEs and Window Functions Day 2 Lecture
4Applying Analytical Patterns: Aggregations and Cardinality Reduction Day 2 Lab

Real-time pipelines with Flink and Kafka

1Flink Lab Setup
2Streaming Pipelines: Mastering Streaming and Real-time Pipelines Day 1 Lecture
3Streaming Pipelines: Setting up Streaming Pipelines Day 1 Lab
4Streaming Pipelines: Exploring Data Collection and Processing Day 2 Lecture
5Streaming Pipelines: Kafka, Postgres, Spark Integrations and Parallelism Day 2 Lab

Data Visualization and Impact

1Data Visualization and Impact: Mastering Data Engineering Day 1 Lecture
2Data Visualization and Impact: Hands-On with the CSV files Day 1 Lab
3Data Visualization and Impact: Insights and Best Practices Day 2 Lecture
4Data Visualization and Impact: Exploring Data Visualization and Aggregation Techniques Day 2 Lab

Data Pipeline Maintenance

1Data Pipeline Maintenance: Navigating the Complexities of Data Engineering Day 1 Lecture
2Data Pipeline Maintenance: Strategies for Maintenance and Dock Building Day 2 Lecture

KPIs and Experimentation

1KPIs and Experimentation: Decoding Business Success: Metrics, Growth Strategies and Collaborative Approaches Day 1 Lecture
2KPIs and Experimentation: Setting up and Analysing Experiments Day 1 Lab
3KPIs and Experimentation: Leading and Lagging Metrics Day 2 Lecture

Data Quality Patterns

1Data Quality Patterns: MIDAS Process from Airbnb Day 1 Lecture
2Data Quality Patterns: Spec-Building Document Day 1 Lab
3Data Quality Patterns: WAP Patterns Day 2 Lecture

Before you join

Prerequisites

You are eager to learn Databricks
Join the Discord communityConnect with instructors and fellow participants before you start.
Join Discord

Platform Access Included

DiscordDiscord
GitHubGitHub

Get updates for The Rise of the AI Data Engineer

We'll notify you about updates, new content, and special offers.