# How do data pipelines work

> Understanding data pipelines is crucial for aspiring data and AI engineers. This lesson covers the definition of data pipelines, their importance in decision-making, and the three main types: regularly scheduled, real-time, and on-demand pipelines. Key technologies such as Cron for scheduling tasks…

- Web page: https://learn.dataexpert.io/lesson/how-do-data-pipelines-work
- Program: [The Absolutely Free Beginner Data Engineering Boot Camp starting August 14th](https://learn.dataexpert.io/program/the-absolute-beginner-data-engineering-boot-camp-starting-august-7th-6453.md)
- Access: Free — no enrollment needed
- Length: 9 min video
- Skills: ETL/ELT, Airflow, Data Warehousing, Cloud Platforms, AWS
- Academy: DataExpert.io Community Academy

## About this lesson

Understanding data pipelines is crucial for aspiring data and AI engineers. This lesson covers the definition of data pipelines, their importance in decision-making, and the three main types: regularly scheduled, real-time, and on-demand pipelines. Key technologies such as Cron for scheduling tasks are discussed, along with the shift from traditional databases to cloud storage solutions like S3, providing a comprehensive overview for those looking to enhance their data engineering skills.

## Transcript excerpt

What is a data pipeline? That is one of the most fundamental components of being a good data engineer and a good AI engineer nowadays. So, let's talk a little bit about what a data pipeline is and how there are 3 different types of data pipelines that we will be covering. We will cover, uh probably 2 of them in this boot camp. So, let's start off with what is a data pipeline. So, a data pipeline is the ability to have flow of information. That can then be translated into insight and knowledge so that the business can make a better decision. That is at the highest level, that is what a data pipeline is. So, for example, maybe someone is like clicking buttons and uh stuff in your app and that generates events, and then those events are pumped into a database and then enriched with other data, and then we can know like there was 17 clicks in. China or something like that, right? That is gonna be like the flow of information and like data can come from all sorts of different sources. So there is like click stream data that might come from your application. There's production data that might be things like, who are your users, what posts have they created? What are uh different things like that that are in your production. Database. You can also have third party data like maybe you have data from Salesforce or maybe you have data from Google Analytics, or maybe you have data from HubSpot or whatever.

## Navigation

- Previous lesson: [Building a REST API Extractor with Python](https://learn.dataexpert.io/lesson/building-an-api-extractor-with-python.md)
- Next lesson: [Pipeline Scheduling Basics Lab](https://learn.dataexpert.io/lesson/pipeline-scheduling-basics-lab.md)