Back to Projects
Data Engineeringintermediate

End-to-End Realtime Flight Data Engineering with Airflow & Snowflake

This project implements an end-to-end flight data engineering pipeline using Apache Airflow for orchestration and Snowflake as the data warehouse, following a medallion architecture pattern with bronze, silver, and gold data layers. The bronze layer ingests raw flight data from JSON files, the silver layer performs data cleansing and transformation, and the gold layer aggregates insights for analytics, all automated through scheduled Airflow DAGs. Built with Python scripts for each layer and Docker Compose for containerized deployment, this solution ensures scalable, reliable data processing from ingestion to warehousing, supporting real-time flight operations analytics.

9 lectures

What You Will Learn

Students will learn to design, implement, and manage end-to-end ETL/ELT pipelines using Apache Airflow, including task dependencies, scheduling, and error handling in a production-like environment.
Students will gain hands-on experience with medallion architecture (bronze/silver/gold layers) and cloud data warehousing using Snowflake, including data ingestion, transformation, aggregation, and loading techniques.

System Architecture

End-to-End Realtime Flight Data Engineering with Airflow & Snowflake Architecture Diagram

High-level architecture overview of the End-to-End Realtime Flight Data Engineering with Airflow & Snowflake .

What You'll Build

  • An end-to-end ETL pipeline that ingests real-time flight data from the OpenSky Network API, processes it through bronze (raw), silver (cleaned), and gold (aggregated) layers, and loads in
  • A containerized data architecture using Docker Compose, featuring Apache Airflow for orchestration, PostgreSQL for metadata, and Snowflake as the cloud data warehouse, supporting scheduled batch processing and manual runs.

Project Instructor

Sudhanshu

Sudhanshu

4+ years exp
LinkedIn
End-to-End Realtime Flight Data Engineering with Airflow & Snowflake
Premium
One Subscription. 40+ Projects. Unlimited Access.
AccessMobile & Web