Build data pipelines and ETL workflows

Design and implement data pipelines, ETL, schema, and data quality checks

Building reliable, tested data infrastructure when source systems are messy or heterogeneous

• · source system description
• · target schema requirements
• · data volume/frequency

• · pipeline orchestration code
• · schema definitions
• · data validation tests

• · pandas
• · SQL
• · dbt/Airflow
• · data warehouse connectors

• · source systems accessible
• · target warehouse identified

• · source data quality poor
• · schema changes mid-pipeline
• · performance bottleneck at scale

• · validates data at each stage
• · logs lineage and transformations
• · handles incremental loads
• · schema evolution versioned