End-to-end retail lakehouse on Databricks: dual-path ingestion from PostgreSQL and S3, dbt-modeled galaxy schema with SCD2 history, Airflow orchestration with parallel DAG execution, CI-gated dbt tests, and a dual consumption layer via Power BI and a Genie Agent.
Designed a production-grade Azure Data Engineering project centered on Azure Data Factory. Built dynamic, metadata-driven pipelines to ingest data from on-prem systems, REST APIs, and Azure SQL into ADLS Gen2 using Medallion Architecture, incremental loading, and enterprise-scale orchestration patterns.
End-to-end data warehouse for multi-country market-share analytics (FMCG / nutrition) across two markets — Spain (ES) and Portugal (PT) — built on a medallion architecture in Snowflake, feeding Power BI dashboards.
Incremental batch data pipeline for e-commerce order data, built with Airflow and Polars using a medallion (bronze/silver/gold) architecture, Pandera data quality gates, and dimensional modeling on DuckDB.
End-to-end analytics platform on Microsoft Fabric — metadata-driven pipelines, incremental loading, SQL Warehouse, semantic model, Power BI
End-to-end ETL pipeline: incremental API ingestion, dbt transforms, Airflow orchestration, and CI
End-to-End Retail ETL Pipeline using Apache Airflow, PostgreSQL, Docker, and Python with Incremental Loading and Data Quality Validation.
Enterprise SQL data platform demonstrating metadata-driven ETL, staging, dimensional warehousing, incremental loading, auditability, and Docker/LocalDB deployment using SQL Server and SSDT.
Production-inspired ETL pipeline built with Python and PostgreSQL featuring data validation, transformation, incremental loading, audit logging, reporting, archiving, CLI support, retry mechanism, and automated testing.
Building my first ETL pipelines using Python to learn more about data engineering. Learning by doing.
A Python library that tracks which rows of a table loaded from another system you can trust, and as of when
Azure Data Factory end-to-end project demonstrating real-world data engineering workflows including API ingestion, on-premises data migration, incremental loading with watermarking, pipeline orchestration, Logic App alerts, audit logging, and REST API pagination.
Pipeline de données Azure DevOps de bout en bout sur les rivières en temps réel (Hub'Eau) : chargement incrémental, bronze/argent/or, porte qualité, CI/CD DEV et PROD approuvée, paquet versionné dans Azure Artifacts, guide en 16 étapes. Coût Azure : 0 €.
Production-oriented Data Engineering pipeline with Python, DuckDB, Data Quality, incremental loading, automated testing, and dimensional modeling.
End-to-end Azure data engineering solution for U.S. energy market analytics with ADF, Functions, ADLS Gen2, Synapse, Terraform, and Power BI.
Azure SQL → Azure Data Factory → ADLS Gen2 → Databricks: watermark-based incremental loads, Auto Loader, Delta Live Tables with SCD Type 2. Guided project, extended.
Real-time AWS ecommerce data pipeline using MSK Serverless, MSK Connect, S3, AWS Glue, Delta Lake, SCD Type 2, and Redshift Serverless.
This project pulls historical and forecast weather data for multiple cities, cleans and transforms it, performs quality checks, and stores the results in tidy daily and monthly summary datasets.
An end-to-end data engineering project. Airbnb-style booking data is loaded from S3 into Snowflake, modeled with dbt into a star schema (with SCD Type 2 history for hosts and listings), and served to a Power BI dashboard.
End-to-end Retail Sales ETL Pipeline using Python, PostgreSQL, Apache Airflow, Docker, Star Schema Data Warehouse Modeling, KPI Generation, and Incremental Data Loading.
To associate your repository with the incremental-loading topic, visit your repo's landing page and select "manage topics."