Tag: #data-engineer
3 items with this tag.
Awesome Data Engineer
Data Engineer Articles Medium - Data Pipeline Development with MinIO, Iceberg, Nessie, Polars, StarRocks, Mage, and Docker Medium - ETL and ELT 🌟 (Recommended) Medium - ELT with Fabric, Azure and Databricks Medium - The Change Data Capture (CDC) Design Pattern 🌟 (Recommended) Medium - Apache Airflow Overview Blog - Data Lake vs Data Warehouse 🌟 (Recommended) ML4Devs - Scalable Efficient Big Data Pipeline Architecture MonteCarlo - Data Pipeline Architecture Explained: 6 Diagrams and Best Practices 🌟 (Recommended) Medium - Data Engineering Best Practices: How Big Tech & FAANG Firms Manage and Optimize Apache Kafka Medium - 5 Steps to Build Efficient Data Pipelines with Apache Airflow Blog - A Primer on Data Warehouses 🌟 (Recommended) Blog - Data Industry Primer 🌟 (Recommended) Onehouse - Comprehensive Data Catalog Comparison 🌟 (Recommended) Awesome Repositories awesome-apache-airflow: Curated list of resources about Apache Airflow awesome-bigdata: A curated list of awesome big data frameworks, resources and other awesomeness.
From GitLab to Data LakeHouse Pipeline, The Orchestration Strategies
Quote Hello, hello! It’s me again, and today I’m returning with a new solution. This tool seems similar to Airflow but is designed to more directly tackle the challenge of orchestrating data pipelines, especially when dealing with data-lakehouse components.
The First Data Lakehouse and Challenge
Quote “Hi everyone! It’s been a while since I’ve blogged. I encountered a couple of health problems a few weeks ago, but I’m fully recovered now and eager to get back to what I love.