Browsing Tag
dataengineering
38 posts
My First GitHub Project: From a Local Folder to GitHub Using Git and SSH
I thought that when i join Lux Dev i would jump straight into building complex data pipelines and…
Why Spark Couldn’t Read from Kafka: A Real Debugging Journey Across PySpark, Hadoop, Docker, and Kafka
I thought this would be a simple task. I already had a Python Kafka producer running. Kafka was…
Why We Built MicroLeague Sports Vol. 3
Why Sports Data Is Harder Than Most People Think Building believable cross-era simulations turned out to be less…
RAG vs. Semantic Layer: Why AI Needs Deterministic Governance
Half the market is arguing about whether RAG or a semantic layer is the right foundation for enterprise…
Data Is the Real Model: Governance, Lineage, and Provenance
Part 3 of “Trust the Machine” -> a series on building AI infrastructure that is secure, compliant, and…
The Day I Realized “It Ran Successfully” Means Nothing in Databricks Production
Here’s a scenario that plays out at every company that has ever adopted Databricks: A senior engineer ships…
What Is Apache Kafka? Plain-English Guide for Engineers (2026)
TL;DR Apache Kafka is a high-throughput, fault-tolerant distributed event streaming platform ideal for real-time data processing. It organizes…
An Engineer’s Guide to DuckDB and Modern OLAP Databases
TL;DR Cloud warehouses are built for petabyte-scale enterprise needs, and for teams working with a few terabytes, they…
Your Scraper Collected 50 Rows. There Were 4,000.
A scraper can pass every check you wrote and still be wrong about the one thing you actually…
HTTP 200 Is a Lie: A 30-Line Schema Canary for Source Drift
A scraper that returns HTTP 200 is not a scraper that returns good data. Those are two different…
