Browsing Tag
dataengineering
34 posts
Data Is the Real Model: Governance, Lineage, and Provenance
Part 3 of “Trust the Machine” -> a series on building AI infrastructure that is secure, compliant, and…
The Day I Realized “It Ran Successfully” Means Nothing in Databricks Production
Here’s a scenario that plays out at every company that has ever adopted Databricks: A senior engineer ships…
What Is Apache Kafka? Plain-English Guide for Engineers (2026)
TL;DR Apache Kafka is a high-throughput, fault-tolerant distributed event streaming platform ideal for real-time data processing. It organizes…
An Engineer’s Guide to DuckDB and Modern OLAP Databases
TL;DR Cloud warehouses are built for petabyte-scale enterprise needs, and for teams working with a few terabytes, they…
Your Scraper Collected 50 Rows. There Were 4,000.
A scraper can pass every check you wrote and still be wrong about the one thing you actually…
HTTP 200 Is a Lie: A 30-Line Schema Canary for Source Drift
A scraper that returns HTTP 200 is not a scraper that returns good data. Those are two different…
Stop Naming Your Healthcare Columns Wrong — ISO-11179 Explained
If you’ve ever inherited a healthcare database with columns named DOB, PatientID, or CLAIM_NUMBER — this guide is…
How I cut Python JSON memory overhead from 1.9GB to ~0MB (11x Speedup)
The Problem: The “PyObject” TaxWe all love Python for its developer velocity, but for high-scale data engineering, the…
The data engineer’s Cortex Code cheat sheet
A practical guide to the commands, prompts, patterns, and habits that make Cortex Code useful in real data…
How I built a 39 compression pipeline with AES-256-GCM in Python (and why the dictionary is everything)
I store LLM training data. Every tool I found either compresses it or encrypts it — nothing did…
