Browsing Tag
dataengineering
31 posts
An Engineer’s Guide to DuckDB and Modern OLAP Databases
TL;DR Cloud warehouses are built for petabyte-scale enterprise needs, and for teams working with a few terabytes, they…
Your Scraper Collected 50 Rows. There Were 4,000.
A scraper can pass every check you wrote and still be wrong about the one thing you actually…
HTTP 200 Is a Lie: A 30-Line Schema Canary for Source Drift
A scraper that returns HTTP 200 is not a scraper that returns good data. Those are two different…
Stop Naming Your Healthcare Columns Wrong — ISO-11179 Explained
If you’ve ever inherited a healthcare database with columns named DOB, PatientID, or CLAIM_NUMBER — this guide is…
How I cut Python JSON memory overhead from 1.9GB to ~0MB (11x Speedup)
The Problem: The “PyObject” TaxWe all love Python for its developer velocity, but for high-scale data engineering, the…
The data engineer’s Cortex Code cheat sheet
A practical guide to the commands, prompts, patterns, and habits that make Cortex Code useful in real data…
How I built a 39 compression pipeline with AES-256-GCM in Python (and why the dictionary is everything)
I store LLM training data. Every tool I found either compresses it or encrypts it — nothing did…
From Silent None to Insight: Debugging PySpark UDFs on AWS Glue with Decorators
Last month I was debugging a PySpark UDF that was silently returning None for about 2% of rows…
Building a Real-Time Data Pipeline: Streaming TCP Socket Data to PostgreSQL with Node.js
Real-time data streams are the lifeblood of many modern applications, ranging from financial market tickers to IoT sensor…
How I Redesigned a Failing Data Pipeline to Eliminate Cascading Failures
My client’s activity tracking system was breaking under load. During peak hours, employee activity submissions would time out,…
