Articles
Notes on data engineering, open-source infrastructure, and building data platforms — published across Medium and other outlets.
You’ve scaled up your Kubernetes cluster. EC2 instances are running. CPU and memory metrics look reasonable. And yet — the number of completed Dagster runs stubbornly refuses to climb. If you’ve been in this situation, you know the particular frustration of a platform that looks healthy but processes jobs at a trickle.
How a simple configuration tool can remove weeks of infrastructure work for solo developers, small teams, and anyone evaluating modern data stacks.
A step-by-step tutorial using Data Platform Wizard to generate a production-ready Docker stack of fully equipped Data Platform scaffold.
Your Dagster instance works flawlessly for months — until one day the disk fills up and everything grinds to a halt. Here’s how to reclaim that space safely.
Dagster’s internal PostgreSQL database grows fast, and there’s no built-in retention for event logs. Pruning it later is possible — but always comes with trade-offs. The smarter move is to stop the bloat at the source.