Meerschaum v5 Reflections

Since 2020, I have been developing/maintaining Meerschaum, the lightweight, extendable ETL framework that has been the bedrock of my data engineering.

I originally picked a Python stack for a number of reasons:

Python is the perfect glue language for relying on faster projects
- ("the goal of a Python developer is to write as little Python as possible")
rapid prototyping and experimentation
ease of use / quality-of-life for users

As Meerschaum has matured, the rate of development slowed and focused on maintenance. I've been building other things on top of Meerschaum, and, after all, I'm still only one developer. In some ways, it is satisfying to have a "complete" framework to rely on (production-ready and very stable), but whenever I reviewed my features wish-list, part of me was sad that I would never have the time or willpower to implement some of the most ambitious features.

Meanwhile, I had been keeping in touch with the pulse of these coding agent tools. I first experimented with Gemini last summer and was not particularly impressed (although I found uses for boilerplate refactor work). But you know the rest of this story: I was blown away with Claude and then Codex. Suddenly, the features I had deemed "too ambitious" looked feasible.

Relying on agents for end products is one thing, but I have been extremely cautious to vet every line touching Meerschaum. I decided to take the dive by tackling the least invasive of these ambitious features: replacing Pandas with Polars. I defined the spec and at a breakneck pace with Claude's help, I shipped Meerschaum v4 in late August. Of course subsequent patches needed to follow, but the v4 series proved to me that the gloves were ready to come off.

Since August, I have been drawing up plans for Meerschaum v5. Since the Bun announcement, I have seen the winds change in the industry, and I have finally decided to make the call: Meerschaum v5 is written almost entirely in Rust.

Like uv, the PyPI package is still a first-class package for end users. Backwards-compatibility is top priority, so Meerschaum v5's Python package is a drop-in replacement (it's a wrapper atop the new Rust core). I was inspired by Polars, so cargo install meerschaum is now another first-class installation method.

This gets to the heart of what Meerschaum means to me. I have written and maintained 100k lines of Python; I know this codebase like the back of my hand. This Rust rewrite is a major shift in my career as a software engineer, and it has taken a lot of humility to part with 6+ years of handwritten code. I recall a discussion I had in 2019 where I felt the need to justify my choice of Python over a "real" language like C++. That same feeling has cropped up this year with the use of agentic models. After spending 2026 contemplating this shift, I realized that just because my tools make the job easier doesn't make me any less of a software engineer. At the end of the day, I am still solving problems that matter in a way that is enjoyable. What gives me pause is that this is the first time I will be maintaining a Rust codebase: but AI coding models have given me the encouragement to knock out a New Year's resolution and seriously learn Rust.

So here is where things are: Meerschaum v5.0.0rc1 is now available for testing (available on PyPI and crates.io), and rc2 is on its way. My internal testing has shown substantial performance improvements (more figures to come), and the CLI is lightning-fast. Give a test now:

pip install --upgrade --pre 'meerschaum==5.0.0rc1'

# or the Rust-only build
cargo install 'meerschaum==5.0.0-rc.1'