What Actually Happens When You Try to Use Rust For Data Science In Production
I spent about three weeks trying to build a proper ETL pipeline in Rust. The library landscape looked promising on paper, but the reality is much more boring and occasionally frustrating. Let me walk through what I learned. The most common entry point is the kale crate, which gives you DataFrame operations similar to pandas. You also have polars for fast computation, and ndarray for raw numerical work. There is no single unified ecosystem the way Python has. You are assembling tools yourself. Here is the setup I actually used:
Add these to your Cargo.toml: polars with the "lazy" feature enabled, arrow2 for memory management, and serde for serialization. That gave me roughly 90% of what I needed for ingesting CSV files, doing joins, and exporting to Parquet. The performance was genuinely impressive. A join that took 47 seconds in pandas dropped to about 1.2 seconds in polars on the same dataset, which was about 2.3 GB of messy sensor data.
The Memory Model Is Not Optional
This is where most people get stuck. In Python, you can load a 50 GB CSV into memory and move on with your life until the kernel dies. Rust does not allow that. You need to think about column types explicitly. A column you think is integers might have a few nulls or a stray string value, and your entire parse operation fails. I encountered a case where a date column in a 14-million-row dataset contained a single malformed entry like "2023/13/01" that made the whole parser bail out. The workaround was wrapping the column in Option<DateTime> and using a custom parsing function with a fallback to NaiveDateTime::parse_from_str. It added maybe 15% overhead but prevented the entire pipeline from crashing.
Get the Full Details
Working With DataFrames in Practice
Polars uses a lazy evaluation model by default, which means queries are not executed until you call .collect(). This is actually useful because it lets the query optimizer reorder operations. I once had a filtering step that was running before a column cast, and the optimizer automatically pushed the filter first. The result was an 8x speedup on a groupby aggregation. The API is less intuitive than pandas though. Groupby and agg in Polars look like this: df.lazy()\n .groupby(["category"])\n .agg([\n col("value").sum().alias("total"),\n col("value").mean().alias("avg")\n ])\n .collect()\n
It compiles fast, runs fast, but the error messages from the compiler when you mess up the type annotations are genuinely brutal. I spent a full afternoon tracking down a type mismatch between UInt32 and Float64 that came from a subtle auto-cast in an intermediate step.
The Missing Pieces
There is no equivalent to scikit-learn for machine learning. You have burn and tch-rs, but they are not production-ready for the kinds of modeling workflows most data scientists use daily. If your pipeline involves training models, you will likely still be calling Python functions through PyO3 bindings. That adds complexity without solving the underlying problem. Vizualization is another gap. You can use plotters for basic charts, but it is nothing close to matplotlib or seaborn. I ended up exporting data to Parquet and doing all my visualization in Python anyway.
When It Actually Makes Sense
Rust is worth considering when you have a data processing step that is a bottleneck in your infrastructure. If a Python microservice is burning CPU on ETL work that runs every 5 minutes across terabytes of data, rewriting that service in Rust can pay for itself quickly. I replaced a Python-based log aggregator with a Rust equivalent and dropped the container's memory footprint from about 1.8 GB to roughly 40 MB. CPU time went from 12% average to under 2% on the same workload. But for exploratory analysis, prototyping, or anything that requires a broad ecosystem of ML libraries, stick with Python. The tradeoff is not worth it.
A Realistic Workflow
Here is what my actual setup looked like after settling in: Rust handles the heavy ingestion, cleaning, and aggregation layer. The output goes into Parquet files stored on S3. Python scripts then read those Parquet files for any modeling or visualization work that needed more specialized libraries. The boundary between the two languages is clean and explicit. Communication happens through file formats, not through runtime interop. The compilation time is the part nobody warns you about. My project took about 45 seconds to do a full rebuild. For iterative data exploration, that is painful. I switched to using cargo-watch to only recompile changed crates, which brought it down to roughly 8 seconds for most iterations. If you are serious about Rust For Data Science, start small. Pick one heavy processing task, benchmark it in Python first so you know what you are optimizing for, then rewrite just that part. Do not attempt to rewrite your entire pipeline at once. The ecosystem simply is not mature enough to support that kind of migration without significant friction.
