rust A field note by Vikrant Sharma
PyO3 is why Polars runs circles around Pandas
The Python library you call from a Jupyter notebook might be Rust underneath. PyO3 makes that translation invisible.
Polars, Pydantic V2, tokenizers from Hugging Face. All Python libraries. All Rust underneath. PyO3 is the bridge that makes this work.
The appeal is speed without rewriting your codebase. You import a library that looks like pure Python. The hot loop runs in compiled Rust. You never touch unsafe pointers or lifetimes. You just call .filter() on a DataFrame and get back results faster than Pandas could manage.
PyO3 handles the type conversions. Python integers become Rust i64. Python strings become Rust String. The library author writes the performance-critical code in Rust, wraps it with PyO3 macros, and ships a wheel that pip installs like any other package.
This is not a new FFI trick. ctypes existed. cffi existed. Both required you to think about C calling conventions and memory layouts. PyO3 generates the glue code. You annotate a Rust function with #[pyfunction], and PyO3 makes it callable from Python. The Rust compiler checks your types. The Python interpreter never sees the Rust source.
The article walks through a toy example: a Rust function that counts words, exposed to Python. The Rust code is ten lines. The PyO3 wrapper is three more. The resulting Python module imports and runs like native code.
This matters for ML pipelines. Data preprocessing is often the bottleneck, not the model. You can keep PyTorch or TensorFlow for the model and rewrite the CSV parsing or the feature extraction in Rust. The Python script stays the same. The wall-clock time drops.
Polars is the proof. It beats Pandas on every benchmark I have seen because the core is Rust with Apache Arrow under it. You write Python. The execution is zero-copy Rust. PyO3 is the reason you do not notice the seam.
