PyO3 is why Polars runs circles around Pandas
The Python library you call from a Jupyter notebook might be Rust underneath. PyO3 makes that translation invisible.
Read the note 1 min read
From the notebook
data-engineering.
6 notes on this topic.
I turned my bank alerts into an encrypted pipeline
Sixteen parsers, a server that writes what it cannot read, and what a hostile data source teaches you about production data engineering.
Read the note 2 min read
SQLite-utils now ships with proper schema migrations
Simon Willison's sqlite-utils 4.0 adds schema migration tracking. The library that made it trivial to load data into SQLite now makes it trivial to evolve those tables.
Read the note 1 min read
Someone built a ray tracer in SQL and it actually works
ClickHouse's query language is Turing-complete. Someone proved it by rendering 3D scenes with recursive CTEs and array functions.
Read the note 1 min read
Haystack is what happens when you need RAG pipelines that actually run in prod
Deepset's Haystack framework caught my eye because it treats retrieval-augmented generation as a data pipeline problem, not a chatbot wrapper problem.
Read the note 2 min read
Databricks says they unified OLAP and OLTP. I am skeptical.
LTAP is Databricks' answer to running transactions and analytics on the same engine. The pitch is clever. The implementation details matter more.
Read the note 1 min read