foundationdb A field note by Vikrant Sharma
Database as queue, then Kafka, then FoundationDB
Tigris Data swapped their database-backed queue for Kafka, then swapped Kafka for FoundationDB. The reasons why tell you more about queue design than any tutorial.
Tigris Data wrote up their progression from using a database as a message queue to Kafka to FoundationDB. The interesting bit is not that they moved off a database. Everyone does that. The interesting bit is why they moved off Kafka. They hit Kafka’s rebalancing stalls. When a consumer joins or leaves, the whole consumer group pauses. For a search indexing pipeline that needs to keep up with a write-heavy workload, those pauses stack up. They also needed strict ordering within a partition, which Kafka gives you, but the operational overhead of managing topics and partitions did not fit their team size. FoundationDB gave them ordered key-value storage with ACID transactions and no rebalancing pauses. They built a simple queue abstraction on top: each message is a key with a timestamp, consumers track their position, and deletes happen in the background. No brokers. No ZooKeeper. No rebalancing. The post does not claim FoundationDB is better than Kafka for every use case. It is better for their use case: small team, strict ordering, cannot tolerate rebalancing stalls, already running FoundationDB for other parts of the system. That specificity is what makes this useful. Most database-as-queue stories end with “we moved to Kafka”. This one ends with “we moved past Kafka”. The lesson is that the right tool depends on your constraints, not the Hacker News popularity contest. If you already run FoundationDB and your queue needs are simple, you might not need Kafka at all.
Source: We used a database as a message queue. Now we use Kafka
