What SQLite taught me about adding a queue
An experiment in doing less: a local database, a little contention, and a much better question.
This is a sample field report about a deliberately small experiment: taking a task that sounds like it needs a queue and trying the simplest persistent version first.
The question
Could a single process collect a handful of jobs, remember them across restarts, and process them one at a time? That is a narrower question than “Can SQLite replace a message broker?” Narrow questions are easier to learn from.
CREATE TABLE jobs ( id INTEGER PRIMARY KEY, payload TEXT NOT NULL, status TEXT NOT NULL DEFAULT 'pending', created_at TEXT NOT NULL DEFAULT CURRENT_TIMESTAMP);The useful failure
The interesting part starts when you add another worker. Selecting a pending row and claiming it are separate operations unless you make them atomic. A design that was pleasant with one process can be wrong with two.
Rather than hide this behind a neat demo, I would test interruption, contention, and repeated execution explicitly. Those are the moments when the persistence model starts to matter.
A smaller conclusion
The lesson is not that every application should use a database as a queue. It is that trying a small version can expose the requirements you were about to outsource to a tool you did not yet understand.
What would you test first: duplicate execution, retries, or recovery after a crash?
Keep a little room for curiosity. — Q.
Comments
Different experience? A better approach? Leave a thought.