Readshot

Ongoing small irritations finally piled high enough to push me to evolve my feed reader further.

A brief history of this project:

January 2005 — A proof-of-concept toy that used an IMAP store and copied posts from a single RSS feed to an IMAP server. I was still using Thunderbird and rawdog to do my feed reading at this point. October 2005 — began using my new aggregator. It had two components — a fetcher which ran from cron, used sqlite to keep track of state, and emitted new article messages to a Spread group and an IMAP storer which read from the Spread group and copied articles into an IMAP folder. I read the IMAP folder using Thunderbird. March 2006 — I collapsed the components into a single fetcher which ran from cron, used a SQLite data store, and put messages into an IMAP folder. I also improved the feed scheduling to reduce the number of fetches on relatively inactive feeds. January 2007 — moved from the SQLite-based aggregator to a new aggregator based on PostgreSQL. The new reader implements its own IMAP server using Twisted. It’s much faster than the old SQLite solution and keeps all of the state in the database rather than having “feed state” in a SQLite database and “user state” (flagging, read state) in an IMAP folder. I made a very basic web UI with the SQLite-based aggreagtor which allowed me to subscribe to new feeds with a bookmarklet. I migrated that UI to the new aggregator. It was always inadequate and I supplemented it with manual SQL commands to manage feeds. I’d sit down to enhance the interface but keep getting distracted playing with Ajax toolkits because the UI I had in mind didn’t fit very well into the web browser.

This weekend I began working on a desktop client. It’ll be backed by the same database so the IMAP server will continue to work in case I want to read via IMAP for some reason such as to do offline reading. I have a very crude GUI which has already let me do some management of the feeds which was awkward to do before but the desktop client still exists more as doodles in a notebook than code.

The first version connects directly to the PostgreSQL database. Eventually I plan to split the database access out into its own tier. I don’t yet know enough of what I want that protocol to look like to do that now and want to get a better feel for what the client will need before taking that step. The plan to split the storage layer into its own tier guides the design of that layer.

There’s always been a bit of a mismatch between my needs in a mail client and my needs for a feed reader but until recently I put up with all the little cuts that resulted from using a mail client for feed reading because the benefits of a desktop experience and shared server state outweighed the pain of using a mail client (and IMAP) for feed reading.

I’m using wxPython to develop the client so at least in theory it should work on Windows, Linux, and Mac OS. I only plan to test and develop using Windows for the foreseeable future, though.

No project of mine advances as quickly as those that scratch an itch or alleviate a pain for me.

I still don’t know if I’ll ever release any of this. It would probably be academic since I imagine the demand for this kind of thing is very low given how excited people get about “no install” web readers. I think the the thought process that guides design decisions in the reader is more interesting to other folks than the actual code. I suppose the RSS fetcher part of the system might be interesting to other folks as its own component.