william's blog | 2012-02-04 11:49:45 +0000
===========================================
Rethinking Sup part II
----------------------
Date: July 20, 2008 5:58pm
Author: William Morgan
Labels: sup
URL: http://masanjin.net/blog/old13.txt
In Rethinking Sup part I [1], I concluded that Sup the MUA is an evolutionary
dead end, and that the future lies in Sup the Service (STS). But what does
that mean?
One thing I want to make clear it does _not_ mean is any abandonment of the
Sup curses UI. That particular "user experience" has been refined over the
past few years to become my ideal email interface. It would be silly to throw
that away.
What _will_ happen to the curses code is that it will become one client among
(hopefully) many. Once there's a clear delineation between UI and backend, you
can make a UI choice independent of making a choice to use Sup in the first
place. You can run sup-curses-client if you want. Or you can
build a web interface, or an Openmoko interface. Working with ncurses has
always been the least enjoyable part of Sup, so maybe I'll actually enjoy
learning Javascript.
What backend functionality will STS actually provide? If I were simply
reworking Sup into a client and a server, the obvious answer would be "a
searchable, labelable, threaded view of large amounts of email".
But reworking Sup is a great time to extend its original goals. In particular,
I would love for STS to handle to other types of documents besides email. I've
always used my inbox as a mechanism for writing notes to myself. I've
experimented briefly with reading RSS feeds through it. I'd like STS to
support email, of course, but not to be limited by it.
My grand vision: _STS will be a searchable, labelable, threaded view of large
numbers of documents._
You can throw whatever you want in there, and STS will store it, thread it,
and let you label and search for it. Email, RSS feeds, notes, jabber and IRC
logs, web pages, RI documents—I want you to be able to throw them all in
there. I want you to be able to annotate any of those things by adding notes
and threading them against the original objects. Basically I want STS to be
the primary tool you use for organizing and recalling all the textual
information you've ever encountered in your life.
Cool, huh?
There's another convenient benefit to this transformation: no one will expect
STS to act like a MUA. STS does its own storage. You add your email and your
other documents to the server and then you can throw those files away (or
not). There are no more questions of supporting IMAP or various mbox dialects
or "why doesn't Sup treat Maildir correctly". The files are in STS, and once
they're their, they're out of your hands. You'll be able to export them, of
course, and if you're crazy you might be able to write an IMAP _server_
translation layer for STS, but there will be no more expectation of realtime
Maildir handling. As I explained in part I, that's a game I don't want to
play.
STS is a grander vision than a MUA, and it no longer has to be hobbled by the
constraints of being expected to act like one.
Some other nice benefits of reworking Sup into SYS:
* You'll be able to run multiple clients at once.
* It's an opportunity to rework some things. For example, one of the most
noticeably slow operations in Sup ("Classic") is assembling a large thread.
This is because I made a decision early on to do all threading at search time.
That made certain things easier (in particular, I could change the threading
model without having to rescan the entire index), but in retrospect the cost
is too high. STS will maintain document trees directly.
* I can replace Ferret with Sphinx. It's been a good couple years, but the
periodic non-deterministic index corruption that's been an issue for over a
year [2] is an exit sign to me. Working with Sphinx is nowhere nearly as nice
as working with Ferret, but speed and stability go a long way.
I've been working on the code for STS on and off for the past couple weeks and
it's slowly starting to come together. Once the major components have at least
been all sketched, I will host a git repo.
[1] http://all-thing.net/2008/06/rethinking-sup.html
[2] http://ferret.davebalmain.com/trac/ticket/279
Replies
--------
John, on June 26, 2008 2:26pm:
["| \n", "| I love sup and I'm not bothered by any of the problems you describe so this\n", "| whole thread just seems philosophical and worrisome to me. But I'm reassured\n", "| that you think you can do whatever without breaking the Ideal Email\n", "| Experience. Thanks again for sup!\n", "| \n"]
Anonymous, on July 3, 2008 11:54am:
["| I'm not sure not being able to move mails between stores and setting state is\n", "| such a big deal. But what is a showstopper, is that sup probably doesn't save\n", "| the mails I send anywhere?\n", "| \n"]
William, on July 4, 2008 12:57am:
["| Anonymous: Sup does save a copy of your send emails. Look in ~/.sup/sent.mbox.\n", "| Moving mails around between mailstores and setting state is a frequent\n", "| request, because people (rightly) expect Sup to behave like a regular MUA like\n", "| mutt.\n", "| \n"]
Anonymous, on July 7, 2008 6:26pm:
["| \n", "| Out of curiosity, what are your mailstores? I ask because I've had to abandon\n", "| sup and go back to mutt, because anything over 50k messages, spread throughout\n", "| several Maildir mailstores, caused a LOT of instability and weird behavior in\n", "| sup, while mutt seems to handle everything without a hiccup.\n", "| \n", "| Of course, I'm now growing ever-farther behind on keeping up with my mail,\n", "| since mutt makes it too easy to ignore things, but I had to make the trade-off\n", "| of \"works\" vs \"global view\".\n", "| \n"]
William, on July 11, 2008 2:19pm:
["| \n", "| I've always used mbox files. The only instability I've seen is Ferret's\n", "| inevitable index corruption.\n", "| \n", "| I don't use Maildir, but I know a lot of people do. I'm surprised that you had\n", "| so much trouble because AFAIK they are all happy with it.\n", "| \n", "| At any rate, I am in the midst of writing \"Sup the Service\" which should fix\n", "| these issues in that *I* will control how things are stored on disk. :)\n", "| \n"]
ryan, on July 15, 2008 8:10pm:
["| This makes a lot of sense. Is the follow-up to this post on the way? Have you\n", "| made any concrete decisions about sup's future?\n", "| \n", "| I ask because I've been meaning to try sup, and possibly switch over, for a\n", "| while now. This post stopped me in my tracks, though, since it implies that\n", "| sup's future probably _isn't_ its current incarnation as a mail client.\n", "| \n", "| Switching mail clients is a big thing. It's at least a little traumatic for\n", "| most people, including me. If I'm going to do it, I'd like to know that I'm\n", "| not going to get stranded and have to switch again later.\n", "| \n"]
William, on July 15, 2008 10:07pm:
["| \n", "| The followup post is on the way, and Sup's future is pretty concrete. I went\n", "| on vacation last week and spent a lot of time hacking on the foundation, so\n", "| progress is progressing.\n", "| \n", "| \n", "| I understand completely. FWIW, there will be an easy upgrade path to the new\n", "| Sup, and the interface will be the same, and it will be possible to get your\n", "| email back out of Sup if you decide it ain't for you.\n", "| \n"]
Brendan, on July 20, 2008 7:12pm:
["| I'm a little scared by big grand software service visions. There's been lots\n", "| of failures to make big-ass browseable/searchable stores of one person's\n", "| documents. Well at least you're constraining yourself to textual information\n", "| and it sounds like you have a specific use cases in mind.\n", "| \n", "| case in point ...\n", "| \n", "| http://chandlerproject.org/ [1] your post makes it sound like their server.\n", "| but try not to be like them...\n", "| \n", "| OK well here's a more positive example. I've had a lot of success with a mac\n", "| application, Journler, for taking notes that go into categories/tags/dates,\n", "| plus draggable file attachments (finder replacement) plus full text search.\n", "| http://journler.com/ [2]\n", "| \n", "| \n", "| [1] http://chandlerproject.org/\n", "| [2] http://journler.com/\n"]
ertai, on July 21, 2008 12:56pm:
["| Great plans! Can't wait to make the switch :)\n", "| \n", "| This separation between the UI, and a fast/reliable/robust component is a very\n", "| good decision.\n", "| \n", "| However do you think that Ruby is still the good choice for STS, I mean for\n", "| the fast/reliable/robust parts?\n", "| \n", "| In the mean time I think the answer is yes, but having a good high-level\n", "| specification of the STS interface could help us to change it's implementation\n", "| more easily.\n", "| \n"]
William, on July 21, 2008 3:52pm:
["| \n", "| I hear ya. But this isn't really that grand of a vision, at least not\n", "| incrementally from what's there now. You take Sphinx + some kind of a document\n", "| store (I'm using \"files on disk\" for now) + Thrift for the interface + Sup's\n", "| email threading code, and glue it together with a little logic. Hardly a\n", "| Chandler.\n", "| \n", "| Journler seems based on a similar idea, though I think I'm envisioning\n", "| something more cohesive. Hard to say without using it. Maybe I'll give it a\n", "| try when I'm at work and have access to a macintosh apple computer.\n", "| \n"]
William, on July 21, 2008 4:19pm:
["| \n", "| Definitely, because 1. I don't want to write in any other language, and 2. the\n", "| computational and IO bottleneck is the search, and that's not in Ruby.\n", "| \n", "| \n", "| Yes, definitely. I will publish the Thrift interface once it's gelled a bit\n", "| more and we call all write Erlang backends for it. :)\n", "| \n"]
Adrian Quark, on July 29, 2008 11:31pm:
["| I like sup's UI and focus on indexing/tagging over folders, but the idea of\n", "| sup storing my mail in some opaque format is a huge turn-off. I've had to\n", "| move my mail between clients too many times to trust it to anything but a\n", "| dead-simple plaintext format like Maildir or mbox. If I wanted to be tied to\n", "| a specific platform, I'd just use GMail.\n", "| \n"]
William, on August 3, 2008 4:26pm:
["| \n", "| Sadly, that's the only way I can reasonably support the kinds of things that I\n", "| want Sup to do. Sup has never really supported mbox, Maildir, etc. very well,\n", "| because it just wasn't able to handle the concurrent modifications files in\n", "| these formats undergo when there's more than one client involved.\n", "| \n", "| With the addition of non-email document types, a custom format kinda makes\n", "| sense.\n", "| \n", "| If it makes you feel any better, there will be tools to extract content from\n", "| Sup into standard formats.\n", "| \n"]
Kelly Clowers, on August 21, 2008 3:09pm:
["| Sounds kind of like Akonadi http://pim.kde.org/akonadi/ [1]\n", "| \n", "| \n", "| [1] http://pim.kde.org/akonadi/\n"]
madduck, on October 14, 2008 7:03am:
["| I sincerely hope that you will rewrite sup so that tags and all the other\n", "| metadata you can attach to messages will be IMAP-synchronisable. Many of us\n", "| use two or more computers, and it would be useless if we had to maintain tags\n", "| independently on two machines. In fact, it would send us right back into\n", "| POP2/POP3 land.\n", "| \n"]
Jason White, on April 19, 2009 8:45am:
["| I like the idea of Sup the Service, as long as there is an NCurses interface\n", "| to it, as planned.\n", "| \n", "| After reading about Sup on a mailing list, I considered switching fro Mutt,\n", "| but decided not to due to the lack of proper mailbox handling. However, I have\n", "| no objection to having messages stored in a database (perhaps I could extract\n", "| all those compressed tar files of old e-mail and index the lot!). The\n", "| robustness of a proper database would help - I'm thinking of atomic\n", "| operations, and the data integrity guarantees provided by proper database\n", "| systems.\n", "| \n", "| Of course, the concept could be extended, as suggested, to blog posts, RSS\n", "| feeds, NNTP newsgroups, and so forth.\n", "| \n"]
Graham Dunn, on May 13, 2009 7:41pm:
["| On Wed, May 13, 2009 at 2:11 PM, William Morgan wrote:\n", "| Did you ever use Zoe (back in the pre-gmail days)? It was a java app that you\n", "| fed mail into and would Lucene everything and give you back a web interface\n", "| with things sliced up by keywords / attachments / threads / etc.\n", "| \n"]
Jason White, on May 31, 2009 5:28am:
["| \n", "| There's an even more grand vision coming from developers at Google.\n", "| http://wave.google.com/ http://www.waveprotocol.org/ (It's based on XMPP, and\n", "| combines features of e-mail, collaborative document editing - wiki style - and\n", "| instant messages). It would be nice to bring something with these capabilities\n", "| into one's ncurses environment.\n", "| \n", "| At least the specifications will be implementable in free software, and Google\n", "| have foreshadowed the release of source code under free terms.\n", "| \n", "| Whether this turns out to be a significant protocol on the net or not in years\n", "| ahead, it's at least worth looking at closely now.\n", "| \n"]
martin f. krafft, on September 30, 2009 10:23pm:
["| \n", "| I read up until here and thought \"cool\", but then I lost hope. I don't think\n", "| the desktop will be important in the future. Given mobile devices and\n", "| ubiquitous computing, we need to focus on data synchronisation and offline\n", "| operation, not the desktop.\n", "| \n", "| And apart from that, I think small visions that turn into working things are\n", "| preferable than grand visions.\n", "| \n", "| -- martin | http://madduck.net/ | http://two.sentenc.es/ \"they redundantly\n", "| repeated themselves over and over, incessantly without end and ad infinitum\"\n", "| -- ibid\n", "| spamtraps: madduck.bogus@madduck.net\n", "| \n"]
This delicious text version served up by Whisper .