The Okopipi Project

1) Give the doc a title so people can refer to it as more docs evolve...

2) Typo at VI compliant (complaint)

3) Distribute everything - I can see the argument for the assigned servers, but all that does is give a hook for the spammers (and others) to lever against - distribute everything. Any other approach is vulnerable by in-band or out-of-band attacks (even through foriegn courts etc)

4) think about some form of block-hashing, rather than the entire body. That way you can create multiple blocks from a single message, and declare anything matching more than (say) one block-hash as spam... This gets around changing in-message text (which I have seen also) and allows header information to contribute to identifying what is spam. Some rules around blocks being either a sentence, or in the absence of punctuation x chars to the next whitepace or something.

5) remember than you can tweak and improve the spam detection algorithms (as per 4) but that the security has to be solid from day one. While my head might work better around how to identify spam, this should NOT be your primary focus at this stage - I'll happily support and install an improvable mediocre system, so long as I'm comfortable with the security mechanisms around it - THAT must be your primary focus getting the network off the ground.

6) I wonder how much you can 'borrow' from existing networks to speed up development - eg: eDonkey search or Azeurus database - but I suspect you are much more expert than me in these matters - just PLEASE don't delay Okopipi go-live by reinventing (and then designing, coding, debugging, etc. any wheels)

6a) Also, if 3rd party network softwae providers are prepared to play along (even as 'secondaries'), launching the network as distributed across hundreds of thousands of nodes from day-one protects the Okopipi network from the certain take-down attempts which will occur before it gains sufficient traction (unlike blue frog, the evil ones will be arming up for this in advanced ways long before go-live)

7) I miss blue frog - my spam levels have now trippled and are back to what they were before... Please keep up the momentum!

--J

Comments:

Section III
-----------
Hashing the body of the message won't work - most spam contains random text.

You need to do something like pick the URLs out of the message.
Be careful though because URLs can easily be randomized, eg. http://ioeiuewusadn.cheapviagra.com/

I'd take just the first two parts of the domain name, ie. "cheapviagra.com" in the above example.

Section IV
----------
There's two databases - the database of complaints and the database of client keys.

The complaints database is short-lived and constantly being rebuilt so it isn't particularly valuable in itself. I think the fewer copies of the complaints database, the better. Synchronizing multiple copies of a high-churn database will be very difficult.

The database of client keys has very different needs. It's low-churn and very valuable. You need multiple signed/encrypted copies distributed around the world. In case of compromise of the client database it should be easy to revoke the keys and get users to generate a new one.

Section VII
-----------
How do clients obtain a key? If the process can be automated then it's pointless - a spammer can set up his botnet to obtain millions of keys then use them to flood the servers.

Other comments:

Updates
-------
Will the clients be able to update themselves if weaknesses/problems are found? Can the network deliver these updates automatically?

Master key
----------
Loss/compromise of the master key would be catastophic. Where is this key stored/who guards it? (eg. Verisign?)