Jeremy Howard of FastMail.FM wrote an analysis of the effectiveness of the Spamcop Blocking List. Julian Haight of Spamcop then provided this response. Mr Haight's response contained numerous inaccuracies and incorrect assumptions; this reply clarifies these issues.
On Fri, 21 Feb 2003 11:42:37 -0800 (PST), Julian Haight said:
> Jeremy never claims his users don't send spam.
>
I am sure Julian would not claim his users don't send spam either. The
Spamcop email system is as open as any other. I know this because I am
personally a paid subscriber to that service - I purchased the
subscription a long time ago, and have analysed it thoroughly.
> However, his freemail service (and all freemail) is an attractive
> neusance which spammers have only recently begun to exploit in force.
Free vs non-free is not a correct distinction in this issue. A great many
spammers that we catch use paid accounts, created using stolen credit
card numbers.
> Many spammers have started using automated tools to script webmail
> systems. Not just for sending mail with an existing account, but to
> create thousands of accounts and send spam through each of them until
> their limits are reached. Spammers also use many hundreds of IPs
> simultaneously by exploiting open IP proxies. So I doubt Jeremy is
> really as successful as he claims at stopping the spam from his system.
>
> I also think he vastly under-estimates the amount of spam sent. Just
> because he locks one account, it does not mean that many other accounts
> are not flying under his (and my) radar.
>
If Julian had done analysis of FastMail.FM, as we had done of his
systems, he would be able to work with facts, rather than assumptions.
Here are some facts:
- FastMail.FM already analyses source IPs for open proxies, and takes
appropriate action.
- FastMail.FM requires that all new accounts be confirmed by replying to
a secondary email address.
- FastMail.FM has automated systems that calculate fuzzy-logic similarity
metrics for all accounts, and automatically lock any account which may
be similar to any other account that has been locked.
- When we have received a genuine spam notification, we have checked the
logs to identify any other email sent by the same account, or with
similar contents (e.g. advertising the same web-site). In this way we
find the number of spams actually sent (so far, always less than 100
due to our controls). On average we get 3-4 Spamcop reports when this
occurs. This is a binomial distribution with a standard deviation of
2, which means that 95% of the time we will get at least one report of
any spam sent.
- For each Spamcop notification, we get on average 3 notifications from
other services, or generated manually. Therefore the probability of
getting no notifications for a spammer is very, very low.
> Wednesday, fastmail.fm delivered 14 spam messages to spamtraps on my
> system. That is surely only a small fraction of the spam sent during
> that "spam run". These spamtraps are not known by spammers - I don't
> think this spam run is the work of revenge-seekers. Rather it is a
> successfull effort by spammers to use Jeremy's system to send spam.
Again Spamcop assumes that their systems are correct... But in fact,
Spamcop provided no notification to us of these "spam messages" (which we
know from our analysis may be nothing to do with spam). Furthermore they
do not appear in Spamcop's database of recent reports from that source.
The accuracy of these "spam traps" is highly suspect. For instance,
here is the current Spamcop BL report on Spamcop's own mail server at
216.127.43.94:
233 spamtrap reports multiply spam score from 233 to 54289; Spam score
216.128: spam report ratio (54289.000) exceeds threshold (0.020), but
there are no reports within the past 48 hours.
> If his system did not allow spam to be sent in sufficient quantity, why
> would the spammers not move to greener pastures? They are motivated by
> greed, not revenge.
>
They have. We never get repeat spammers. They try once, get blocked
immediately, and move on. This is why we get so few valid reports.
> Fastmail is worse than other freemail providers in one respect, and ths
> may be part of the reason spammers favor it. Most webmail providers
> list the sender's true IP address in the headers of the mail, providing
> an audit-trail. Fastmail does not, thus concealing the source of the
> message. This behavior is actually *worse* than most open relays. They
> at least indicate the "injecting" ip address.
This is also incorrect. All messages from FastMail.FM are signed in the
headers with a unique hash that identifies the sender and the date.
Whereas other systems that use source IP reduce the privacy of the sender
(IP addresses can provide an exact geographic location) and are open to
abuse (they can be readily forged), FastMail.FM's cryptographic approach
is both private and secure.
> However, at least the current blocking of fastmail is justified. If it
> makes anyone fell better, several AOL and hotmail servers are also
> blocked, and those sites are also scrambling to stop the spammers using
> their systems as open relays. It is a hopeless, or at least up-hill
> battle, given the nature of free web-mail.
This is correct. Spamcop's approach is hopeless. I do not believe that it
is a useful approach - it is very damaging to the correct functioning of
the Internet. It does not block spammers (for instance, blocked servers
can simply be removed from the pool of sending servers, and
AOL and Hotmail have thousands of sending servers randomly chosen for
each message - as documented in my paper).
However, there are some very hopeful approaches which use more
sophisticated techniques to avoid the problems I have documented. For
instance, the rapidly evolving "Vipul's Razor" uses the following approach:
- Spam reports are sent to a central database (like Spamcop)
- The *body*, not the *source IP* of the spam is analysed (so the
problem of failed source identification can not occur, and senders
using Hotmail/AOL types systems with many servers still get blocked)
- The sender of the notification is classified based on a trust metric,
calculated from how many reports they have previously sent, and how
many have been revoked
- Future messages are flagged spam if their bodies are sufficiently
similar to a message where enough trusted reporters have report a
message as spam
- Razor users can revoke incorrectly reported messages
Whilst Razor is not as yet perfect, it is tackling the right problems.
I believe that it would be more productive for Spamcop to focus on their
excellent reporting and notification service, and support Razor or a
similar system for spam blocking.
> I find it disturbing that Jeremy has decided to shovel dirt about
> SpamCop rather than working with me and addressing the valid complaints
> of people who receive spam from his system. Sounds a lot like killing
> the messenger who brings bad news.
We get 4 non-Spamcop reports of actual problems for each Spamcop report.
We do not need this messenger. This news of this messenger is no longer
trusted because the "bad news" is so often incorrect.
We have worked with Spamcop in the past and identified many of the
problems documented here, through the Spamcop forums, through private
email with Julian, and through messages to the Spamcop Deputies list.
Whilst they have resulted in some changes, on the whole the problems were
not addressed. For instance, a message to the Spamcop Deputies regarding
the "Spamtrap reports" Julian mentioned is as yet unanswered.
Update: Spamcop has now got back to us regarding our message to Spamcop Deputies about this matter. We are now working with Spamcop to determine the nature of these messages.
Therefore we decided to let the wider community know about these problems, so that email providers would know to be wary in utilising this service. Thank you for providing this opportunity to respond, Jeremy Howard