TechTheft

Building a DNSBL

Steve Linford wrote:

"At the current pace of ever-incrementing spam levels Spamhaus predicts that by mid-2006 spam could reach 95% of all email traffic and we would at that stage see visible signs of the beginning of a slow meltdown of email delivery systems caused by overloaded email queues and stressed spam filters."

If you put it into terms that communications engineers use, you are talking about a signal-to-noise ratio of 1/9. or 0.111111.

About -9.5 dB, if I have my decimals straight.

Most noise-reduction systems require that the SNR be no lower than 10 dB to function well. That's because, in order to recover the signal, you have to have enough signal to differentiate it from the noise that is with it. By your prediction, by mid-2006 the noise will be 20 dB too high to allow noise-reduction to be effective.

Think Spamassassin and its ilk as noise-reduction filters.

Much effort has been expended (and CPU cycles wasted) trying to tackle the noise reduction at the receiving end. Some of the proposals for source-based noise reduction (SPF, for example) break the existing protocols sufficiently to be ineffective.

In digital channels which face this sort of challenge (think space fight) the engineers provide more redundancy in the encoding of the source data, transmission encodings that allow the receiver to use tighter discriminators to reject the random noise and to allow larger error vectors in the constellation without symbol substitution.

Technical remedies fail us.

In the Internet mail system, we have the problem that the SMTP protocol is very trusting of the source to "do things right". In particular, there is no effective standard method available to identify the source of a particular message with confidence.

Even a cryptographic-based certificate scheme fails when you consider the reality of trojaned computers on broadband connections -- the certificate, by necessity, needs to reside on the computer being compromised, such that the attacker/intruder could usurp the identity of the sender in spending out spews of spam. The obvious answer is to have authentication tokens within the mail system -- but the majority of users out there resist the imposition of even more passwords on their normal activity. Furthermore, automatic mailers (think email receipts for orders) would become more complex by orders of magnitude in trying to deal with the "full-condom" approach.

Legal remedies fail us.

Most legal bodies do not recognize the definition of spam that is taken for granted here in NANAE. In the United States, the Federal Trade Commission has been handed the spam hot potato, and by its charter it can only consider things of a commercial nature. Non-commercial spam is beyond their authority. And the politicians appear to want to keep it that way so they can spam without stubbing their toes.

Is there *any* country with a content-neutral, consent-based definition of spam?

Another problem is that, even if we have the laws, we don't have the enforcement. Again, those trojaned broadband machines allow a determined perp to stay hidden behind the thicket of victims.

The cost of prosecution is so high that governments and their law enforcement agencies can afford to only go after the big cahunas, leaving small operators free to operate with a minimum of interference.

Social remedies fail us.

Outside of a relatively small community of people, spammers are not held in the same contempt as lawyers, used-car salesmen, and politicians. I haven't heard of any father not letting their daughter marry a spammer. Indeed, very few people couple that flood of crap in their electronic mailbox with that mild-mannered man just down the street; you know, the one who complained when he received 7.5 tons of unwanted paper mail but who thinks nothing of heaping the electronic garbage on us.

Darwinism, the final remedy, won't fail us.

In biology, a disease that kills its victim too quickly eventually becomes extinct, because by its speed of killing it denies itself the time to reproduce and spread. The most successful parasites are those that derive the most succor without killing the host. In some cases, the two life-forms come to depend on each other: symbiosis.

Let's look at other media. Radio and television have, for the most part, discovered that 18 minutes of commercial messages per hour is about the limit that customers will tolerate. (Product placement in television programming and in theater-release movies extend this, but not by much.) Periodicals (newspapers and magazines) have limits set on them by postal regulations; in the United States the content must be 40 percent editorial, 60 percent advertising, averaged over a rather long period of time, in order to utilize the special rates for such items. Paperback books tend to cluster a small number of ads in the middle and after the end. Hardcover books rarely contain any ads at all. The telephone, in terms of content, used to be pretty bad with the telemarketers, but do-not-call lists and the widespread of called-id is helping here. Snail-mail can exhibit the same negative-SNR rate as electronic mail, but the filtering is considerably easier.

• Radio and television: SNR of +3 dB. Filtering can provide a signal gain of 6 dB (bathroom/fridge breaks) to more than 20 dB (mute/TiVo/Replay).
• Newspapers and magazines: SNR of 0 dB -- ads are shown side-by-side with editorial content. There is no requirement to read the ads in order to enjoy the editorial content. Noise filtering (ignore the ads) can provide signal gain in excess of 30 dB by good training -- and our schools are providing that training now.
• Paperback books: SNR better than +30 dB.
• Hardcover books: SNR better than +60 dB.
• Telephone: Varies, but filtering (caller-id, do-not-call) helps the telephone achieve an SNR(c) of at least +15 dB, dipping to +5 dB or so during the political season.
• Snail-mail: SNR of -10 dB. Noise filtering (by envelope examination) provides a gain of more than 20 dB.

Other push-media forms, such as fixed billboards (roadside, building-side, inside subway cars, inside buses), moving billboards (bus sides, taxi triangles, even the painted Volkswagen), sound trucks, beggars, talker-hawkers, and the like are the background noise of cities and larger towns. These can be relatively easy to learn to ignore, and so the effective "SNR" of such environmental sources of mental pollution rises to more than 20 dB with the noise filtering even most city dwellers achieve.

Notice the trend: With suitable filtering, virtually all forms of media we perceive achieve SNR values in excess of +6 dB without much work. I submit that any information system in which its users perceive a +6 dB or better SNR ratio will remain successful.

And the media that doesn't meet +6 dB SNR? The people who depend on that media better pray that something better doesn't come along. If a "less noisy" system is introduced, market forces will cause people to move from the more noisy system to the quieter system. That's the success story Sprint taught us during the telephone wars during and following the breakup of AT&T.

This also explains why newspapers didn't die, as predicted, when television started to become widespread: The perceived SNR of the newspaper is higher than the perceived SNR of television.

E-mail, as we know it today, barely makes the +6-dB threshold given the extensive -- and expensive -- receive-side filtering used by most people. As the volume of spam -- of noise -- continues to rise, even with filtering the SNR drops below the demonstrated comfort level.

Spammers, you are putting yourselves out of business. The system that rises to replace the e-mail system of today will not use the same trust model. Bet on it.

Spamhaus isn't tolling the bell on e-mail. It's just providing the raw measurement that lets us determine when RFC 821/822 electronic mail is polluted to the point of being useless to most people.

Spammers will prove, in the long run, to not be successful parasites. They WILL kill the host. They are.

Copyright (C) 2005 Stephen Satchell -- all rights reserved. Permission to republish the original text, but not for profit, is granted as long as the original text and this copyright notice are unmodified, and that proper attribution is given to Stephen Satchell as the author. Licenses to republish for profit may be granted on request; submit those requests to usenet@satchell.net.