Tempfailing for spam -- where does it lead

Submitted by brad on Fri, 2007-01-26 16:27
Topic: 
Spam

One growing technique for use in anti-spam involves finding ways to "fail" on initial contacts for sending mail. Real, standard conformant mail programs try again in various ways, but spammers, in writing their mail blasters, tend to just have them skip that address and go to the next one in their list.

Two common approaches include simply returning a "temporarily unavailable" status on any initial mail attempt that might be spam. Another approach is to have dead MX records both at the "try first" and "try last" end of the MX chain.

Why does this work? Spammers just want to deliver as much mail as possible given time and bandwidth. If one address fails for any reason, it's really no different whether you spend your resources trying the address again or in a different way, or just move on to the next address. In fact, since many of the failures are real failures, it's actually more productive to just move on.

And, I admit, some of the spam filtering tools I make use of use these techniques, and they do help. But what exactly are they doing? For spammers, the limiting factor is bandwidth. Dealing with failures, especially timeouts on dead servers, takes very little of their resources.

It doesn't reduce the amount of spam they send, at least by much, it just redistributes it to those who don't use the techniques. For a positive spin, you can liken it to putting up a higher fence than your neighbour, so the criminals attack them and not you. For a negative spin, you can imagine it as being like an air filter that filters out the pollution on air coming into your house, and spews it out the back at your neighbours.

So it's a touch question. Is this approach a good idea? Especially at the start, it was very effective. Over time if it becomes very common spammers will see a reduction in spam they deliver and make fairly simple moves to compensate for it. Is this fair game or antisocial?

There is an old joke about two hikers who meet a bear. The first sits down and starts putting on his running shoes. The other says, "What are you doing, you can't outrun a bear!" and the first says, "I don't have to outrun the bear, I just have to outrun you."

Are we passing the bear onto our neighbours?

(This is part of a larger question of some of the other negative consequences of anti-spam. For example, as text filters got better, spammers moved to sending their spam as embedded images which filters could not easily decode. The result is more and more bandwidth used, both by spammers and victims. Was it a victory or a loss?)

Share on:
  • Reddit logo
  • Facebook logo
  • Twitter logo
  • LinkedIn logo

Comments

Anonymous

Mon, 2007-01-29 04:35

I think that your observations show that technical solutions
will never stop spam. As you point out, it gets redirected to
the technically less savvy and/or causes the spammers to change
their strategy in a never-ending arms race.

If everyone accepted email only from trusted SMTP relay servers
which charge a fee for use (peanuts to a normal user, a very small
cost item for someone who legitimately has many emails to send, too
much to make spamming worthwhile for a spammer) and block access
from those who attempt to spam (if they do so despite paying the fee
and agreeing to abide by the terms and conditions which of course
prohibit spam), then spamming would become so unprofitable that
it would stop.

Chris Burkhardt

Mon, 2007-01-29 12:02

I've heard that proposition before, and there are a few things about it which make me nervous.

First and foremost, mandatory whitelisting, as you propose, should be the last resort. It centralizes the mail system and severely limits my freedom to send mail to whom I want from where I want.

Even if such a scheme is implemented, it's possible all it would do is force spammers to become more efficient and slightly less profitable. More efficient in that they would only send mail to known recipients, no more of this dictionary stuff -- and that's assuming these trusted relays charge even for refused connections (which would be controversial). Less profitable if they eat the relay costs themselves instead of billing their clients for all or part of them.

The best thing would be if 1) ISPs do a better job at recognizing and blocking zombie PCs sending spam, and 2) if naive business would stop paying "email marketers" thinking they will make new customers (I can't believe enough people respond to spam that they actually do... but maybe I'm wrong, in which case number 3) would be naive people not responding to spam advertisements).

Chris Burkhardt

Mon, 2007-01-29 12:08

Also, what happens when a spammer cracks the login information for someone's account to a trusted relay and sends 6 million spams through it? The hacked victim would be stuck with the bill, and there would still be spam. A partial (and sane) solution would be for trusted relays to throttle outgoing mail depending on the client... then the spammers would need to crack a cooporate/mailing list account, and the owner of such an account would likely be more capable of taking legal action against the spammer.

just some thoughts...

Phillip Helbig

Tue, 2007-01-30 01:58

> First and foremost, mandatory whitelisting, as you propose, should
> be the last resort. It centralizes the mail system and severely
> limits my freedom to send mail to whom I want from where I want.

First, this battle is already lost. There are already lots of
mailservers which don't accept email from arbitrary IP addresses.
Thus, even now you can't be sure that the email you send from where
you want will actually arrive. With my scheme, you can still send
email to whomever you want. I don't see the need to raise the
spectre of censorship here. If you run your own mail server, do
YOU accept email from an arbitrary address? If so, you will get
lots of spam. If not, what's the difference on relying on someone
else to receive your mail and relying on someone else to send your
mail?

You can still send mail from anywhere by SMTP-authentication. Or,
do as I do: the computers at home are always running, and from
virtually anywhere in the world I can access them remotely and
send email from there as if I were at home. This has the advantage
that one can't tell from the email headers where I was physically
located when I sent the mail. (Not that I care, but if you worry
about burglars checking up to see if you are at home....)

Centralise? In the sense that not every IP address can send email,
yes. But in general, no. It will create a market for cheap,
efficient, reliable, trusted SMTP relay servers.

> Even if such a scheme is implemented, it's possible all it would
> do force spammers to become more efficient and slightly less
> profitable. More efficient in that they would only send mail to
> known recipients, no more of this dictionary stuff -- and that's
> assuming these trusted relays charge even for refused connections
> (which would be controversial). Less profitable if they eat the
> relay costs themselves instead of billing their clients for all or
> part of them.

Spam can be recognised by sending LOTS of email messages from the
same IP. Otherwise, it's not profitable. When the relay server
notices that, it blocks email from that address and checks with
the customer. I'm not talking of anonymous trusted servers, but
rather of a case where, to reliably send email, you sign up with
such a service (as I have and the costs are peanuts). If the
customer is a spammer, the owner of the server takes him to court.
If the problem is a virus-infested PC or whatever, the customer has
to correct the problem.

> The best thing would be if 1) ISPs do a better job at recognizing
> and blocking zombie PCs sending spam, and 2) if naive business
> would stop paying "email marketers" thinking they will make new
> customers (I can't believe enough people respond to spam that
> they actually do... but maybe I'm wrong, in which case number
> 3) would be naive people not responding to spam advertisements).

The businesses are not naive. They pay spammers because it works.
Even the people who respond might not be naive; they might really
be interested in buying the spammers' wares. The problem is that
they don't realise how large the collateral damage is when they
respond. The spamming model works because it is so cheap to send
email that it is still profitable if only a tiny fraction of
recipients respond. I don't see how we can stop that. The only way
is to make sending email more expensive---prohibitively so for
spammers, negligibly so for everyone else.

As to 1), which ISP should recognise and block where? That's what
I'm talking about with my scheme. This is what makes these
SMTP relay servers worth using: they can be relied on to recognise
and block spam being sent out. You might be thinking they should
recognise and block it coming in. They can, but again that's a
service which has to be paid for, directly or indirectly. It's more
efficient, of course, to stop spam going out than to recognise and
block it coming in. Although I might move spam checking for
incoming mail to my ISP, with my scheme the user can handle this
end himself if he wants to and fine-tune it to his needs. Letting
the ISP do it is using a black box, more or less. However, if I send
outgoing email through an SMTP relay server, there is no loss of
functionality on my side.

brad

Mon, 2007-01-29 12:04

Search the blog or my web site for my very negative comments on that. I don't think all technical approaches fail. If you "take one for the team" and accept and then discard the spam, it does not direct the spam at your neighbour as directly as tempfailing.

Tempfailing deserves particulary scrutiny because the more people do it, the worse it actually will work, unlike some other technical measures, such as rate-limiting, which get better the more people do them.

Alex Goldman

Mon, 2007-01-29 14:47

So you like solutions like SenderBase that track the reputation of a sender over time? I like the idea, but if everybody adopts it, wouldn't IP spoofing or (DNS hijacking -- something I don't understand) get around it?

brad

Mon, 2007-01-29 16:07

You can't spoof the IP of an SMTP session, it is TCP based. Only UDP and IP can be spoofed, and that's getting harder. DNS hijacking is an attack on DNS based verifications, and DNS can be and should be secured, but you don't need dns verification to track the behaviour of IPs.

However, no anti-spam solution (other than content filters) easily deals with the botnets. They can bypass flow limiters and use the sucker's money to pay e-mail taxes and even send mail from your friends to get past whitelists. The only way to defeat that is to secure workstations from infiltration, a tall order.

Phillip Helbig

Tue, 2007-01-30 02:04

I'll re-read your comments, but I think that paying to be able
to send email (again, peanuts for legitimate folks, prohibitively
for spammers) is the easiest and, overall, cheapest solution. Other
solutions have costs as well, of course. In my case, and I don't see
why it isn't typical, paying to send email is a negligible fraction
of my internet-connectivity costs. Note that I don't pay per message
or per byte or whatever, but rather a flat fee. No different, really,
than a flat-rate DSL connection.

Phillip Helbig

Tue, 2007-01-30 03:11

I just re-read http://www.templetons.com/brad/spam/endspam.html
and find the concepts remarkably similar. Your "good guys" and
"unknown guys" correspond to my "trusted SMTP relays" and
"everyone else". I don't think your system would be acceptable
in practice since very many people have dynamic IP addresses. Yes,
I can pay more to get a static one, but I would rather pay something,
but less than the cost of a static IP address, to be able to send
email through a trusted SMTP relay server. With dynamic IP addresses,
there is no way one can get on the white list, unless the provider
gives known good customers addresses from that pool. That takes a lot
of organisation. With my system, one essentially pays to get on the
white list immediately. In my case, which I think is a good model
of how to do things, the SMTP relay server is run by the dynamic-DNS
provider (not by my ISP). He knows my IP anyway. He can (and
occasionally does) take action against anyone who abuses the system.
It works. It's cheap. And even the simplest user can get on the
white list immediately. In principle, I could switch to a different
SMTP relay server, if I wanted to (but it would have to use
authentication rather than be IP-based).

brad

Tue, 2007-01-30 12:18

See e-stamps for a page that contains my original essay proposing pay-to-send E-mail from 10 years ago (I was one of the earliest) and the postscript where I later came to denounce my own ideas.

That e-mail is so efficient and cheap remains a feature, not a bug.

Anonymous

Wed, 2007-01-31 06:52

> However, I have since abandoned and now even oppose the idea for a
> variety of reasons. These include the total failure of the several
> serious attempts to build an online money "micropayment" system

My scheme of trusted SMTP servers has no need for "micropayment". It
is not a charge per email, but rather a flat fee. Flat-rate DSL
connections exist; there is no need for "micropayments" per byte
or whatever. Similar concept.

> or
> other such infrastructure, and the almost impossible problems raised
> by any solution that needs new software at both sender and
> recipient.

No new software is needed.

> There are also free speech concerns.

I think that spam is not covered by free speech. ANYONE can still
send me an email under my system. However, I can reject it (if I
choose to; no-one forces me to) if it doesn't come from a trusted
server.

I think the trusted SMTP server scheme is much closer to your
trusted and unknown areas of the net than to your old estamps idea.
The cost (paying for the use of a trusted SMTP server) is a red
herring; it's really no different than paying for the internet
connection itself.

brad

Wed, 2007-01-31 12:39

Correct, flat fees do not have the micropayment problem, nor do CPU-coins or similar. I have indeed pointed out that they are much more workable, though should not be the only option. All the money solutions smell a bit like extortion, though. They aren't extortion, but they smell like it. "Nice E-mail you've got there. Be a shame if our spam filters were to block it. But for a small fee we can assure they won't"

I realize that you _think_ that spam is not a free speech issue. We'll just have to disagree on that one. Every free speech issue has the anti side insisting that it's not a free speech issue. That's a zero-information statement. The truth is E-mail is a form of speech, so any regulation of E-mail, both personal and private, has free speech concerns, if only because no regulation is perfect.

Indeed, can you think of a more important form of speech than E-mail today? It's replaced mail, faxes and phone calls for many forms of communication.

However, I have other blog threads on this, this thread is to focus on tempfailing and whether it's the right tool in the spam war.

Phillip Helbig

Thu, 2007-02-01 02:53

I think we're not so far apart on the technical side. I don't
see the "smells like extortion" problem. You seem to be thinking
of this AOL scheme where the SENDER could pay to have an email
bypass a spam filter. I agree with you here: that is wrong, wrong,
wrong. What I'm thinking of is that the RECEIVER can decide (if
he wants to; no-one forces him to) if he wants to accept from only
trusted SMTP servers. If he runs his own email server (as you and
I do), this is entirely within his control and can be managed by a
RBL (at no cost). If he doesn't run his own email server, this is
a service his ISP could offer. Perhaps the ISP would charge for it.
If the customer thinks he charges too much, then he can move to a new
ISP. Free market. The ISP might even charge LESS since he can just
drop such SMTP connections without having to forward the mail to
the customer, i.e. a business model "pay if you want everything, pay
less if you want to accept mail only from trusted SMTP servers".

Again, this is happening already. Soon after I set up my stuff
at home, some emails bounced (most of them to academic addresses).
It turns out that they were rejecting emails from dial-up IP
addresses (probably via a RBL). For a SMALL additional fee (we are
talking about a few dollars a year, much less than I pay for my
DSL connection per month (which costs me EUR 30 per month for 16 Mb/s flatrate) my dynamic-DNS provider lets me send outgoing email through
his SMTP relay server. Now, I let most of the spam through (I do
reject a few IP addresses and From: addresses which have spammed me
several times, as well as dropping all connections to non-existent
usernames (dictionary-attack spam)), but since the volume has picked
up recently, I plan to be more aggressive in filtering. For a small
fee I could run all INCOMING mail through my dynamic-DNS provider,
who would scan it and tag it as spam based on content (and some other
criteria); I could then filter these out at my end (spending less
time on them the more I'm convinced that I have set the proper degree
of strictness for the spam tagging). Another alternative would be
to continue to receive stuff directly, but drop more connections,
specifically those from "bad" IP addresses. Actually, although I
might have an additional white list, it would primarily be a black
list, perhaps using publicly available RBLs.

With a) many people using volatile IP addresses and b) much spam
coming from virus-infected PCs, it seems that the strategy of
accepting email only from trusted SMTP servers is the only way to
a) cheaply and b) efficiently fight spam which is c) relatively easy
for everyone to use.

Free speech? That doesn't cover shouting "fire" in a crowded theatre,
right? While there might always be an issue, the outcome of the
issue is not always that "everything is allowed since it is free
speech". Maybe the laws are different in Europe than in North
America. In Europe it is illegal to place unsolicited commercial
telephone calls or send faxes (i.e. telemarketing). Spam is
essentially similar. Legally, it is the same. Practically, it is
easy to trace a telephone or fax number and press charges.

Again, this is not that much different than throttling email from
unknown addresses and letting the stuff from known good guys through
quickly. However, with the trusted-SMTP-server concept, everyone
who runs a server (an end user, or his ISP if he doesn't run his
own mail server) can decide for himself which servers are trusted
and a customer can move to an ISP with a different concept. If
email is slowed down, it's slowed down, and this is a decision
made by the person running the server which the customer can't
influence. That is less control in the hands of the end user, so
I'm surprised you prefer that concept. My concept is "pay to be on
the white list". Since the payment is a) voluntary, b) small, c)
might even be NEGATIVE since it means less work for the ISP (i.e.
you would get a discount for accepting only mail the ISP thinks is
OK) and d) much, much less than other internet-connection costs,
I don't really see the payment as an issue at all, especially since
no-one is forced to do it. (That is, no-one is forced to do it
with respect to receiving mail. You can still run your own mail
server and do what you want. With respect to sending mail, you might
get rejected by some people, but again that is happening already.)

brad

Thu, 2007-02-01 12:38

In my proposals is that you don't refuse mail from unknown or untrusted servers, you throttle its volume down to the volume of person to person mail. In this case, the only people who truly need to get on the trusted list (something that might take money, or at least time) are people hosting (not running, hosting) mailing lists and large sites.

I want the minimum collateral damage, and decided if I could limit the collateral damage only to people who host mailing lists and possibly big sites, I would be doing a lot better than the systems we see out there today. Some people in the anti-spam community are too rabid, unfortunately, and reject anything that lets even a tiny number of spam through.

Phillip Helbig

Fri, 2007-02-02 12:17

I see your point, and I think your scheme is a good one and
would solve many problems. However, 99% of the spam I am getting
now is coming from virus-infested PCs (i.e. spambots on PCs the
owners of which don't even know they are being used for spamming).
Since I run my own mail server, they send stuff directly from
their address to my port 25. Since there is no middle man, then
I have to deal with these connections myself. I can't throttle them;
I can only reject them or accept them. (More precisely, I could
run a tarpit and slow them down, but also slow down real mail I want
to get quickly.)

How could I avoid this with your scheme? I see how your scheme
would work if the email was relayed somewhere on the way, but
what about point-to-point connections?

Of course, one could combine the two schemes: I could accept from
trusted servers, and these servers, if relaying, would throttle
stuff from addresses not (yet) known to be non-spammers.

brad

Fri, 2007-02-02 14:07

But in effect, there is a magic low-score MX record which is a network of throttling relays. Sites that know they are trusted ignore that MX record and mail you directly (and get through because they are trusted.) Sites of unknown status follow the spec and mail that MX and are throttled if not trusted. Non-trusted sites mailing directly, disregarding the spec, don't even open a socket.

This requires a large network of cooperating sites ready to run this large MX network as a central front against spam. However, as the goal of the machines is to be slow and throttling, that is not a hard engineering problem. (At least the being slow part.)

Phillip Helbig

Mon, 2007-02-05 04:38

I think some spammers have already caught up on this one.
Anyone can do a SHOW MX on an address and get one or more
MX records. Normally, the highest-priority should be taken,
and if that fails, move on to the next etc. Apparently, some
spammers are selecting them randomly, or even going for the
lowest-priority one on the assumption that the most spam checking
will be done on the highest-priority records. (This assumption is
not always true, of course, but as long as it is true more often than
it is false, it is worth it to the spammers.)

The large MX network that you mention, though, can be used by me
only if I use it for my MX records. It won't help me at all if
I continue to receive email directly (i.e. not only receive it
directly on my computers, but have someone sending me email make
a direct connection to my port 25).

If this network comes into existence and I choose to use it, then
in effect I am letting someone else decide what gets through and
what does not. Apparently SOME spam would still get through.
If I am going to let someone else decide anyway, I don't see any
reason not to choose someone who will just reject spam outright
(rather than throttling it), according to criteria I choose (my
dynamic-DNS provider offers such a service). I don't have any
more qualms about dropping connections which are a) from known
spamming IP addresses or b) which are trying to transmit spam
outright than you do setting up spam filtering for comments to
your blog. You don't let spammers and bots into your blog because
it is "free speech"; why the different stance on email? Yes, your
scheme would allow someone to send legitimate mail from any IP
address. How often does one need to do this? I think most people
send from the same computer most of the time (if it has a dynamic
IP address, then this "same computer" might be one other than their
own) and/or use SMTP-authentication.

brad

Mon, 2007-02-05 11:39

I suggest you read the article, it goes into all these details I think.

Sites don't accept e-mail connections from unknowns connecting directing. Unknowns can only go through the top priority MX records which point to the MX network because those are the only sites that will even let them open an SMTP connection. They can try random addresses all they want, it won't do any thing for them.

Phillip Helbig

Tue, 2007-02-06 13:08

I've read the article several times. :-)

OK, to use your scheme I have to stop accepting
connections directly to my port 25 and accept them
only from "trusted servers". That's essentially
the same thing that I have in mind. With your scheme,
I can send email myself, but only through a trusted
server (at least if I am sending to someone wanting
to avoid spam and who thus only accepts mail from trusted
servers), and I should accept connections also only from
trusted servers (unless I want to get spam again). Under
my scheme, it's essentially the same: I send through a
trusted server (provided by my dynamic-DNS provider) and,
if I drop connections from non-trusted servers, essentially
also accept email only from trusted servers. Not much
difference really.

The only difference is that, under your scheme, an unknown
could send an email into the MX network and it would
eventually get through. In my scheme, he could do so
with no penalty if he sent it through a trusted server.

Do you know of anyone running such a throttling server who
invites everyone to send email through it? If I had the
resources to run such a server, I would rather relay email
from people I know to be trustworthy (and who contribute to
the cost) rather than accepting from anyone and throttling
unknowns (most of which are spammers).

In both schemes, to stop spam computers have to stop accepting
port 25 connections from "just anywhere" and accept them only
from trusted servers. In both schemes, email has to be sent
through trusted servers (one which will relay legitimate email
immediately in my case and not even accept other connections,
one which will relay known senders quickly and unknowns slowly
in your case). In practice, my scheme already works for those
who want to use it. Do you know of anyone running these
throttling servers?

I think your scheme would work well if it existed, but do you
see any way for it to come about? My scheme can be built up
piece by piece. More and more people will stop accepting
connections from arbitrary addresses to stop spam, and accept
them from trusted servers. There is a motivation to send email
through trusted servers, and a market and competition. For example,
dynaccess.com offers a "reasonable" number of emails as part of a
flat fee (and it is possible to register special sender addresses
for sending newsletters etc) with access based on IP address, whereas
dyndns.org offers a micropayment-per-email scheme using
authentication.

brad

Wed, 2007-02-07 12:16

You don't have to break anybody's ordinary email. Your system forces people to change how they send mail. You want a system where the default is to do nothing, and everything still works, unless you are hosting a mailing list.

Phillip Helbig

Wed, 2007-02-07 12:44

What am I missing? With your system as well, people have
to change the way they send mail. They have to send to this
MX network. They can't send directly to port 25 of my (or
any) address. Thus, your system makes people change the how they
send mail as well.

Again, most of my spam is coming from virus-infested PC, spambots
etc, most on volatile IP addresses, who are sending directly to
my port 25 on my IP address. The only way to avoid these emails
is to drop the connections. The only way for real mail to get through
is if I accept connections from trusted relay servers. If other
people do the same, the only way for my mail to get out is to send
it through a trusted server.

All of this applies to both schemes.

I think the only difference is the way the trusted servers
operate. My scheme: non-spammers get someone to relay their
mail (I pay for this, but there are other possibilities). This
server rejects attempts to relay through it from non-customers.
Your scheme: the servers relay everything, but unknown stuff
is throttled. Spammers will realise this, and will send directly.
It will only get stopped if people accept email only from trusted
servers.

With your scheme, there are problems for people who legitimately
send many emails. For normal users, either the server has
to keep track of when the last email from a particular address
arrived, or it slows stuff down coming from unknown addresses.
The former is probably too much work. The latter will slow down
legitimate email from people without a static IP address.

With most people having a volatile address these days, one would
have to use SMTP authentication or the owner of the relay server
would have to know my current IP address. I think your scheme
would only work for someone with a fixed IP address.

brad

Wed, 2007-02-07 18:46

With my system you make no change to your mail system. You don't change any configuration or any practice (as long as you don't host mailing lists). For those not hosting mailing lists, nothing breaks, nothing fails, nothing stops working.

Making mail stop working for everybody until they reconfigure their servers is a non-starter.

Phillip Helbig

Thu, 2007-02-08 10:21

Presumably, you mean that no-one has to change anything because
in your scheme this will be handled transparently by MX records
which belong to this "white hat" MX network. This DOES prevent
me from receiving direct connections to my port 25, though.
(I could still receive them, but most would be spam.) So, someone
has to run the MX server so that I can get email addressed to me.
I don't see any advantage of this as opposed to someone running a
server which I can send email out through.

Your system will work if the MX servers do the throttling. What,
other than the wish to be a good guy, is their motivation to do
this?

How, as someone with a dynamic IP address, can I avoid the
disadvantages of being suspected of spamming if I send several
legitimate emails within a short time?

Again, I think your system will work IF enough MX servers are
willing to operate as you propose. Are you aware of any which
are?

With your scheme, people stop getting spam if they have such an
MX server send them mail. I think this is considerably more work
than finding an MX server to send mail out through.

brad

Thu, 2007-02-08 11:30

In the system:

  • People who want to use it to filter spam set up special low priority MX records.
  • People who know they are on the trusted list are encouraged to, but not required to, modify their SMTP sender to ignore the special MX records (they have a magic name)
  • People who are not on the trusted list, or who have a rules-conformant SMTP sender relay mail through the magic MXs. It delivers single mail but not bulk mail.
  • People who try to bypass the special MX servers who are not on the trusted list are rejected immediately, perhaps with errors the first few times for politeness.
  • People who are on the trusted list with a modified SMTP sender just send mail directly to the lowest priority MX that does not have the magic name. Ie. it's just the same as before. Peer to peer SMTP, highly efficient.

If most people join the trusted list (and all list hosts need to) then mail operates just as today. No sender is forced to change a thing which is essential. For trusted senders, it's as though the throttles are not there (just like today).

Phillip Helbig

Fri, 2007-02-09 13:00

Thanks for the explanation. OK, some things are clearer
now. However, it seems that I could no longer receive
email directly, i.e. have an MX record pointing to my IP
address or, if I did, I would have to reject stuff from
non-trusted senders. Right? (This is certainly possible.
In fact, I could do it now. But then I have to have someone
not just carry the MX record, but route all of my mail through
his server.) Also, would it be possible for a dynamic IP address
to get on the trusted list? It seems that these folks will
always be on the non-trusted list, unless an ISP wants to vouch
for all of his addresses, in which case he would probably require
customers to send email through his own SMTP relay server.

brad

Fri, 2007-02-09 14:18

Dynamic IPs are indeed more difficult to put on trusted lists. It's not impossible, but it's a lot harder. Most people with dynamic IPs don't send directly today, and they don't host mailing lists. In fact, most ISPs these days have taken to not even letting dynamic IP hosts use port 25, you have to manually ask them to remove the block -- if you can even ask. However, their mail will still work, just through the throttles.

The goal of the plan is for most of the main SMTP servers in the world to join the trusted lists, so that 99% of the non-spam flows peer 2 peer as it should. Then 1% of the real mail, and almost all the spam, goes through the throttle servers.

Phillip Helbig

Fri, 2007-02-09 16:04

I suspect that many people, like myself, who don't send email
directly from dynamic IPs can't do so because someone they
want to send email to has rejected their mail since it comes from
a dynamic IP address. Technically, there is no reason I couldn't
send email directly. Many probably don't host mailing lists
for the same reason. I can avoid the problem by sending through
the SMTP relay server provided by my dynamic-DNS provider, at
negligible cost and with no need for SMTP authentication (since
he knows my current IP anyway). From my point of view, I just
need to change one item in my SMTP configuration.

A static IP address is another option. That would cost more than
using the SMTP relay server at the provider.

Right now, I still receive email directly. I could channel that
through my dynamic-DNS provider as well, but I lose some control
in the process. Still, I might go this route and let him sort out
or rather tag the spam rather than implementing more anti-spam
measures at my end. (I can adjust the anti-spam level rather
finely along a scale, but not decide on whether to use one method
or another, if I let him do it. If I do it myself, I can also
experiment with various methods.)

While my scheme (everyone using a trusted server to send email
through and rejecting email from elsewhere) would cut out spam
from getting through, your scheme would actually reduce the
volume of spam. Dynamic-IP folks could still send through a
trusted server, as I do now.

It seems to me that your scheme does offer many advantages, but
someone needs to run these magic MX servers. Suppose I opt to
have my provider have the main MX record for me (i.e. route all
incoming mail through him) and try to convince him to run one of
these magic servers. If someone tried to send a lot of spam to
my addresses, or other addresses he handles, then they would get
slowed down. However, a typical spammer probably has a more or
less random list of addresses, meaning they would go through
many MX servers. So, to get off the ground, a lot of folks would
have to start cooperating at once; it can't be built up piece by
piece.

Is there an out-of-the-box, turnkey solution for people who would
be interested in running such a server?

I read that Richard Branson is offering 25 million for someone who
finds a method of disposing of excess carbon dioxide. Why not
convince him (or someone similar) to offer a similar prize (perhaps
somewhat smaller) for coming up with such an anti-spam solution.

Some folks do lose money through spam (due to the time and effort
applied to filtering etc); I think it might not be that difficult
to get enough people together to offer quite a substantial reward
for coming up with a foolproof anti-spam system.

brad

Fri, 2007-02-09 18:49

Right now anti-spam is a big business, not much need for a prize.

In my plan all that's needed is a large enough network of spare servers to act as these MXs that you can be sure at least one of the ones you use will be up, and the network has the capacity to handle the real mail. The machines are sharing via an internal channel the frequency counts of mail attempts from the billion untrusted IPs, and start to simply refuse connections for a while from IPs sending mail too fast, so their mail load can be managed. But it will still be large.

However, since speed is only a secondary consideration it's something to easily run on spare servers, older servers, machines sitting around etc. You could also run one of your own if you like but generally you would just subscribe to somebody else's. (You could also run one of your own on a 2nd IP address used by your own mail server, so no new hardware is needed, and just nice the process down low.)

The servers do delay mail by long enough to send and receive the regular updates on mail volumes over IP blocks. I haven't designed a protocol for that but I think a fairly efficient one should be possible.

But again the key gain is the rest of the world doesn't change their SMTP senders at all, not right away. Very little breaks because you install it (only large mailing list hosting on unknown sites.) Most other anti-spams tend to break things right away for large numbers of innocents, which makes them poor choices.

Anonymous

Mon, 2007-02-26 03:41

Why don't you set up an "Ask Brad" feature, either here or elsewhere.
The idea would be that instead of only you initiating threads in your
blog, i.e. blog entry, comments and responses, readers could ask for
your advice, and your answer could be handled like a normal blog entry,
i.e. spawn comments and responses etc.

Of course, you could screen the entries before posting the questions and/or
your responses, require registration for those asking questions etc.

The idea is that many readers of your blog have common interests and often
readers ask questions in the comments to your blog entries. The only difference
is that the readers (with your screening and approval) could start a new blog
thread.

Let me start (in this thread since it has to do with spam): Most people don't
have time to manually screen their spam. What do you recommend: using SpamAssassin
or something similar to mark messages as potential spam, so that the user can then
take whatever action he wants (delete them all, screen all by hand, apply additional
processing (semi)automatically (perhaps dependent on the score) but otherwise ACCEPTING
all spam messages, or, on the other hand, rejecting spam messages above a certain
spam threshold?

With some sort of automatic rejection (which I would like to implement for spams with
a score so high that I can be sure that I am not rejecting any non-spam messages), do
you recommend sending an email to the (perhaps forged) sender, saying that the message
could not be delivered because it is suspected of being spam? The reason I ask is that
my provider can let me have all incoming mail scanned by SpamAssassin before it is
delivered to me. (More precisely, all mail using a particular set of MX records. My port
25 is still open so that other email addresses are possbile which can be delivered bypassing
the spam screening.) Above a certain score set by him (5 SpamAssassin points at the moment),
it is tagged as spam. If I want, I can reject mail above a certain threshold. If this happens,
the (perhaps forged) sender will get an email saying that the mail couldn't be delivered because
it is suspected of being spam. I have no control over this.

I see two potential problems. One is backscatter spam. A spammer uses the addresses he wants
to spam to as the forged senders, sends to an address known to be screened for spam, and thus
has the spam assassin forward the message to his recipients. My provider doesn't seem worried
by this. Should I be worried, since my address might appear in the spam? That is, a spammer
sends me an email with your address as the sender. My provider sends a message to you (perhaps
including the original message) saying it is spam, and you see me as the original recipient and
suspect I might have something to do with the spam.) (If you do similar spam screening, then
one email might start a vicious circle and multiply until the systems are overloaded!) In other
words, is sending an email to the apparent sender a good idea (maybe the message was mistakenly
tagged etc)? (My provider should perhaps be concerned with all these informational emails and/or
with being tagged as a spam relay because of backscatter himself.)

The other problem would occur if, instead of sending information emails, the provider would send
some sort of error code during the SMTP dialog. This is OK for real spammers, but might be a problem
for people which forward email to me---they shouldn't get any error codes which indicate that they
might be sending spam. (For example, I moderate a newsgroup, and posts to the newsgroup go to another
site which emails them to the active moderators. If spam comes in, the relay site should not experience
any problems.)

brad

Mon, 2007-02-26 12:05

I don't do that is I want to keep the blog posting rate to just a few a week. Many people think a good blog has many postings a day, but I prefer lower volumes.

Chris Burkhardt

Mon, 2007-01-29 11:49

For now, perhaps, greylisting (tempfailing) does simply make the neighbors an easier target. But once enough mail servers are utilizing the technique (which I've personally found to be very effective), then spammers will begin to send every piece of spam twice in order to get through, which WILL slow them down. A little.

Also, in the case of the criminals and the bear, it is a case of EITHER me OR them. In spam that's not so; my neighbor will be spammed whether I'm protected or not. Though I guess your point is that the less bandwidth they spend on me, the more they have for everyone else, which is likely true. I don't like the air filter analogy; greylisting doesn't relay spam to its neighbors. A better illustration might be a house in the path of a mudslide; the owner of the house puts a barrier up which deflects the mud away from his house and into his neighbors'. Or something.

Thanks for the thoughtful post.

brad

Mon, 2007-01-29 12:11

Spamming engines run thousands of simultaneous threads sending mail, and are bound by their upstream bandwidth. Having a thread attempt to connect to your server and sit there waiting for it to time out slows the spam volume a negligible amount. So yes, you are releasing the pollution back into the atmosphere. Only by taking it and sequestering it do you remove it from the net.

I don't know how much spammers charge, whether they charge by the attempted address or the delivered address. If it's by the attempted address, then tempfailing is actually making the spam-service-supplier easy money, and costing the spamming client wasted money. Not sure this is good.

If it's by the delivered address, then yes, it is either you or them, because the spam campaign stops when it gets to the number of delivered addresses paid for. If I were a "smart" spammer, this is what I would pay for.

Alas, once tempfailing becomes very popular, it loses its effectiveness, as the spam software will start dealing with it. There are spam arms races worth fighting, but this one's path to failure is remarkably clear, even though it is effective right now.

And it comes at some cost -- it does delay your legitimate mail somewhat. For me, that's real. I run my own SMTP and e-mail is like IM for me. I realize a lot of the rest of the world has lost that and doesn't care.

Chris Burkhardt

Mon, 2007-01-29 12:39

[textile]

(I'm attempting to mark this up in Textile, but the preview doesn't look right. I'm hoping it's just because the Textile isn't processed until it is actually posted and not that I did it wrong...)

bq. If it's by the delivered address, then yes, it is either you or them, because the spam campaign stops when it gets to the number of delivered addresses paid for. If I were a "smart" spammer, this is what I would pay for.

Ah, yes. That's assuming the contractual agreement is of the "send exactly this many emails at $X per email" nature, in which case you are correct. I was assuming is was more of the "for $X I'll send your email to everyone on my list" nature, in which case it is not a me-or-them scenario. But at least for the dictionary attack method of spamming, your assumption seems more plausible.

bq. Alas, once tempfailing becomes very popular, it loses its effectiveness, as the spam software will start dealing with it.

Except, as I noted, spammers will have to send everything twice... at least the first time they send to a particular server.

bq. And it comes at some cost — it does delay your legitimate mail somewhat.

Yes it certaintly does. But today not greylisting has an even greater cost. I run an SMTP sever on a home server which provides email to myself and my family. The less processing and the less bandwidth I deal with the better. I agree with you in principle that the altruistic approach is best... but I'll let the guys with the resources be altruistic O:-)

Anyway, I just read your proposed solution at http://www.templetons.com/brad/spam/endspam.html and it all sounds good to me.

Anonymous

Mon, 2007-01-29 12:40

yeah... my Textile didn't work AND I replied from the wrong place which messed up the threading. Sorry.

- Chris

brad

Mon, 2007-01-29 12:47

I have not tried textile myself, it just comes with drupal, so I can't help debug but I think you need to have a registered account on the site or in the drupal net, and you may have to put tags around it.

I have not bought spamming services so I don't know how they contract, but I don't think it's good either way. We either make the spamming services richer or we deflect to neighbours. Neither is tenable long term.

Spammers don't send everything twice with tempfails, as I understand how most people do them. They attempt to send, and get a failure (either MX not responding to tcp open, or smtp server returning temporary fail.) They do not send the mail in these cases. Later, if they try again it does send the mail.

Alex Goldman

Mon, 2007-01-29 14:44

Reminds me of comments by Ayers and Nalebuff (you should talk to Nalebuff if you ever have the chance; I've never met Ayers) on car theft

http://islandia.law.yale.edu/ayres/forbesstopthief.PDF

They point out that car alarms and the Club simply cause a car thief to move on to the next victim but that LoJack is the reverse -- it benefits everyone even if only a small percentage use it.

Those who are really tracking down spam and fighting it benefit everyone (such as Blue Frog http://en.wikipedia.org/wiki/Blue_Frog).

You're right to be searching for answers; I think you'll find that people have them.

P.S. Have you ever seen a properly designed (http://www.templetons.com/brad/spam/challengeresponse.html) challenge response system? Just asking ;-)

Add new comment