SPAM: Blacklists & Whitelists
Spam. Anyone connected to the Internet has heard the word. Anyone who has an e-mail account dreads the word. It has been estimated recently that one third of all e-mails transferred from computer to computer are unsolicitated: i.e, spam. But rather than talking about the generalities of spam this article attempts to inform you of two methods used by companies to combat spam and how they may affect you. These two methods are called Blacklists and Whitelists.
As the name suggests Blacklists are lists which exclude something. In the case of spam Blacklists exclude IP addresses which are associated with spam and block all e-mails coming from those IP addresses. But how does a Blacklist affect you? Before you can determine how this affects your web site you must understand IP addresses and how they work with web hosting.
If you have shared hosting on a server, chances are you are also sharing an IP address. An IP address is the numerical representation of where you server is. While humans use alphabetical names to remember a web site; computers and servers don't know where www.navicosoft.com is but rather know that it is located at 184.108.40.206. The numbers 220.127.116.11 is the IP address of the web site www.Navicosoft.com. Whenever a computer needs to contact www.Navicosoft.com or send mail to www.Navicosoft.com it knows that the IP address is associated to that domain. Now, www.Navicosoft.com is not the only domain (or hosting account) using this IP address. With the aid of server software a single IP address can be reused by more than one domain at a time. This is called virtual hosting. There are many reasons why several domains may be using the same IP address but realize that tens or possibly hundreds of domains may be sharing the same IP address on a server. Now let's examine why this is important when talking about Blacklists.
When a spammer sends out spam, either through a hosting account, or through exploiting a security hole in a server the e-mails are sent from a hosting account which has an IP associated to it. When other computers, and users, find out that they have received spam and they trace the unsolicited e-mail back to the domain which sent the spam. What happens next is that this IP address, which is associated with the spam, is now listed as sending spam and is listed on a Blacklist. What happens next depends on the people running the Blacklists the IP was just listed on.
Blacklists can be characterized as three different types: the temporary, the permanent and the obscene. Before the differences are discussed let's talk about what a Blacklist does: a Blacklist is nothing more than a list of IPs in which all e-mail originating from them is blocked. Any e-mail coming from a Blacklisted IP address is returned to the sender without ever reaching the intended recipient. Now the computers which are using the Blacklists do not determine what is spam and what is not spam and only block the spam--they block all e-mail whether it is spam or not.
The differences between the different types of Blacklists is a function of how the people running the Blacklists deal with the spam. The individuals running the temporary Blacklists monitor for spam and when they find spam coming from an IP address they distribute this IP address to their followers and all e-mail from that IP address is blocked for a period of time. Usually this period of time is several hours. The reasoning for this is that most spammers use an IP address once and then move on after sending millions of e-mails to the next IP address. The temporary Blacklists essentially have a rolling target of IP addresses they block as the spammers move from IP to IP.
The permanent Blacklists don't bother with the rolling aspects of trying to catch spammers as they move from IP to IP--they just keep adding IPs as they go with the thought process that the spammer may use the IP in the future, either by exploiting the same hole or reusing the account again. In the case of permanent Blacklists they block all e-mail from the IP permanently.
Now, the third group of Blacklists is like the other two in that when they receive spam from an IP address they block all e-mail from that IP however. However, they differ in that rather than waiting for spam to appear from a new IP address and then adding it to the Blacklist they take the proactive stance and say that if 18.104.22.168 sent me spam the other IP addresses next to it "might" send me spam so they will be blocked as well. So an obscene Blacklist might block hundreds of IP addresses next to the spamming IP even though: they might not be sending spam; they might not be on the same server; and might not even belong to the same person or company. Their attitude is that they would rather lose valid e-mail at the expense of receiving any spam at all because the IP next to it "could" do something.
So how does this affect you? Remember your domain (hosting account) uses an IP address. If this IP address is shared by others you might find your e-mail being refused because someone else sent spam on the same IP address you are sharing or because a spammer exploited a security hole and sent spam on the IP address. If you find your IP address is blocked or Blacklisted then your e-mail may be refused for a few hours or permanently. Or you might find that you might have not done anything at all and just happened to have an IP next to someone who did spam.
So how do you know if are on a Blacklist? There are two ways to determine this. First, if you receive an e-mail bounced back after it was refused you can look at the headers of the e-mail. The headers of an e-mail are all the stuff before the actual message of the e-mail. Most of the headers will be numbers and letters but there is a section which can tell you the reason why your e-mail was not delivered. If you see in there that the e-mail was refused or blocked then this is a flag that something is wrong. This does not absolutely mean you are on a Blacklist; it could also mean that you are not on the Whitelist (discussed in a bit). At this point you need to look into the matter further and use the second method
The second method to determine if you are on a Blacklist is to contact the people running the Blacklists and find out. If your IP is listed they will tell you. This is simple enough to do in theory but contacting all of these Blacklists can be time consuming. Fortunately, there is a faster method. There is a web site, a very useful one at that, which allows you to enter in an IP address and search the current Blacklists for that IP address. If the IP is listed on a Blacklist it will tell you. Now if you don't know your IP this site can still search because this site will translate your domain name into an IP address and then search for it. This wonder site is www.dnsstuff.com. To search for Blacklists you simply go to www.dnsstuff.com and at the top of the page there is a box titled "Spam Database Lookup." Just enter you IP address or your domain name into the box and press "Lookup" and it will query the major Blacklists and provide you with the results. If your IP has sent spam you will be able to find out who is blacklisting you.
Now before you even press the button you should realize that on this list there are several obscene Blacklists here. There is a very good chance that your IP is listed with them. Let me give you a concrete example. The IP we have been using in this article, 22.214.171.124, is listed on two of these Blacklists. Even before it was used by www.Navicosoft.com it was listed on these two Blacklists. In fact even before the IP was owned by the management of Navicosoft it was listed. In fact, six months before it was owned by Navicosoft it was blacklisted. What happened is that the person running the Blacklist determined that the IP, 126.96.36.199, was owned by someone who, in their mind, was soft on spammers. Therefore rather than taking the chance of receiving spam from this IP address they blacklisted the IP and keep it on the Blacklist. So even though the IP is no longer used by the person who was soft on spammers, and is now used by a completely different company they are still blacklisting the IP. Why? Because they run the Blacklist and can do anything they want with it. Now are e-mails being refused by the hundreds? By the thousands? No. In fact since taking ownership of this IP not a single e-mail has been refused. Why? Because very few, if very many at all, are using this obscene Blacklist. Realistically, why would you want to use a Blacklist which never removed IP addresses from the Blacklist even after they have new ownership? Why would you Blacklist an IP which never spammed? It is partly ridiculous to do so and most companies and individuals realize this. These are not the Blacklists you need to worry about. You need to worry about the most used ones: SpamCop, SpamHaus, etc. If you are listed on one of these chances are your e-mails will be coming back to you.
Now if you do find your IP address listed on one or several big Blacklists you need to do something about it now. You should contact your web host immediately or your server provider. If you do nothing chances are you will stay on some of these Blacklists. To get off a Blacklist you have do very specific things to satisfy the owners of the Blacklists. And if you have mail being refused on a daily basis you need to do something about it now rather than later.
There has been a lot of time spent talking about Blacklists, but what about Whitelists? A Whitelist is the opposite of a Blacklist--in order to send any e-mail to someone using a Whitelist you have to be approved and verified. You are approved and verified by by listed in the Whitelist. A Whitelist is a list of IP addresses which are trusted implicitly. That means all e-mail will be accepted from them if the sender of the e-mail is on the Whitelist. Anyone sending e-mail to a server using a Whitelist must be on the list or it will be sent back.
Why would someone use a Whitelist? For a lot of reasons. Consider a company with two locations, one on each side of the world. They want to block spam but want to send e-mail back and forth. They each set up Whitelists with each other and they now can send e-mail back and forth without ever having any spam enter anyone's mailbox because the only two IP addresses on their Whitelist is each other. Now if they only wanted to send e-mail to each other they could stop there. But if they want to receive e-mail from anyone else they need to add the new people to their Whitelist. And this is what they do; when they find a person or company they want to receive e-mail from they add them to the Whitelist. The only downside is that Whitelists require that they be set up before they are used or people won't get their e-mail. Whitelists are gaining a lot in popularity so you will see them gaining popularity in the coming months. In fact a major player in the dial-up ISP market is promoting the fact that they are using Whitelists. Basically, before any e-mail is received by anyone using the service the person sending the e-mail must identify themselves in the e-mail and then be approved by the recipient. If the recipient doesn't approve the e-mail it is sent back and never seen by the recipient.
Blacklists and Whitelists are here to stay. The abundance of spam has precipitated their use. Chances are you personally will not be involved with maintaining either a Blacklist or a Whitelist in your lifetime. But, there is a good chance in your lifetime you will run across either Blacklist or a Whitelist in your hosting account. Also, before you consider you next host you should consider whether the IP address you will be placed on is Blacklisted.