Fight Spam with an “intelligent” spam detection plug-in!

by Arif Mamdani

Spam, also known as unsolicited commercial bulk email, is a problem that we all have to deal with.  The last two years have seen a dramatic increase in the amount of junk-mail messages that clutter our inboxes.  Read on for tips on reducing the amount of spam you receive, as well as a guide for installing and using a new plug-in for Microsoft Outlook that’s delivering over 98% accuracy in Spam detection.

Tips for preventing spam:

One of the easiest things you can do to prevent spam is to make sure that your email address doesn’t appear on the web as a recognizable email address.  Spammers use programs that are similar to search engines to scour the web for pages with email addresses in them.  They look for text with the following format: username@domain.net and if they find text with that format, that address is added to their list of addresses that will receive spam.  To avoid this, make sure that email addresses are presented in a different format.  You may have seen this on the web already.  A common format looks like this: usernameATdomain.net or username AT domain.net.  In either case, the text is still a link to the email address, and when people click on the link, it still brings up a new email message addressed to the person in the link – the only difference is that spammer’s email harvesting programs don’t yet know how to deal with that minor formatting change, and that makes all the difference.

Avoid entering in your email address on websites that you don’t know and trust.  Most sites have a privacy policy on their website.  If you have any doubts about how your email address is going to be used, read the privacy policy and make sure that filling out that form for more information isn’t also going to result in your email address being sold.

Consider having two email accounts – one “public” and one “private.”  Use the public one for things that you suspect may lead to spam, and use the other one for email correspondence that you actually want to receive.  Admittedly, this is a rather drastic step, but if your volume of spam is so high that you’re having trouble separating the spam for the mail you actually need to read, it’s worth considering.

Install and use a spam filtering system.  Most modern email programs have basic junk-mail filtering built-in, and more and more Internet Service Providers are bundling this service with email hosting or offering it as an add-on.  If you host your email within your organization, talk to your system administrator about installing a spam detection and filtering system on your server to cover your whole organization.

Using the SpamBayes Plug-in for Outlook:

In the last few months, a number of spam fighting programs have sprung onto the scene using a variety of different techniques for identifying, classifying, and helping you deal with spam.  Of the different methods, products using Bayesian classification are emerging as the most successful.  A major reason for this is that unlike other systems, products using Bayesian classification “learn” from you as you use them, so they not only become more and more accurate over time, they also excel at learning what you consider to be spam.  For example: as a techie, I’m actually interested in some of the commercial email that I get.  I want to see some of the advertisements for inexpensive hardware and discounted software so that I can pass that information along to groups we work with.  I don’t however want to see advertisements for generic Viagra, and using a Bayesian system, I can train the system to differentiate between the commercial email I want to see, and the email that I don’t want to see.  Bayesian classification is pretty complicated, but in a nutshell, it uses statistical analysis to give each email message a score for its likelihood of being “good” email or being spam.  The scores are then compared, and if a message receives a higher spam score, it’s considered spam.  The programs that use Bayesian classification maintain frequency distributions of the types of words and phrases that appear in your “good” and “bad” email, and over time build these lists up to point where they’re very accurate at making sure that you don’t see the email that you don’t want to see.  For a more detailed description of Bayesian classification, see the SpamBayes website at: http://spambayes.sourceforge.net/

Speaking of SpamBayes, that’s the product that we’re going to walk through installing and setting up.  SpamBayes isn’t the only anti-spam system using Bayesian classification, but it’s one of the most popular.  It also integrates directly with Outlook, making it just a bit easier to use than other spam fighting systems.  If you’re not running Outlook 2000 or Outlook XP you might want to check out POPFile, a similar system that works will work with any email program you want to use.  Note: SpamBayes will not work with Outlook Express.

Installing SpamBayes:

The first step is download the program from: http://spambayes.sourceforge.net/windows.html

Clicking the link for the Outlook plug-in installer will start the download.

Make sure Outlook is closed, and double click on the file you downloaded to start the installation:

Start the installation by clicking next, and answering the questions that the installer asks you.  Generally, the default answers are fine.

When the installer is finished, it will ask you if you want to view the help file – make sure that the option to view the file is checked, and go ahead and let the installer finish.  Make sure to read the information file because unlike many other information files, the SpamBayes information is very useful, and does a great job at walking you through configuring and using the SpamBayes Outlook add-in.

Once you’ve followed the instructions in the help file (which also includes a great troubleshooting page), you’re all set.  The more spam you get, the faster the plug-in will learn what you consider spam.  One thing to note though is that spam characteristics change over time, so if you’ve got very old spam messages sitting in your mailbox, don’t use those to train the plug-in.  Using old spam to train the plug-in can actually result in a less accurate spam filtering system.

Finally, please be aware that this plug-in is being actively worked on, and while very usable, may still have some undiscovered bugs.  The SpamBayes project is an Open Source project that’s being worked on by volunteer programmers around the world.  They’ve got a system on their webpage http://spambayes.sourceforge.net for filing bug reports, and are generally quite responsive to fixing bugs as they get reported.  If you do encounter a problem considering filing a bug report and/or using the email link on the SpamBayes web page to get in touch with the developers.

Last updated: September 22, 2004