CensorNet Blog

If you ask many people why they install a filtering proxy they will first of all roll off the old adages of they want to block pornographic or other obscene web sites from being viewed and follow up with the need to control people.

The CensorNet can, of course, be used in such a negative way, but we prefer to see it in a more positive light.

I think we can all agree that web sites showing pornographic images or other material do get in the way of the working day, whether you be in an educational or office based environment.

But as an interview with Tim Lloyd, our CEO, shows we don’t intend the filter to be something that is just there to block the negative, to the contrary, it is there to enable people to be able to visit the sites they should be viewing in order to do their work whilst blocking those sites that might otherwise pose too thrilling a view to some colleagues.

If only the employers of the person described in this article had had such a positively discriminating filter in place, they would still have an employee, and more importantly, that employee would not have gone to jail, nor have been charged restitution fees.

As this article goes on to make clear, you should have a rock solid Acceptable Use Policy (AUP) in place as part of your security armoury. Make your statements of acceptable use therein and then use the CensorNet to enforce them. That way, not only you, but your users, will come to see it as a benefit and not something that they consider as something that gets in the way.

In the latest releases of CensorNet (1.4.20 and 1.5.7 at the time of writing)[1] we have introduced time based quotas.

In one sense these have always been available as you could have defined a specific policy with specific rules which you could then schedule into the day at a set time. However, this did not cover the fact where you were happy to allow your users to browse certain sites for a certain time, but did not want to specify exactly when that usage should occur within the working day.

Time based quotas resolve this by working in exactly that way. When you create a new policy, or modify an existing one, there is a Time Quota setting field immediately under the Block/Allow rules and Dynamic sites modifiers. The default value is 15 minutes, but you can select a time limit from a wide range of values available in the drop down selector.

Having specified the quota limit that applies for that policy, for any category set to “Allow” in either the Custom URL module or Content Classifier module, you can check the quota box.

URLs that fall within categories under quota control can then only be browsed for the time limit you set. When the quota is exceeded the user receives a block message explaining that the site is now blocked because the quota has been reached.

Each user has their own quota so the fact one user on a policy is suddenly being blocked due to quota exception, another may still be able to reach a site. The quotas for each user are reset at Midnight.

For a URL that falls into multiple categories, the quota will apply if any one category is quota controlled, even if the other categories are not.

[1] For the time being we are keeping the 1.4.x and 1.5.x versions of CensorNet synchronised. They are functionally identical, only the underlying operating system is different. If you are still running CensorNet 1.3.x we recommend that you upgrade to CensorNet 1.5.x by following the migration instructions. If you are running 1.4.x you will eventually have to upgrade to 1.5.x. We will announce when this is necessary.

Those of you who have been using our products since the Open Source version will remember that installation simply involved downloading an ISO which contained both the underlying operating system and the CensorNet code. You will be pleased to hear that we have reverted to this method of distribution with CensorNet 1.5.x.

The reason for bringing this out in such an easy to use package is because we were forced to change the underlying operating system we used in our VMA package (the ready to go VMWare instance you can install on top of your hypervisor). As a consequence we decided to provide CensorNet with the Ubuntu operating system in an easy to use ISO.

This means that users who wish to install CensorNet for the first time on bare metal can now just download the ISO from the nearest download mirror. Users who want to install on top of VMWare software can download a pre-built VMA. Alternatively, they can download the ISO and use it to install into a VMWare instance they have prepared to receive an operating system.

If you just want to try out this package and see if our product will work for you, you can license your download for 24 hours immediately. If, as most people do, you need longer than this to conduct a full test then you can request a trial license which typically runs for 14 days.

All of the traditional filters are present and of course there are all the hooks you need to use our optional modules (Malware Prevention, Phising Protection and On-Demand Anti-Virus) providing your license supports them.

All of this should mean that new users can get from download to functioning system even more quickly.

Last November spam levels dropped markedly when an ISP known as McColo was disconnected. They were disconnected when researchers discovered that a very high percentage of all spam generated by various botnets were commanded and controlled by IP addresses pinpointed with that provider. As soon as their upstreams pulled the plug the amount of spam flowing around the Internet plummeted.

Now the level of spam is rising again and has nearly reached the pre-McColo take down levels according to statistics issued by various anti-spam monitoring services. If you have not taken steps to protect your inbox yet, you may well like to consider protecting it now.

First, the good news is that you don’t have to throw out your existing email infrastructure. Your server, whether located in-house, or out there in a hosted environment, can remain where it is and do what it is good at, delivering mail to you, and sending your mails to whomever you’re sending them to.

The solution is an add-on service that requires no additional hardware, just a small amount of re-configuration so that emails, both inbound and outbound, go via an email filtering system that cuts out 99.997% of all spam and protects against email-borne viruses into the bargain.

Once you have made the small re-configuration to use the service to protect your inboxes you can add on additional services, such as a Business Continuity service which means that even if your mail server should fail, you can still read and write emails, and an email archiving service so that, should you lose an email, or need to prove that you did send an email months or years after you’ve deleted it locally, you can recall it and provide it in evidence.

Clearly, protecting oneself from spam as far as possible is useful, but the additional benefits available as an adjunct to this protection make this “Software as a Service (SAAS)” solution indispensable. The value increases when you discover you can take a 14 day no obligation free trial of the service to see if it really suits your needs.

If you have been keeping your eyes on the various IT related web sites that are available you have probably read varying statistics on the number of very popular web sites that have inadvertently become distributors of unpleasant payloads.

They have been duped into doing this due to bugs in the technologies they use to drive the sites, from SQL injection attacks to false re-direction by unpatched DNS servers. Whatever the reason for a popular site offering malicious payloads, it is becoming more important than ever to deploy tools to check what is being downloaded via the web because even if you have visited a perfectly kosher and well known web site, it may have fallen victim to an attack.

I mention this as an introduction to announcing the CensorNet 1.4.x branch which now runs on Debian Lenny. The upgrade process is a little more involved than a normal upgrade, but we shall be providing migration tools and documentation to make the process as painless as possible.

When you have upgraded, you will not only have a product that still provides all of the facilities CensorNet has always offered, but which has hooks allowing it to work with some additional software to help defeat some of this malware.

The first of these is the Panda Anti-virus product. The Panda engine not only scans on-demand pages and files for viruses but also includes a set of heuristics which enable it to hunt down spyware and other threatening software.

We are also able to offer a couple of extensions to the URL Database; the first of these works by adding an additional category to the content classifier which you can then configure in each policy. The feed contains millions of URLs with an indication of their location and whether they are deemed to be clean or malware.

The URLs are the full paths to many executables available for download. This module makes use of an automated system at the feed source which downloads and tests every Windows executable available on the Internet. As the tests are carried out, an entry is put in the feed, and this feed is updated by your CensorNet during the usual daily URL database update process.

The second available extension is the Phishing Feed which was contains the URLs reported by the Anti-phising Working Group. With this extension installed, the CensorNet will be able to protect you against sites which may attempt to elicit your users’ personal details.

Full details of all these add-on products are available from our Sales Team.

Its been a while since our last posting, but you shouldn’t think all development has stopped; far from it.

In response to many requests, the CensorNet now supports In-line (transparent) mode.

This means that you can set all of your workstations to point to CensorNet as their gateway and you don’t need to configure your browsers with the CensorNet’s proxy details. However, this does enforce certain rules.

The first is that SSL Intercept Mode is forced on and cannot be disabled. Consequently you will need to install the CensorNet’s certificate in all of your browsers. You can read our instructions in the SSL Certificate Installation Guide which will explain how to install the CensorNet’s certificate on all of your browsers.

Transparent mode also means that no kind of user authentication can be used and so only Workstation Policies are possible, however, it is possible to have a mixed arrangement whereby some workstations use the traditional “sideways proxy” mode, with authentication, and others use transparent mode.

Other developments that have come along in the pipeline are a new version of the image filter. You can now adjust the sensitivity, there is a new entry in the Filters menu. We default to a setting of 70%. The higher the sensitivity the more “picky” the filter will be. This is a global setting and applies to all policies where the image filter is active.

In addition, there are some modifications under the hood, which you won’t see from the menus. Various streaming protocols are now faster. If you allow your users to visit YouTube, they should start to receive the stream when it starts, rather than waiting for the policy engine to parse the entire content before delivering it. This should cut down on complaints from your users.

It must be twenty years since I last had anything to do with magic numbers, and I’m not talking about the magic squares they used to teach in primary school maths.

Magic numbers are a few bytes at the beginning (or just a few bytes in from the beginning) of a file that indicate just what sort of data the file contains.

Many files carry this and it’s very useful, especially in these days of trying to block Trojan horses and other kinds of malware.

If you see a file with a .txt extension, you are so ingrained into believing this is a text file that you may well open it, only to discover that it is a Microsoft Word file with an embedded Trojan.

If you’d have checked the magic number, you still wouldn’t know about the embedded Trojan, but you’d have known it was a Microsoft Office document. On a Unix/Linux system, typing a command of the form :-

file somefile.txt

would have shown you that far from being a text file, it was indeed a Microsoft Office document.

I was re-introduced to them just the other day when in a support query, I was forced to ask our developers how our file upload module worked. They explained all about magic numbers and I was suddenly taken back those twenty years to when I had initially learned them.

What this means is that if I have set my policy to block the upload of pdf files via the web, then simply changing the filename’s extension won’t suddenly allow the file out of the building as the magic contained in the file will still give away its real type. Yep, looks like a pdf, is a pdf, whatever the extension on the file name. If I really want to get that information out of the building, I’m going to have to cut and paste the text into a real text document or sneak it out via some other means. If the pdf contains anything more than text, I’m sunk.

We are very pleased to announce that CensorNet Professional has won West Coast Labs Premier Checkmark award. The judges tested the CensorNet’s ability to block pornographic and other unpleasant content and ran it against a list of URLs garnered from various honeypots and as such this list contained the latest threat web sites. We are pleased to say that the CensorNet was able to block every URL that was thrown at it.

You can read about the CensorNet in October’s issue of SC Magazine.

As I promised you in this document we have now released CNv4 (1.2.9) and it is available for download now. Not to be halted, however, our team are now busy on CNv4 (1.3.0) which will include SSL filtering.

As you may recall from this article one of the caveats in the use of blocking attachments on Webmail and similar sites was that we could not block attachments in HTTPS communications because the content was already encrypted and the CensorNet could not determine the file type.

The SSL filtering module works by presenting the browser with a certificate of its own. If the browser accepts it, then the CensorNet has access to the raw data. Now it can check for included attachments in the data stream and apply the blocks you have configured in your policy definitions. If the policy is such that it would permit the connection to continue, the CensorNet then re-encrypts the data in its onward journey to the target site.

All of this can put an excessive load on a server, and so the functionality can be turned on or off. This means that the previously mentioned caveats will still apply if you disable the module.

The reason that we had to come up with this module is because Google now provide a setting on their Gmail/Googlemail platforms to enable end to end HTTPS use. Whereas it used to drop to HTTP after authentication, this is no longer the case if this new setting is configured. It is likely that other Webmail providers will offer similar functionality soon.

In the same way that policies are the core of the CensorNet web filter, rules are the core of CensorNet MailSafe. Whilst the product filters out around 98+% of spam without you having to perform any configuration, there are always exceptions, and your mail policy may require the configuration of certain rules.

For example, do you want to be alerted when a user sends out an attachment in the mail? Do you want to receive a copy of that mail? Do you want to quarantine incoming attachments until you can check that they are business related for yourself? Perhaps you want to block all mail from hotmail/gmail/yahoo addresses except for people you know you have valid business dealings with. You can do all of this via rules.

It has always been possible to configure rules whereby you could block, or allow, mail that contained certain words or phrases in either the subject or the body of the message. This facility has been expanded so that now you can define a dictionary.

A dictionary is either a word list (words or phrases with a score) or a regular expression (I’ll talk about those another time), also with a score.

Having defined your dictionary, you can configure a rule so that it is triggered either if there is any match within the dictionary, or, is only triggered if the number of words and phrases in the mail hit a certain trigger level.

There are a number of pre-defined dictionaries already available. These are

- Credit Card Data
- Credit Card Number
- GLBA
- HIPAA
- Personal Identifiers

You may or may not recognise the couple of acronyms in there. The HIPAA is the Health Insurance Portability and Accountability Act and the GLBA is the Gramm-Leach-Bliley Act. Both American enactments but they relate to Health Insurance and Commercial Investment Bank requirements respectively and so you find spammers using the phrases these people put inside their emails in their scam mails to you.

The thing to do, then, is see if there are enough phrases in the mail to determine whether its genuine or not. As explained before, each word or phrase in a dictionary definition carries a score. The scores get added together to form a total.

Now, you can define a rule which only lets a mail through if the score is greater than a certain value. Alternatively, you can allow it through if there is but a single match. Of course, where I’ve used the term allow, you can opt for the “mark it as spam” option. The choice really is yours.

These pre-defined dictionaries exist because of the large corporations that use the types of phrases contained therein, but you can always define your own dictionary to suit the line of business you are in.

We ourselves have various phrases we use all the time, so we could easily make up a dictionary with suitable scoring and use that in a rule, and there is no reason why you could not do the same.

All in all, the use of rules will help you apply your corporate mail policies as well as keeping the spammers at bay.

It seems everyone is running trains that are running on time at the moment. Yes, even on the oft laughed at British rail network. I had two journeys (with two changes each) where all trains ran exactly on time. This is almost unheard of.

In the world of CensorNet, our DevTeam are excelling themselves and new betas are coming out almost faster than our Support Team can learn the new features.

So I thought I’d take the time to write about the new features that will be available in v4 (1.2.9), and hopefully get them in my head as well.

The CensorNet has always supported complex AD structures, but importing users from an AD tree with multiple OUs into multiple groups on the CN was something of a long winded exercise.

Either you had to import each OU separately into the relevant group, or you imported all the users into one group and then used the CensorNet’s User Management system to migrate users to alternate groups.

Both methods worked, but were cumbersome. So an enhancement to the Objects->Import sub-menu now allows quick and easy import into multiple groups.

Simply run the new import wizard and the CN will not only create CensorNet groups with names which match your OUs but will also import the relevant users into them. It really is that easy.

Another addition is the Filter Bypass. Long time users of the CensorNet, by which I mean those who remember CNv3 systems, will remember the “Authentication Exceptions” option that was available.

The Filter Bypass is the same sort of animal, but in a much easier to use package.

Anti-virus software frequently fails when run over proxies, and there are some sites which object to the NTLM authentication method that many of you use in order to avoid seeing a sign-in prompt from the CN.

The Filter Bypass module deals with both.

To configure it, you can create as many categories as you like, and put as many URLs as you like in each category. So you’ll know how to use it, as it works in just the same way as the Custom URL filter in that respect.

However, you need to remember that anything you put in here will totally bypass all filtering (hence its name) as well as bypassing all authentication. So use it with care. Remember, these settings are global.

I’ve added avg.com to my AVG category and microsoft.com to my Microsoft category. The latter allows me to use Clipart in Word documents now, which I couldn’t do previously (if configured to use the CN).

I’m awaiting the next AVG update to see if it solves its problems with the proxy.

In addition to all this, we recently added Advisory policies. These work in just the same way as Filtered policies except that instead of a “Request Unblock” button, users on such a policy instead receive an “Override” button.

These are not, therefore, intended for general users or pupils, but for senior staff.

Once a block has been overridden, that user can visit the URL until they finish a session. The session times out after 30 minutes. Visits made under the override will be logged as visits to blocked sites, and so it will become obvious if someone is abusing the privilege.

By the time this posting is published, it will probably already be old, as the release train seems to have been supplied without brakes of any description. 

CensorNet has allowed you to block inappropriate content from being viewed at, or downloaded to, your network. Now CensorNet is going even further.

Given that the CensorNet blocks only one attack vector by which inappropriate material can enter your network, there are still ways that users can use your network to then upload unpleasant files to the rest of the world.

Until now, the CensorNet has not addressed that issue.

With version 1.2.4 or later, we have introduced an “Upload module”. To use it, you will need to review your policies, and configure the “Upload Module” to control what your users may and may not upload, via means of a web form, from your site.

For example, the most pernicious file types are executables and archives. You can now set the CensorNet to block these types of file from upload via a site offering a web form. In most occasions, users will receive the traditional CensorNet block notice advising them that the Upload Module has blocked their post, and why.

But there are other good reasons for this module, other than mere perniciousness.

As you’ll be well aware, we all have Office Documents, PDF files, and other file types on our corporate networks. All of these contain information valuable not only to us, but also to our competitors.

The files arrived on our network via perfectly valid means. The connection to the web that we are largely forced to provide, just to stay in business, is a way that our users can then send company confidential information to our competitors, via means of web mail sites, and other web based forms.

So our module also addresses this, and allows you to be able to prove that you are meeting your fiduciary duties under the HIPAA act (for example) in using your IT infrastructure to protect your corporate secrets from leaking.

There are, however, some cases where the upload module cannot help you.

If a site is using HTTPS, then the encrypted transmissions between the user’s client and the site means the CensorNet can’t tell what is being sent, and can’t, therefore, choose to block it. However, since all HTTPS traffic is blocked by default, this should not present an overwhelming problem. It is, however, a good idea to revisit your policies if, in the past, you have chosen to simply allow all HTTPS transit.

The other case is not one where the block won’t work, but one where the CensorNet will not inform the user that their upload has been blocked.

Sites using AJAX mean that the error return is sent to the site. To the user, it will appear that their upload has simply frozen. So you, as an administrator, may have to fend off the odd call from a customer asking why they couldn’t upload such and such a file. As you’ll know what file types you’ve blocked, you should be able to advise them accordingly. 

If you are a school or company, then there is really no need for your users to anonymize themselves. Unfortunately, rogue anonymizers are cropping up around the ‘Net with alarming speed.

We have been working with our business partners for some time to overcome this, because the security measures we already had in place (blocking HTTPS and numeric IP addresses) on their own are simply not enough.

So in the latest release, we have included a new rater which checks for sites offering anonymization. All you need to ensure is that the Anonimizers/Proxies category is set to block in your policies.