CensorNet Blog

Here at CensorNet things have been moving along so fast that it is difficult to know where to start. So I will begin by referring back to my previous article. In that item, I enthused about using a restricted policy to provide an almost open policy in the evenings, except for the ads. On some of my favourite sites.

With the changes we’ve made to the handling of dynamic sites and the new real time raters we’ve received from our business partner, I’ve changed this to be a fully filtered policy. It is still largely open, apart from annoying adverts, but makes good use of the real time raters when dynamic sites are involved.

Dynamic sites are those that carry information on lots of disparate subjects, so search engines and reference sites are particularly difficult to allocate to one single category.

Consequently, the CSRV categorises them as dynamic, which is a short hand for saying “analyse the web page when it’s requested and make a decision based on its contents”.

This means that the CensorNet will add additional categories to the list of categories the site is classified under as it goes, meaning that you, as an administrator can allow access to sites such as Wikipedia whilst blocking some of its more “sensational” pages.

You can even have this level of convenience when you are trying out the CensorNet. Read my other article on our new Global Network to find out how.

The main use of the CensorNet, of course, is to block access to sites containing material that is non-conducive to the environment being fostered.

Many System Administrators (SysAdmins), myself amongst them, like to have an open policy some of the time. This is partly so that we can determine if there is a problem with the proxy, we want to remove all the blocks, or, when we’re going through the lists of “Unblock Requests” that our users have sent us, we need to be able to surf to the requested site and see how we want to classify it. Should it be permitted to our users?

The trouble with an open policy is that it is just that, an open policy. As soon as it kicks in, you can surf anywhere, and there is no longer anything in place to block those annoying adverts.

Those of you who have used CNv4 1.1.0 already will have become aware that a restricted policy now includes the Content Classifier module, in addition to the Custom URL module.

The original reasoning behind this was so that you could configure a policy with the normal lists of known good URLs configured in the Custom URL module, but if you were actually running a class where access to sites of a certain type (say Telecoms) was necessary, you could also permit that type of site without having to know all the URLs you might otherwise have to issue unblock permissions for.

But I found myself reasoning this way. “I want a policy that lets me go anywhere I want, except I want a block on those adverts.”

I set out to do it and created a new policy.

Since this was supposed to be a largely permissive policy, I set the Conflict rule at “Prefer Allow rules over Block rules”.

I then configured the Custom URL Module to permit access to all the HTTPS sites I needed, grant access to MSN (more HTTPS sites), specifically block one site I hope never to see again, but left the remaining categories on ignore.

Finally, I configured the Content Classifier Module with Allow on all major categories. I then drilled down into the sub-categories of Internet and put a block on Internet Advertising, leaving the other sub-categories on allow.

Next, I configured a schedule on my workstation group to apply this new policy outside of office hours. The default (business) policy would be applied when my permissive policy wasn’t active.

I got on with other things until 17:30 ticked around.

The first thing to try was one of my favourite sites, www.theregister.co.uk. With the old configuration, outside office hours I could visit the site just fine, of course, but I also got the adverts. When I visited it during office hours, the adverts were nicely not displayed.

The first test, therefore, was to see what would be displayed now that I was on the permissive policy. The result was that indeed I could visit the site, and there were no adverts. Let’s hope the hacks at El Reg. don’t read this blog.

The second test was to put one of the categories on “Ignore” and see what happened.

As it happens, The Register is in two separate categories, and I only put one of those categories on ignore. This meant that after the requisite five minute wait, I was still able to reach that site; this due to the other category having an “allow” on it. Visiting another site, I knew to be only in the one category that I’d set to ignore, and I was blocked, the reason being that it was not in my permissive list of URLs as defined in the Custom URL module.

This was exactly the correct response. Since the Content Classifier wasn’t making a decision on that type of site, I having set it to ignore, the system could only fall back on what was in the Custom URL module, and the relevant site wasn’t.

One last test that occurred quite by accident; The Register has a sister domain, www.regdeveloper.co.uk. It turns out that this URL is currently unclassified. Fifty-eight million classified sites and I found one that wasn’t. The system took the correct action. Having no classification to fall back on, the CN blocked access to the site because it wasn’t in my list of permissive URLs. Nothing for it but to fall back on the time honoured procedure of adding said URL to the Custom URL filter personally, at least until our upstream classifies it later.

In-house, we are now using CensorNet Professional v4 1.1.0.x.

The new release features a major new feature, meaning that nearly every site you visit will have been classified, and have an entry in our database. This will almost completely irradicate false positives and improve the performance of the proxy server as only unknown sites will have to be rated in real-time.

Initially after installation, the CensorNet will send a request to our Category Server's (CSRVs) to ascertain which category the URL that is being visited falls into. This lookup request is transmitted over the Internet and operates in a similar way to DNS. Online lookups will only happen if you are evaluating CensorNet OR if you have only just installed or migrated from a previous version.

Overnight, however, providing you have a valid license (not a trial license), the CensorNet will download the classification database and store it locally. The download is approximately 2GB in size and should take anywhere from 2-6 hours to download providing you have a broadband connection.

How many URLs are classified?

There are over 58 million classified now, with more being added all the time. Your CensorNet will check for updates daily.

When can I get it?

We are currently using it in-house and beta testing it with a few clients. Providing the beta goes well, we then have to wait for one of our business partners to modify their infrastructure, after which we should be able to release it for general deployment; we think about the third week of February.

Is the upgrade process the same as normal?

No. There have been some major database changes in the new version. Consequently, you have to run a database migration tool first. This will convert all of your data, but note, you are given an option as to whether you want to convert your logging information or not, with the default being to skip this step. The reason is that logs are large, and in our tests, it can take five hours or more to convert the logs of a busy site. Skipping this process means you lose the old logs, but can migrate much more quickly.

But all my policies are kept?

Yes, all policies, users, groups, etc. will be kept. If you are going to run the new version on the same box as the old system, then you’ll keep all configuration settings too.

How can I control access to all of these new categories?

The Policy Management section has been totally re-written. Categories have major sections, with subsets. So if you wish, you can just choose to block all “Adult - Mature” material, including its sub-sections. If, however, you have a need to permit access to, for example, “Adult - Mature; Art Nudes”, so that an art class can take place, you may permit access to that whilst restricting the rest.

What about my manually entered overrides and filters?

As I mentioned earlier, these are all kept. We have, however, combined the old Overrides and Filters into a single URL module. When we convert your old data, the various categories are converted to the correct allow or block settings in your policies.

Any other interesting changes?

Restricted policies have more scope for configuration. These, you will remember, are the policies which, under earlier releases, only allow users to visit specific URLs that you have permitted. This is still the case if that’s all you want. However, we’ve added the Content Classification Module into these policies, so if you want to permit access to a certain type of site, that you may not have all the URLs for, you can.

Have you ever wondered what you’d do if your mail server fell over?

On Thursday we had a connectivity failure in the main office. Fortunately, that didn’t cause the loss of any mail. Our mail is hosted at a server located somewhere in the West of London. It just meant that some of my colleagues had to scramble for other means of access in order to obtain their mail. Fortunately, those of us who work outside the office were entirely unaffected.

But I repeat the question. What if your, and let me add our, mail server fell over? In our case we have yet another fallback.

When CensorNet MailSafe is first viewed, people look at it as something that protects one from Spam and Virus attacks, but it is much more than that.

Those of you who have used it are used to the fact that in the event of a false positive (I can count about three in the last nine months personally), you simply log into the portal, locate the trapped mail, release it, and then add a rule to ensure that this doesn’t happen again.

Recently, we added Business Continuity to our list of offerings. If we were to lose our mail server now, we would still be able to read, and reply to, mail that arrived, just so long as we had access to the Internet. We could do this through the same web portal that allows us to release falsely trapped mails.

The portal now gives us a rolling 28 day window on our mail, so this is plenty of time for us to repair our mail server, and in the meantime, business can continue.

Even better, you don’t have to wait for a failure to occur before you make use of the facility. Going on holiday? Well, you can take your laptop, your PDA or even a smart phone with you, but with anti-terrorism checks restricting what you can carry on your flights, it is just sometimes too much trouble to take them with you.

Now, you can just find that Internet Café in downtown wherever, log into the portal, and read your mail. Respond to anything you wish, or just think about what you want to say when you get back. None of this interaction stops you from reading the mail again in the mail client you use at work when you return.

As if that weren’t enough, mail sent through the Business Continuity service will also be archived if you’ve subscribed to our Mail Archiving service, just as if you’d sent the mail directly from your office PC.

If you’re a business, your fiduciary responsibilities probably don’t allow you to not take up the service. If you’re anyone else, just consider how well you could function if mail was denied you.