I'm not talking about vaguely human looking machines here, the sort that crop up in Forbidden Planet and Lost In Space, waving their metal arms and saying things like "Danger Will Robinson". What I'm talking about is a small file called robots.txt.

If you run your own web server you probably have one of these. It tells the web robots sent out by the search engines, such as Google, Yahoo! and Bing, what pages on your web site should and shouldn't be indexed and searchable. This doesn't mean that those pages can't be viewed, just that they shouldn't be able to be searched for.

Most of the time, a web site's robots.txt file contains stuff that is only of interest to the owner of the site and to people who specialise in getting the content of your web site to figure prominently in search engines. But sometimes, if you're willing to poke around a bit, they contain hidden gems, like a job advert for one of those aforementioned web search specialists, hidden in the UK Daily Mail's robots.txt file.

I've never run or hosted my own search engine. I've run and hosted web servers, mail servers, proxy servers and caching servers (I'm even contemplating running my own URL shortener), but never a search engine. There was a time when I ran an enterprise instance of Alta Vista back when I coded for a living and was part of the team building Factiva.com, but that doesn't count.

If I had have run my own search engine I would have known just how important canonical URLs are and that having multiple copies of the same content hosted on different domains would cause search engines to penalise you and loose search engine ranking, fast.