There must be a truism somewhere out on the interwebs that goes something like this ...

if a computer geek finds himself or herself doing a task repeatedly, he or she will invariably find a way to automate this task

... and if there isn't a truism to this effect, then I've just written it for the first time.

In this particular case, the repetitive task was converting text written using John Gruber's Markdown syntax into HTML. Those of you who know Markdown will be asking the question "but Markdown is already a text-to-HTML conversion tool, why would you want to do this?". They'd be right too, so an explanation is due.

I'm not talking about vaguely human looking machines here, the sort that crop up in Forbidden Planet and Lost In Space, waving their metal arms and saying things like "Danger Will Robinson". What I'm talking about is a small file called robots.txt.

If you run your own web server you probably have one of these. It tells the web robots sent out by the search engines, such as Google, Yahoo! and Bing, what pages on your web site should and shouldn't be indexed and searchable. This doesn't mean that those pages can't be viewed, just that they shouldn't be able to be searched for.

Most of the time, a web site's robots.txt file contains stuff that is only of interest to the owner of the site and to people who specialise in getting the content of your web site to figure prominently in search engines. But sometimes, if you're willing to poke around a bit, they contain hidden gems, like a job advert for one of those aforementioned web search specialists, hidden in the UK Daily Mail's robots.txt file.