If you run your own web server you probably have one of these. It tells the web robots sent out by the search engines, such as Google, Yahoo! and Bing, what pages on your web site should and shouldn’t be indexed and searchable. This doesn’t mean that those pages can’t be viewed, just that they shouldn’t be able to be searched for.

Most of the time, a web site’s robots.txt file contains stuff that is only of interest to the owner of the site and to people who specialise in getting the content of your web site to figure prominently in search engines. But sometimes, if you’re willing to poke around a bit, they contain hidden gems, like a job advert for one of those aforementioned web search specialists, hidden in the UK Daily Mail’s robots.txt file.

User-agent: *
Disallow: /tvshowbiz/tvlistings/
Disallow: /home/ireland/
Disallow: /home/scotland/

# August 12th, MailOnline are looking for a talented
# SEO Manager so if you found this then you're the kind
# of techie we need!
# Send your CV to holly dot ward at
# mailonline dot co dot uk

Sometimes the lines between a metal robot and the robots.txt file gets blurred; yelp.com‘s robots.txt file starts with the famous Three Laws Of Robotics that Isaac Asimov wrote about back in 1942.

#
# 1. A robot may not injure a human being or, through
#     inaction, allow a human being to come to harm.
#
# 2. A robot must obey orders given it by human beings
#     except where such orders would conflict with the
#     First Law.
#
# 3. A robot must protect its own existence as long as
#     such protection does not conflict with the First or
#     Second Law.

User-agent: *
Disallow: /advertise?
Disallow: /biz_share?

Social music site last.fm has taken this one step further and represents Asimov’s laws as faked URLs on their web site.

User-Agent: *
Disallow: /music?
Disallow: /harming/humans
Disallow: /ignoring/human/orders
Disallow: /harm/to/self

And finally, in a particularly creative use of robots.txt, Scottish whisky brand Whyte & Mackay have hidden a giveaway promotion in their file.

An SEO nod of the hat must go to Tyler Bell for spotting the robots.txt file on yelp.com and Malcolm Coles for the Daily Mail and Whyte & Mackay robots.txt gems.

If I had have run my own search engine I would have known just how important canonical URLs are and that having multiple copies of the same content hosted on different domains would cause search engines to penalise you and loose search engine ranking, fast.

But I’ve never run my own search engine. So I didn’t know any of this. I probably should have, but I didn’t. Mea culpa.

So what has any of this to do with Posterous? I use Posterous. I like Posterous, a lot. I’ve written about Posterous, quite a bit. I also use Posterous to not only post to my Posterous blog but also to my own WordPress powered blog, on a domain I’ve owned for a goodly number of years, via Posterous’s autopost function … and which nicely and neatly produces an exemplar of how to have duplicate content hosted on multiple domains, with multiple URL addressing systems, for each and every post I produce.

How could I have not noticed this? Other people have, including Ian Delaney‘s excellent write up, punnily entitled Past Posterous.

Sadly, it looks like despite the ease of blogging that Posterous offers, there is such a thing as too easy and so for now, with regret, I’ve postponed my use of Posterous, possibly in permanence. Unless of course, they offer a way of specifying canonical URLs.

And with profuse apologies for the overuse of alliteration in this post.

Photo Credit: I Bought a Mac on Flickr.

Written at home (51.427051, -0.333344) and posted from the Yahoo! London office (51.5141985, -0.1292006)

• Got a colleague who keeps wandering away from their desk and leaving their mobile phone behind, which then keeps on ringing? Maybe they need one of these signs left on their desk. Maybe.
• Fancy a challenge? How many times a day do you type the incorrect command at the shell? Once, twice, three times a day? More? Maybe you should give Suicide Linux a try; it helpfully turns any mistyped command into rm -rf / thus helpfully erasing your root file system. Concentrate now.
• The WordPress Flickr Manager is a wonderful plugin which integrates your Flickr photostream into blog posts. Alas it doesn’t work with WordPress 2.9. Until now.
• Posting the same article to multiple blogs severely impacts your search engine ranking results. How did I not know this? It’s stopped at least one person from using the Posterous autopost function.
• Sometimes, just sometimes, sub-editors trim just a little bit too much from an article prior to publishing.
• We’re used to online slippy maps being able to zoom in and out; but zooming in and out of paper maps? That’s something else indeed.
• What’s happens in Vegas stays in Vegas; but sometimes it stays on FourSquare as well.
• Photo of the year so far; the Space Shuttle Endeavour, caught in silhouette from the International Space Station. That phrase alone sounds like it’s been lifted wholesale from an Arthur. C. Clarke novel.
• ˙uʍop ǝpısdn ǝdʎʇ oʇ pǝǝu noʎ ‘sǝɯıʇǝɯos ʇsnɾ ‘sǝɯıʇǝɯos