Gary's Bloggage » data

Service Suspended On The London Underground (API)

Gary — Fri, 02 Jul 2010 14:44:43 +0000

If you build it they will come. Or to put it another way, sometimes demand outstrips supply. After the phenomenal success of the Transport For London Tube API, the London Datastore blog sadly notes:

Owing to overwhelming demand by apps that use the service, the London Underground feed has had to be temporarily suspended. We hope to restore the service as soon as possible but this may take some days. We will keep everyone informed of progress towards a resolution.

In the meantime, if you want to see how it does looks when the API is up and running there’s a video clip of Matthew Somerville’s recent Science Day hack visualisation over on my Flickr photo and video stream.

Photo Credits: Martin Deutch on Flickr.

Written and posted from Berlin Tegel Airport (52.5545447, 13.2899969)

Where’s My Tube Train? Ah, There’s My Tube Train

Gary — Thu, 24 Jun 2010 19:52:09 +0000

Back in December of 2009, I wrote about Paul Clarke trying to solve the problem of where’s my train; that there must be a definitive, raw source of real-time (train) information and that

I assert that train operators know where their assets are; it would be irresponsible if they didn’t

Whilst the plethora of train operators that fragmented from the ashes of the old British Rail network haven’t answered this challenge yet, Transport for London has, opening up just such data as part of the London Datastore API. In today’s age of talented web mashup developers, if you release an API people will build things with it if the information is useful and interesting and that’s just what Matthew Somerville of MySociety did at the recent Science Hack Day … a (near) realtime map of the London Underground showing the movement of trains of all of the Tube lines. A screen grab wouldn’t do it justice and it takes a while to load, so a video grab might help here.

Coming down the escalators at Waterloo and want to know whether to head for the Bakerloo or the Northern Line to take you north of the river? Now you can tell which line has a northbound train closest to Waterloo.

Want to see just how close the gap is between Leicester Square and Covent Garden on the Piccadilly Line really is? Now you can.

Of course, this doesn’t solve every problem …

If you’re on the escalators at Waterloo how do you get 3G data coverage to view this mashup on your phone as Transport for London still haven’t manage to achieve cellular coverage underground, unlike Amsterdam, Berlin and other cities?
The site will probably be the target of a tutting campaign from the Health and Safely police insisting that such a visualisation will cause people to run for the train and of course, they might trip and hurt themselves.
If you’re at the top of the escalator and the train is in the station, now, right this very minute now, how do you get down to the platforms quickly?

Whilst I can’t answer the first two of these questions, this publicity stunt from Volkswagon at Berlin’s Alexanderplatz U-Bahn station might just hold the solution for the third question … a slide!

Written and posted from the Ramada Hotel Berlin Mitte in Berlin (52.529858, 13.383858)

When Maps and Data Collide They Produce … Art?

Gary — Tue, 01 Jun 2010 18:45:24 +0000

Last month I wrote that a map says as much about the fears, hopes, dreams and prejudices of its target audience as it does about the relationship of places on the surface of the Earth. With the benefit of hindsight I think I was only half way right.

Sometimes a map becomes more than just a spatial representation and becomes something else.

Sometimes a data visualisation becomes more than just the underlying data and almost takes on a life of its own.

When these two things meet or collide the results can be spectacularly compelling and produce, unintentionally … art? Look at the image below … filigree lace work? Crochet for the deranged of mind? Silk for the sociopath? Macrame for the mad? Sadly none of the above.

It’s instead an image from the Geotagger’s World Atlas but it’s still unintentionally beautiful.

The maps are ordered by the number of pictures taken in the central cluster of each one. This is a little unfair to aggressively polycentric cities like Tokyo and Los Angeles, which probably get lower placement than they really deserve because there are gaps where no one took any pictures. The central cluster of each map is not necessarily in the center of each image, because the image bounds are chosen to include as many geotagged locations as possible near the central cluster. All the maps are to the same scale, chosen to be just large enough for the central New York cluster to fit. The photo locations come from the public Flickr and Picasa search APIs.

I could look and stare at the all the images in Eric’s Flickr set for hours. Correction, I have stared at the images for hours.

Photo Credits: Eric Fischer on Flickr.

Written and posted from home (51.427051, -0.333344)

Reaching The Limits Of Unlimited

Gary — Thu, 06 May 2010 08:34:53 +0000

Consider for a moment the word unlimited; it’s an adjective and, if you’ll pardon the condescension, it means the following:

not limited; unrestricted; unconfined
boundless; infinite; vast
without any qualification or exception; unconditional

Except in the world of mobile data or mobile broadband, where unlimited means, in a vaguely disturbing twisted, inverted, doublespeak sort of way, the exact opposite.

Vodafone, my current UK mobile provider, helpful tells me that I have unlimited data, subject to their fair use policy which promptly redefines unlimited as very much limited indeed and your limit is 5GB per month. That’s a lot of data. Even being the compulsive photo uploader, web browser, Foursquare and Gowalla check in, Twitter and Facebook poster and checker that I am, I’m hard pressed to go above 500MB per month let alone 5GB.

So I was both vastly amused and somewhat shocked when this text arrived on my iPhone on the way home from work last night.

A quick call to Vodafone luckily cleared this up as being a glitch in their billing systems and I would not, as stated be charged, nor had I gotten anywhere near the 5GB limit of unlimited.

I found the whole process rather amusing in hindsight but shouldn’t the mobile companies either come clean about what unlimited really means or just don’t sell unlimited data as a concept at all and just sell a, 5GB in my case, data limit?

Written and posted from the Yahoo! London office (51.5141985, -0.1292006)

Reclaim and Own Your Short URLs

Gary — Wed, 03 Mar 2010 11:05:27 +0000

There are many reasons to like the use of URL shorteners such as bit.ly and tinyurl.com. These free services take a long URL such as this post – http://www.vicchi.org/2010/03/03/reclaim-and-own-your-short-urls – and compresses them down to a much more manageable shorterned version – http://bit.ly/aG1RBx or http://tinyurl.com/ylaodny.

They increase link sharing; the vast majority of social networking sites use 140 characters as the maximum size for an update, using the full version of a URL you’re sharing reduces the amount of space for you to put your own thoughts into the update. Just compare the full URL http://www.vicchi.org/2010/03/03/reclaim-and-own-your-short-urls at 65 characters against http://bit.ly/aG1RBx at 21 characters.

They can track and yield click and referrer information; the information that bit.ly provides is so useful, showing live clicks, geographic and referrer information amongst others.

But almost a year ago, Delicious founder and ex-Yahoo! Joshua Schachter made some pretty compelling arguments against short URLs:

The worst problem is that shortening services add another layer of indirection to an already creaky system. A regular hyperlink implicates a browser, its DNS resolver, the publisher’s DNS server, and the publisher’s website. With a shortening service, you’re adding something that acts like a third DNS resolver.

…

But the biggest burden falls on the clicker, the person who follows the links. The extra layer of indirection slows down browsing with additional DNS lookups and server hits. A new and potentially unreliable middleman now sits between the link and its destination. And the long-term archivability of the hyperlink now depends on the health of a third party.

Or to put it another way, you no longer own your links or the data clicks that those links yield. If the service dies, your links break, pure and simple, and that does happen, as the demise of the original tr.im and cli.gs services show.

But there is a way to take all the benefit that short URLs offer and keep ownership of your links and all the data that clicks on those links will give you and that’s to run your own URL shortening service, which is precisely what I’ve done with vtny.org which is running the YOURLS code behind the scenes. This gives me all the benefits and metrics that other URL shorteners provide but with the added and crucial benefit that I now own the links and the data they generate, in this case via the vtny.org/4 short URL.

Photo credit: playerx and revrev on Flickr

Written and posted from home (51.427051, -0.333344)

Geographic and Transport Data; a Tale of Capricousness, Whimsy and Downright Insanity

Gary — Fri, 11 Dec 2009 23:38:45 +0000

The industry I work in thrives on data; we consume loads of the stuff and in turn we generate petabytes of it. I’m talking about data in general, not the geographic, mapping or place data that I usually write about.

But the longer I work in the Internet industry the more convinced I become that, as an industry, we need to get our act together. How else to explain the bizarre, rapidly changing and capricious nature of how we gain access to, use, pay, don’t pay and disseminate data?

We’re socially conditioned to assume that free does not equate to good, hence the adage “there’s no such thing as a free lunch“. So stuff that costs is good and stuff that’s free isn’t. But normal rules don’t apply here.

Let’s take geographic data; I’m on home ground here so this should be relatively straightforward.

The proprietary data vendors, Navteq, TeleAtlas and others, charge for their data and limit what you can and can’t do with it. OpenStreetMap on the other hand charges nothing for its’ data and only places limits on the data to protect the data by way of the Creative Commons Attribution Share Alike license.

So naturally the data you pay for should be good and the data you don’t pay for should be … less than good. Naturally.

Except OpenStreetMap data isn’t less than good. UCL’s Muki Haklay summed this up neatly as “How good is OpenStreetMap? Good enough” at the OpenStreetMap conference in Amsterdam this year. Conversely, the proprietary data vendors don’t always get it right. One data vendor, who will remain anonymous, shipped a release of data with wildly incorrect centroids, the lat/long coordinate which represents the nominal centre of a place, which meant that amongst others, Covent Garden ended up being centred on Holborn Underground Station.

This isn’t an isolated incident.

On the one hand, the City of Vancouver in British Columbia makes its data, all of its data, free and open. On the other hand, the City of Tempe in Arizona decides to charge a “fair approximation of market value” for its data, which as James Fee recently discovered means that you’ll need to cough up $100,000 to use it commercially.

In San Francisco, BART, the Bay Area Rapid Transit, makes their data which includes train times freely available and taking a refreshingly prosaic approach to accessibility and licensing.

Getting an API key: Psyche: you don’t need one. We’re opting for “open” without a lot of strings attached. Just follow our simple License Agreement, give our customers good information and don’t hog resources. If that doesn’t work for you, we can certainly manage usage with keys and write more terms and conditions. But who wants that?

Here in the UK TFL, Transport for London, give you some data for free but not the train times and for overground trains the Association of Train Operating Companies (pdf link) value this data at a staggering £27,430 per year

And elsewhere in the world, other operators are closing down people who want to use this data, in New York, in Berlin, in New South Wales and we can’t really seem to work out who owns the data and whether there’s intellectual property being infringed or a public service being undertaken.

… and don’t even talk about the British postal code data was closed, was then going to be opened up but now isn’t. Apparently.

With all the data we consume and emit, we spend a lot of time and effort evangelising APIs and web services that use it. But as an industry we really need to start to act clearly and consistently in order to be taken seriously and in order for the Internet industry to realise the potential that we all think it’s capable of.

Posted via email from Gary’s Posterous

The (Geo) Data Dichotomy Dilemma

Gary — Mon, 16 Nov 2009 18:30:57 +0000

Before Web 2.0, before mashups, before FreeOurData.org.uk and other pleas, before the Internet itself, things used to be so much simpler for geo data. You were either an end user and accessed the data as a map or you were a GIS Professional and accessed the data via a (frequently very expensive and very specialised) Geographical Information System. But now we have geo data, lots of geo data, some of it free, some of it far from free, both in terms of usage and cost and a fundamental problem has replaced the paucity of data.

Everyone wants free, open, high quality geo data and no one wants to pay for it. But it’s not quite that simple.

The recent acquisitions of Tele Atlas and Navteq, the two big global geo data providers, by TomTom and Nokia respectively show the inherent value in owning data. But owning the data isn’t enough any more as the market for licensing the data is a shrinking one, despite the phenomenal growth of the satnav market, both in car and on mobile handsets. Why is the market shrinking? Because no one wants to pay for it, at least directly.

TomTom, primarily a hardware vendor, are differentiating into the software and data market, seems to be concentrating on the PND usage of the data, although we’ve yet to see how the outlay necessary to acquire Tele Atlas coupled with the overall economic downturn will effect their overall 2009 earnings. Their Q1 2009 report somewhat dryly notes that “market conditions were challenging” and that “we are making clear progress with the transformation of Tele Atlas into a focused business to business digital content and services production company“. There may be other aspirations at play here but for now at least, the company is keeping quiet.

Nokia, also primarily a hardware vendor in the form of mobile and cellular handsets, are also moving away from their roots and into a wider market, hopefully in an attempt to stop the encroachment of upstarts such as HTC, Apple and RIM into Nokia’s traditionally strong smartphone heartland. Again, Nokia has yet to make a public play into this arena but all the composite elements are in place to enable this to happen.

Taking the opposite route, Google, which started off as a software player are now moving to being a player in the data market by gathering high quality geo and mapping data under the smokescreen of gathering Street View. This has allowed them to gather sufficient data to supplant Tele Atlas as a data provider, at least in the Continental United States.

All three companies are either making or have the prospect of making determined plays in the location space but all three of them have ways of leveraging the value inherent in their data. Google has their unique users, their search index and a vast amount of advertising inventory; TomTom their satnav customers; Nokia their handset customers, albeit one level removed with the Mobile Network Operators as an uneasy partner and intermediary.

So what of the open data providers? It’s important to remember here that open doesn’t always mean free, it means the ability to create derived works and to use the data in ways that the originator may not have immediately foreseen. True, a lot of open data is free, but even then it’s the Free Software Foundation’s definition of the word.

“Free (software) is a matter of liberty, not price. To understand the concept, you should think of free as in free speech, not as in free beer.”

The poster child of open geo data is OpenStreetMap, the “free editable map of the world”. Founded in 2004 by Steve Coast, OSM has enjoyed phenomenal growth in users and in contributions of data that can be used anywhere and by anyone and which espouses the values of free as in speech and as in beer. As with all community or crowd sourced collaborative projects, OSM’s challenge is to sustain that growth and once complete coverage of a region is reached, in keeping that coverage fresh, current and valid. We’ll leave aside that fact that complete coverage is an extremely subjective concept and means many things to many people.

Traditionally strongest in urban regions, one of OSM’s other key challenges is to match the expectations of their user community who consume that data rather than those who create it. Both internationalisation of the data and expansion out of the urban conurbations will potentially prove challenging in the years to come. That’s not to say OSM isn’t a significant player in this space and the quality of the data, though varying and in some places duplicated, is for the majority of use cases, good enough. This was backed up by research undertaken by Muki Haklay of UCL which answered the perennial question of “how good is OSM data” with a pithy “good enough”.

Attempts to capitalise on and monetize the success and data corpus of OSM through the Venture Capital funded Cloudmade have yet to deliver on the promise and with the exception of a set of APIs, Cloudmade has announced the loss of their OpenStreetMap Community Ambassadors and the closure of their London office. All of which lends credence to the fact that simply owning the data isn’t enough.

So how to solve the dichotomy of geo data? Everyone wants it but no one’s willing to pay for it with the exception of the big players, the Googles, the Yahoos and the Microsofts of the world and control of the proprietary data sources has centralised into TomTom and Nokia, both of whom are well placed to capitalise on their data assets but who haven’t yet delivered on that promise.

Maybe the answer is twofold. Firstly develop an open attribution model whereby the provenance of an atom of data can be tagged and preserved; this would remove a lot of the prohibitions on creating derived works at the original data provenance could still be maintained. Secondly allow limited usage of proprietary data at varying levels of granularity, accuracy and currency, thus creating a freemium model for the data and stimulate developer involvement in donating data to the community as a whole.

It’s too early to see whether this will come to pass or whether an already tight hold on the data will become tighter still.

Posted via email from Gary’s Posterous

O2 in Positive Customer Service Shock?

Gary — Wed, 07 Oct 2009 03:11:15 +0000

O2, the UK Telefonica brand and soon-to-be-loosing-the-iPhone-exclusivity-to-just-about-anyone mobile operator, have a reputation which is, to be honest, just a little bit crap. Their coverage in the rural wilds of Central London, especially around Soho and Covent Garden, seems to be scaled for a single user and a web search for “o2 customer service problems” throws up such gems as “O2 customer service consists of PAY UP OR ELSE” and “O2′s customer service has to be the poorest I have ever come across“.

So we’ll leave aside for one moment the fact that I have to pay an additional £20.00 for a measly 10MB of data when abroad via O2′s Data Abroad 10 bolt on and accept that I ordered this to be added to my account so I could use data on my iPhone when in the US for this week’s Open Hack NYC.

The first mailed response from O2 didn’t inspire confidence.

“Hi, Thanks for getting in touch. We’ll look into your query and get back to you as quickly as we can, normally within 24 hours.”

So I waited and less than 24 hours later I got this

“Good Morning Gary. Thanks for emailing us about adding the 10Mb Data Roaming Bolt On to your account.
Gary, you’ll be pleased to know that I’ve added the 10Mb Data Roaming Bolt On to your account effective from your next bill onwards (10 October 2009). You’ll be charged £17.02 excluding VAT (Value Added Tax) per month for this Bolt On.

If you want to add the above Bolt On on a different date, please reply to this email and we’ll help you further.”

Data roaming on; WIN. Data roaming on from the date of my next bill and after the event in New York; FAIL.

So I asked them, nicely.

“I’m having to travel at very short notice so I really need this up and running from my first day out of the country which is this Wednesday, October 7th. Can the bolt on start date be brought forward to this day?”

That automated reply came back again

“Hi, Thanks for getting in touch. We’ll look into your query and get back to you as quickly as we can, normally within 24 hours.”

I’d expected a cut-and-paste response that they could only start services such as this on the first day of a new monthly bill, which basically means minimal work for them and maximum inconvenience for the customer. Then this morning I got this, which was emphatically not what I was expecting.

“Good Evening Gary. Thanks for emailing us as you want to pre-phone your Bolt On start date. I’ve pre phoned your Bolt On start date to 07 October 2009 as requested by you. Important – When you email us please provide: your date of birth, postcode and mobile number as it helps us answer your query faster”

So fair play to you O2; I’m not entirely sure what pre-phoning is and a bit surprised that you expect me to provide personal data including my date of birth and postal code in every email, but I went into this dialogue with you with zero expectation of success and you pleasantly surprised me. Now if we can just fix that “No Service” in Central London …

Posted via email from Gary’s Posterous

Deliciousness: data, licensing, WordPress autosaves, cheese in space and lots of Nutella

Gary — Fri, 31 Jul 2009 14:15:40 +0000

More intriguing, interesting and just plain bonkers stuff from the information hose pipe we call the internet:

Starting off with a serious note, Ed Parsons, my opposite number at Google, wrote a great blog post on the knots that data licensing can tie you up in and why you end up paying more for a leased digital version than you do for the physical paper version.
WordPress started bugging me about an auto-saved version of a blog post I didn’t want to keep but couldn’t get rid of. Turns out there’s no way to do this from the WordPress dashboard but some MySQL hackery did the trick.
“I am, and am VERY badly affected by being in close proximity to WiFi and other microwave transmission sources. Not that I’d expect you or anyone else who isn’t adversely affected to believe me“. The rest of the story on the Daily Telegraph blog is priceless.
Ofcom confirmed what anyone with the UK ADSL line already knows, that the average UK broadband speed is just over half of what’s being advertised and paid for.
A US highway exit sign got every word misspelled, apart from the word “exit”.
Forget putting men on Mars or getting the Space Shuttle working; we put cheese into space, tracked it, lost it and found it again. Makes you proud to be British.
Someone likes Nutella. A lot.
And finally, if your iPhone gets a text message containing a single square character. Turn it off. Turn it off now.