Posts about data

Open Data Yields Tangible Results - And Tangible Maps

2013-05-07 18:54

2013 would be the year of the tangible map.

This hope was prompted by the maps I saw at one of London's geomob meetups in November of 2012, where I saw and, importantly for a tangible map, touched Anna Butler's London wall map and a prototype of David Overton's SplashMap.

The hopeful prediction was made as a result of literally getting my hands on one of Anna's London maps and it's a treasured possession, though still sadly needing a suitable frame before it can take pride of place on a wall at home.

But what of SplashMaps? In November 2012 the project was on Kickstarter and I was one of the investors in this most tangible of maps. In December 2012 Splashmaps met their funding targets and went into production and today, through the letterbox came my own, tangible, foldable, scrunchable and almost indestructible SplashMap of my local neighbourhood.

In January of this year I made a hopeful prediction that 2013 would be the year of the tangible map.

This hope was prompted by the maps I saw at one of London's geomob meetups in November of 2012, where I saw and, importantly for a tangible map, touched Anna Butler's London wall map and a prototype of David Overton's SplashMap.

The hopeful prediction was made as a result of literally getting my hands on one of Anna's London maps and it's a treasured possession, though still sadly needing a suitable frame before it can take pride of place on a wall at home.

But what of SplashMaps? In November 2012 the project was on Kickstarter and I was one of the investors in this most tangible of maps. In December 2012 Splashmaps met their funding targets and went into production and today, through the letterbox came my own, tangible, foldable, scrunchable and almost indestructible SplashMap of my local neighbourhood.

Now all if this could be taken to be simply my crowing with delight over maps. But there's a deeper context to all of these tangible maps. Both the London Wall Map and SplashMaps have come about due to one single thing ... open data. The case has often been made, though equally as often misunderstood, that open data is an economic stimulus. As many people ask why should we give something away for free as ask for data to opened up to the public.

Both of these maps wouldn't have been financially possible without access to open data; the pre-open data era licensing costs and restrictions alone would have put paid to any startup opportunities an aspiring entrepreneur came up with. But in these maps, the proof of what open data can do has become very real, indeed very tangible.

Map Push Pins vs. Dots? Google Map Engine vs. Dotspotting?

2013-03-28 14:32

Google launched their Maps Engine Lite beta; a way of quickly and easily visualising small scale geographic data sets on (unsurprisingly) a Google map. The service allows you to upload a CSV file containing geographic information and style the resulting map with the data added to it. I thought I'd give it a try.

I turned to my tried and trusted data set for things like this; a data set I derived from a Flickr set of geotagged photos I'd taken of the London Elephant Parade in 2010. It's a known data source and I know what the results of this data set will give me; it lets me do a reasonably meaningful visual comparison of how a particular product or service interprets and displays the data.

Yesterday, Google launched their Maps Engine Lite beta; a way of quickly and easily visualising small scale geographic data sets on (unsurprisingly) a Google map. The service allows you to upload a CSV file containing geographic information and style the resulting map with the data added to it. I thought I'd give it a try.

I turned to my tried and trusted data set for things like this; a data set I derived from a Flickr set of geotagged photos I'd taken of the London Elephant Parade in 2010. It's a known data source and I know what the results of this data set will give me; it lets me do a reasonably meaningful visual comparison of how a particular product or service interprets and displays the data.

Reading up on Map Engine Lite, I noted that I could only upload a maximum of 100 data points into a layer on the map, which wasn't a problem as my data set is localised to London and contains only 10 pieces of information, one for each photo I'd taken. Once I'd uploaded the data I could style the colours of the push pins and the background style of the map. It looks pretty good, even if you are limited to 100 points per layer and it's for strictly personal and non commercial use only.

But I was sure I'd seen this sort of thing before and I had, in the form of Stamen's Dotspotting. I already had an account with Dotspotting and, even though I'd forgotten about it, I'd previously made a map from my London Elephants data set.

The parallels are many. Both Map Engine and Dotspotting allow you to upload data in CSV format. Both services try to work out coordinates from the data, if there's no lat/long coordinates already. Both services allow you to style the resultant map.

There are differences. Dotspotting allows you to download your data; it doesn't appear that Google does. Map Engine allows you to style the map markers; it doesn't seem that Dotspotting allows this. Dotspotting supports Excel spreadsheets, CSV files, Flickr and Google My Maps feeds; Map Engine only supports CSV files.

There's also one other key difference; Map Engine was launched yesterday, whilst Dotspotting was launched 2 years ago.

But there's an old saying that imitation is the sincerest form of flattery.

Mapping Meteor Strikes; There's A Lot More Than You'd Think

2013-02-20 14:56

meteor that exploded over and hit the Russian city of Chelyabinsk in the Urals made several thoughts go through my mind. In this order.

I feel for the 1200 people who were hurt and injured
Thank goodness it didn't happen where I live
With all the asteroids and smaller pieces of rock zooming over our head, this has got to have happened before, hasn't it?

Last week's 10,00 ton and 55 feet's worth of meteor that exploded over and hit the Russian city of Chelyabinsk in the Urals made several thoughts go through my mind. In this order.

I feel for the 1200 people who were hurt and injured
Thank goodness it didn't happen where I live
With all the asteroids and smaller pieces of rock zooming over our head, this has got to have happened before, hasn't it?

On the subject of the last thought, it turns out this has happened before. A few times. Actually close to 35,000 times. The Meteoritical Society has a data set detailing these. It would make a great map. Which is exactly what Javier de la Torre, co-founder of CartoDB has done.

A map of impact points would be effective enough, but Javier's use of a heatmap not only shows the global spread of the debris which has been raining down on our planet since 2,300 BC but also shows the density of strikes, which makes the map simultaneously more effective and accessible.

There's also been far more strikes in the United Kingdom than I would have either thought or feel vaguely comfortable about, if you can ever be comfortable with things falling from the sky with horrifying effect.

Definitely a map to file under the I wish I'd done that category.

GeoPlanet Data Resurfaces For Download; On The Internet Archive

2013-02-08 08:53

GeoPlanet Data page on Yahoo's Developer Network site and instead of a set of download links, you see "We are currently making the data non-downloadable while we determine a better way to surface the data as a part of the service.".

Although I can't find the originator of the saying that there's no delete button for the internet, it's a saying that's very true. If you put something up on a web site, be it a photo, some text or perhaps a file of geographic data there's a very good chance that someone else has a copy, even if you subsequently take the original down. It's a sort of digital whack-a-mole.

This is all too apparent in the story of Yahoo's GeoPlanet Data download. When I was part of the Yahoo! Geo Technologies team, we released a public download of the Yahoo! WOEID data set, under the CC BY 3.0 license, in 2009 at Where 2.0. More about that license in a moment.

As Yahoo! continues to undergo change under the leadership of Marissa Meyer, the current data file and all earlier versions were taken offline. Visit the GeoPlanet Data page on Yahoo's Developer Network site and instead of a set of download links, you see "We are currently making the data non-downloadable while we determine a better way to surface the data as a part of the service.".

But the digital mole that is the WOEID data has resurfaced, and versions 7.4 through 7.10 of GeoPlanet Data can now be found on the Internet Archive.

But Yahoo! has taken down the downloads, so how can this happen? That's where the CC BY 3.0 license comes into play. The Creative Commons Attribution 3.0 license, to give CC BY 3.0 its full name, gives anyone the right to share the data, in other words to copy, to distribute or to transmit the data, providing users of the data attribute it back to Yahoo! Once issued under such a license, it can't be revoked; you may choose to issue a new version under a different license scheme or stop issuing new versions entirely, but the earlier versions remain under the original license.

I've always had a soft spot for the WOEID and for the GeoPlanet API and data download. Maybe this new availability of the data set will stimulate new usage of WOEIDs. Who knows, the data may even be forked and added to?

The Problem With Location Based Mobile Services

2013-02-03 12:50

privacy or tracking. Nor is the problem one of an LBMS dying and going away. The problem isn't whether I can get a good location fix or whether the results I get are accurate or not. The problem isn't even of the value of the data we, the customer, put into a service and whether we can get it back again.

There's a problem with today's crop of location based mobile services, commonly referred to as LBMS; those little apps which sit on our smartphones and allow us to geotag status updates or photos, find relevant local place information or check-in at a place.

The problem isn't one of privacy or tracking. Nor is the problem one of an LBMS dying and going away. The problem isn't whether I can get a good location fix or whether the results I get are accurate or not. The problem isn't even of the value of the data we, the customer, put into a service and whether we can get it back again.

No, the problem is whether we can actually use the service from our smartphone at all.

It's 2013 and I live in the suburbs of the capital of the United Kingdom and this happens all the time. Not in the uncharted wilds of the UK. Not in obscure regions of the world. But in my local neighbourhood and in the heart of London. And it's not just a problem with Vodafone, my current cellular provider. Over the last few years I've been on T-Mobile, on Orange and on O2 and all the cellular carriers seem to have exactly the same problem; one which makes a mockery of their coverage maps. According to Vodafone's map, I should be getting high or at least variable 3G data coverage where I live, but instead I get variable or no coverage at all when walking in my local neighbourhoods.

3G data coverage that drops in and out; that's the problem with today's location based mobile services.

I'm getting off of my soapbox now ...

2013 - The Year Of The Tangible Map And Return Of The Map As Art

2013-01-01 10:54

Watercolor, the vast majority of digital maps can't really be classified as art. Despite the ability to style our own maps with relative ease, such as with Carto and MapBox's TileMill, today's maps tend towards the data rich, factual end of the map spectrum. Compare and contrast a regular digital map, on your phone, on your tablet or on a web site in your laptop's browser with a map such as Hemispheriu[m] ab aequinoctiali linea, ad circulu[m] Poli Arctici and you'll see what I mean (and if you don't browse the Norman. B. Leventhal Map Center's Flickr stream you really should).

Looking back at the conference talks I gave and the posts I wrote in 2012, two themes are evident.

The first theme is that while there's some utterly gorgeous digital maps being produced these days, such as Stamen's Watercolor, the vast majority of digital maps can't really be classified as art. Despite the ability to style our own maps with relative ease, such as with Carto and MapBox's TileMill, today's maps tend towards the data rich, factual end of the map spectrum. Compare and contrast a regular digital map, on your phone, on your tablet or on a web site in your laptop's browser with a map such as Hemispheriu[m] ab aequinoctiali linea, ad circulu[m] Poli Arctici and you'll see what I mean (and if you don't browse the Norman. B. Leventhal Map Center's Flickr stream you really should).

The second theme is that despite the abundance of maps that surround us these days, a digital map is almost by definition an intangible thing. It's a view port, hand crafted by a digital cartographer, on a mass of hidden, underlying spatial data. It's ephemeral. Switch off your phone, your tablet, your sat nav or your computer and the map ... vanishes. Until the next time you hit the "on" button, the electrons flow again and the map re-appears. But it's still intangible, despite the irony that a lot of maps these days are interacted with via a touch interface; we tap, poke, prod and swipe our maps, but they're not really there.

But maybe 2013 will be both the year of the tangible map and the year of the map as art. It might be if the closing days of 2012 are anything to go by.

On December 8th, 2012, David Overton's SplashMaps made their funding total on Kickstarter. A SplashMap is a real outdoor map, derived from (digital) open data, but rendered on a light and weatherproof fabric. It's a tangible map in the truest sense of the word; one you can fold up or even crumple up and stick in your pocket, safe in the knowledge that it won't fade away. There's no "off" switch for this map. As one of the SplashMap funders, I'll have a chance to get my hands on one in the literal sense of the word in a couple of months, once they hit production. So more about this map in a future post.

The other map that is both 100% tangible and 100% art is the awesomely talented Anna Butler's Grand Map Of London. A modern day map of the UK's capital city, digital in origin, lovingly hand drawn in the style of the 1800s and printed, yes, printed on canvas. It's a map worthy of the phrase "the map as art" and when I first saw one and handled one in late November of 2012 I wanted one, right there and then.

And then, on Saturday, December 29th 2012, Mark Iliffe and I met Anna for a coffee in the Espresso Bar of the British Library on London's Euston Road and out of the blue, Anna handed over a long cardboard tube containing my own, my very own, Grand Map Of London. People nearby looked on, slightly non-plussed as I crowed like a happy baby, promptly unrolled the map over the table and just looked and touched. The next half an hour or so pretty much vanished as I pored over and luxuriated in the map, lost in the details and revelling in the map under my hands. Truly this is a tangible map which is itself art.

I've often said, half in truth, half in jest, that I'd love a big, as big as I can get, map of London on my wall, probably one of Stamen's Watercolor maps. But Anna's Grand Map Of London will be getting a suitable frame and sitting on my wall, just as soon as my local framing shop opens after the New Year break.

Two maps to wrap up 2012. Both tangible, both digital in origin, both made for looking, touching and feeling. One clever, innovative and utterly practical and one a map you can keep coming back to and which reveals more artistic cleverness each time you look at it.

2013 is shaping up to be a "year of the map" in ways I'd never had hoped for at the start of 2012.

Work+ - A Fantastic Idea For A Location Based App; Shame About The Metadata Though

2012-10-01 15:26

mistaking the context (location) for the end game and that location is (also) a key context, but most people don't know this. Two years or so after I wrote those posts, the concept of location based mobile services and location based apps shows no sign of dying off. I see lots of new location based apps and whilst they're almost always nice and glossy, not that many of them really grab you as a neat and innovative idea. But every so often, one does come along which makes you slap your forehead, like the scientists in the 80's ads for Tefal, and mutter under your breath ... that's so obvious, why didn't I think of that?

I once wrote two posts saying that people are mistaking the context (location) for the end game and that location is (also) a key context, but most people don't know this. Two years or so after I wrote those posts, the concept of location based mobile services and location based apps shows no sign of dying off. I see lots of new location based apps and whilst they're almost always nice and glossy, not that many of them really grab you as a neat and innovative idea. But every so often, one does come along which makes you slap your forehead, like the scientists in the 80's ads for Tefal, and mutter under your breath ... that's so obvious, why didn't I think of that?

[youtube https://www.youtube.com/watch?v=mWWbUd2CGCM&rel=0]

These days I tend to work as much out of the office than I do in the office. My needs for this are relatively few; somewhere to plug my laptop in, free wifi and a half-way decent cup of espresso now and again. Using local location based search services I can find places near me that meet these needs but it's a disjointed experience, using multiple apps to find free wifi, good espresso and so on. Maybe the recently launched Work+ can help me out here?

First impressions are good. I launch the app and connect it to my Foursquare account (the check-in feature within Work+ is a nice touch). Work+ also passes the first hurdle than many location based apps fail at; it actually works outside of the United States.

I install the app, tap on Work and then Go to launch the easy to use search interface. I need wifi ... tap. I need a table to put my laptop on ... tap. I need coffee ... tap.

Ideally I'd also like to see a search setting that says "by coffee I mean decent espresso and no, I don't mean Starbucks" but maybe I'm being overly picky here.

So I tap on Search and I get a list of places that are close by to me that meet my needs or I can view those places on (Apple's new) map. This is great. What is there not to like?

But wait, do all of these places actually meet my needs? The search results seem good, there's no duplicates or places that either don't exist or have since closed; problems which can plague location based services and which are by no means simple to solve. The results are also pretty close to where I am. But ...

The two hits for Costa Coffee are pretty good; as the name implies they both sell (reasonably passable) coffee and have (free-ish but time limited) wifi. Score, 2/7.
The same goes for Caffe Nero, another one of the big UK coffee chains. Score, 3/7.
Caffe Toscana is my local neighbourhood cafe. Great food and coffee ... but no wifi, at least not when I visited last week. Score, 3/7
Astrora Coffee isn't a cafe. They sell coffee in the raw, roasted beans and ground beans. No wifi and not really somewhere you can work; I'd imagine the staff getting somewhat bemused if someone turned up and tried to work there. Score, still 3/7.
Diner's Delight is as the name suggests, a local diner. No wifi here either. Score, 3/7 again.
Finally, The Nearest Cafe is a cafe and they do sell pretty good coffee. But again, no wifi here.

The final score ends up as 3 hits that really meet my needs, out of a possible 7.

It would be easy to take what I've just written as an indictment of Work+ but nothing could be further from the truth. Local search is not an easy thing to do. Tightly focused local search across a wide range of attributes that you can assign to a place (wifi, coffee and so on) is insanely difficult to do. It's true that Work+ doesn't score as highly as I'd have hoped in what is admittedly a very subjective search on a very limited local area. But Work+ shows the direction that local search is headed in. It's no longer enough to ask find me what's around me, we need to be able to ask find me what's around me that fits what I need to know now and more importantly get good answers to that question.

What makes the Work+ experience not quite as good as it could be isn't down to the app, which makes local search a pain free and simple process. What lets Work+ down is the lack of a complete local data set which contains not just the accepted standard place attributes of name, address, location and category but also which adds in more detailed, almost ambient or fuzzy, attributes, such as wifi, capacity (can I fit a large group of people in here?), beverage types (coffee or tea?), noise level and ambience.

Make no mistake, Work+ is a precursor to the local search and location based experiences we can expect to see in the very near future; whether the back-end data with all of the rich attributes that people want to search on will keep up with demand remains to be seen.

Maps, Maps And MOAR Maps At The Society Of Cartographers And Expedia

2012-09-06 08:45

Commission on Neocartography. Cartography, neocartography, maps; what is there not to like? I'd previously spoken at the UK's Society of Cartographer's annual conference so it was great to be asked by Steve Chilton, SoC and Neocartography chair, to speak at the Neocartography Commission.

Updated September 13th. 2012 with embedded YouTube video.

Wednesday September 5th. 2012 was a day of maps. To be precise, it was a day of maps, maps and MOAR maps. Two events, two talks, back to back. Packed choc-a-bloc full of maps. I also cheated slightly.

Firstly there was the International Cartographical Association's first session of the newly formed Commission on Neocartography. Cartography, neocartography, maps; what is there not to like? I'd previously spoken at the UK's Society of Cartographer's annual conference so it was great to be asked by Steve Chilton, SoC and Neocartography chair, to speak at the Neocartography Commission.

For a change, the talk title and abstract I gave Steve didn't vary during the usual researching and writing of the talk.

`Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Apple Message framework v1278) Subject: Re: Neocartography workshop X-Universally-Unique-Identifier: d1c70302-eaba-4132-80fb-f74eb1de2347 From: Gary Gale In-Reply-To: DEC2FCE18B20734CAFA668E438482963834F621862@WGFP-EXMBV1.uni.mdx.ac.uk Date: Fri, 20 Jul 2012 14:13:39 +0100 Content-Transfer-Encoding: quoted-printable X-Smtp-Server: mail.vicchi.org:redacted Message-Id: BEB576E2-3E8C-4136-803A-0CE5E5456C26@vicchi.org To: Steve Chilton

Actually, I'm going to change the title ... what I'd really like to see up on the web site is this ...

Title: History Repeats Itself And So Does The Map Abstract: Steve Chilton says this just MIGHT be interesting; you'll have to take his word for this

... but that might not work. So try this for size instead

Title: History Repeats Itself And So Does The Map Abstract: History has a habit of repeating itself and so does the map. From primitive scratchings, through ever more sumptuous pieces of art, through to authoritative geographical representations, the map changes throughout history. Maps speak of the hopes, dreams and prejudices of their creators and audience alike, and with the advent of neogeography and neocartography, maps are again as much art as they are geographical information.

... will that do?

G`

But then, no sooner had I got one event for that Wednesday when fellow Yahoo! alumni and now Expedia developer and chief evangelist Steve Marshall asked me to team up with ex-Doppleran and ex-Nokian Matt Biddulph at Expedia's EAN World of Data event which was cunningly masquerading as a BBQ that very Wednesday evening. So I cheated. One day. Loads of maps. Two events. But one talk. Only time will tell whether I got away with it or not.

My talk at the Neocartography workshop was filmed and you can watch it below, if you like that sort of thing. Personally I hate seeing myself on video, it's even more excrutiating than hearing myself on audio.

[youtube=https://www.youtube.com/watch?v=mSRWy9kMf00]

As usual, the slide deck, plus notes are embedded below, also if you like that sort of thing.

[scribd id=105081787 key=key-28dj39ezex1j55yczevw mode=scroll]

Slide 3

So, hello, I’m Gary and I'm from the internet. I’m a self-confessed map addict, a geo-technologist and a geographer. I’m Director of Places for Nokia’s Location and Commerce group. Prior to Nokia I led Yahoo’s Geotechnologies group in the United Kingdom. I’m a founder of the Location Forum, a co-founder of WhereCamp EU, I sit on the Council for the AGI, the UK’s Association for Geographic Information, I’m the chair of the W3G conference and I’m also a Fellow of the Royal Geographical Society.

Slide 4

Most of my talk have a lot of links in them and sometimes I see people rushing to take a note of them if they see something they think is interesting. You might want to do this too, but don't ...

Slide 5

... this is the only URL in the entirety of this talk you might want to take a note of. It's nice and short and easy to scrawl down. Although if you go there right now, it'll just take you to the home page of my blog, but sometime tomorrow or the day after this is where this slide deck, my notes and all the links you'll be seeing will appear.

https://vtny.org/kk Slide 6

It's also fair to say that this talk is something approaching a personal first. When I'm asked to give a talk, I'm usually asked for a title and an abstract some 3 or so months before the talk. That's also a long time before I actually start writing the talk.

Slide 7 / Slide 8

... but this time, not only has the talk title stayed the same, the abstract still fits and it's even the talk I set out to write, and I have the email to prove it.

Slide 9

But enough about me. Let's set some context. We live in a connected world of interwebs and mobiles. Some of you probably know of this thing on the interwebs called Twitter which has hashtags to identify common themes. A popular hashtag is for people who like to take photos of their food. They use the hashtag #foodporn.

Slide 10

Well I take photos of maps and there's lots of maps in this talk. You could say it's pure unadulterated #mapporn and I make no apology for it.

Slide 11

But before I talk about today's maps, I want to set a little historical context.

Slide 12

This is one of the earliest maps we know of, of the world as the Babylonians thought of it. Babylon is in the centre of the map and there's seven triangular islands, 3 of which are missing due to damage, in the "river of bitter water", or the sea. To me, the Babylon map is both art, hope and inspiration for the unmapped areas of their world and the best attempt of the age to be authoritative.

Slide 13

Fast forward several centuries to the "golden age of exploration" and while maps are more recognisably accurate, they're still art. But this art came at a price. You needed to be wealthy to commission such a map and such a map was often given as a notional gift to the rich and powerful to curry favour.

https://www.flickr.com/photos/normanbleventhalmapcenter/2675672726/ Slide 14

Furthermore maps were state secrets; sharing maps was sharing power and influence. The entrepreneurs of the time were the great navigators like Columbus and Magellan, their sponsors were kings and countries; their business plan were maps.

https://www.flickr.com/photos/normanbleventhalmapcenter/5385389984/ Slide 15

But maps don't just have to be geographically accurate. They can show data as well. This 1869 map by Charles Minard shows the losses suffered by Napoleon's army in his 1812 Russian campaign. Beginning at the Polish/Russian border on the left, the thick pinkish band shows the size of the army as they advanced towards Moscow. The thinner black band shows the ever decreasing size of the remains of the army as they retreated in the bitterly cold winter.

https://en.wikipedia.org/wiki/File:Minard.png Slide 16

Another type of not necessarily geographically accurate map are the familiar mass transit and metro maps that you probably all recognise, all descended in some shape or form from Harry Beck's iconic map of London's Tube system.

Slide 17

And then, there's the map that most people of my generation will find immediately familiar, the Ordnance Survey map, from the printed version of the pink LandRanger series of maps through to the online version still found at certain zoom levels on streetmap.co.uk.

Slide 18

And no far too quick resume of maps would be complete without the maps we use on an almost daily basis, from Nokia ...

Slide 19

... from Google

Slide 20

... from Bing

Slide 21

... and from OpenStreetMap. All of these are pretty much authoritative, geographically accurate and cartographically pleasing to the eye. But from the maps of 16th and 17th centuries to today's web and mobile maps, there's something missing. There's some brilliant cartography at work but the art seems to have gotten lost somewhere along the way. Or has it?

Slide 22

This map of where I live, using the Watercolor style from San Francisco based Stamen, is as geographically accurate and authoritative as the maps from Nokia, Google and others, but to my mind this is most definitely art. It also happens to be my second favourite map.

Slide 23

So how did we get here ...

Slide 24

... how did we get from cuneiform impressions on a baked mud tablet ...

Slide 25

... to a watercolour style map that, if allowed, I'd want to hang on my wall in a large as possible size as I could get.

Slide 26

I think the answer is data. Lots of data. Easily accessible data, either in bulk or through an API, either free and open or licensed and proprietary. We now have access to the raw spatial data that was previously the preserve of the professional cartographer alone. People can start to make their own maps, their personal, subjective, art maps in ways never previously possible and they can do this because they want to and just because they can.

Slide 27

We can make maps, not only of what is and of what was but also of what might have been but which never came to pass, such as this map of what Berlin would have looked like if Albert Speer's plan for the city had been realised and if the events of 1945 had been very different from what is in our history books.

https://mypantsareonfire.tumblr.com/post/30302438119/albert-speers-new-berlin Slide 28

There's also a flip side to all of this though. Just because we can make maps from all of this wonderful data we now have doesn't always mean we should make maps, such as this gem from the Ottawa Sun which manages to put Saudi Arabia in Africa, puts Iran on the Arabian Peninsula where Saudi Arabia should be, overlooks the fact that the Sudan is now two countries and gets Malaysia and Indonesia confused.

https://ottawa.openfile.ca/blog/ottawa/2012/suns-same-sex-marriage-map-puts-saudi-arabia-africa-among-other-cringeworthy-errors Slide 29

But the topic of bad maps is another talk entirely, so let's get back to good maps. With access to the underlying spatial data and to other data sets with a spatial element, we can now make maps which provide insights into mapping the unmappable. Some cities have formally defined neighbourhoods, London doesn't. But Tom Taylor's Boundaries took a spatially correct map and mashed up Flickr's Alpha data set to show not where London's neighbourhoods are, but where people think the neighbourhoods are. Which may not be 100% accurate but it's a darn sight better to have a notion of where London's neighbourhoods are than no notion at all.

https://boundaries.tomtaylor.co.uk Slide 30

Then there's this map. Definitely one to be filed under the category of "maps because we can", at first sight this map just looks like the US, with lines joining up the notional centroids of each State. Until you start to play with it in a web browser

Slide 31

And all of a sudden you can start to see what would happen if you decided you really didn't want California, Florida or Texas to be where they currently reside. You can play with this for hours. I did. When I really should have been finishing this talk.

https://mbostock.github.com/d3/talk/20111018/force-states.html Slide 32

The more data sets that people produce, the more people can and will make maps with them, so if you'd ever had a yearning to see where people have discovered fossils, for example, then there's a map to show you that. You can argue that this is nothing more than a classic Web 2.0 style maps mashup, but give people a spatial data set and they'll make a map out of it and sometimes that's good enough

https://earth-base.org/fossils Slide 33

But sometimes people will go several steps beyond a maps mash up and produce something which is only just recognisable as a map and is much more about the data visualisation. Like plumegraph.

https://plumegraph.org Slide 34

Here, the map is relegated to a small piece of digital canvas on which the data is projected. But it's still a map, it's still accurate and even if the data being visualised is part of humanity's less attractive side, it's still visually gorgeous. It's still a map.

Slide 35

Now people sometimes make the mistake of assuming that all of this data we're making maps out of is a relatively recent thing. But long before we had the convenient label of "Big Data", organisations such as the NOAA were creating data sets you could make maps out of, we just didn't make the maps until now. So now, we can have temporal as well as spatial maps, such as the tracks of US tornados over the last 56 or so years ...

https://uxblog.idvsolutions.com/2012/05/tornado-tracks.html Slide 36

... or hurricanes over the last 160 or so years. This map is a particular favourite of mine as it subverts the usual mercator projection we tend to see on maps and instead takes a bottom up approach, with Antartica as the focal point, so we can see how these great storms circle around our planet.

https://uxblog.idvsolutions.com/2012/08/hurricanes-since-1851.html Slide 37

Staying with the theme of wind for a moment. This map shows the realtime wind patterns over the United States. Or at least what the wind patterns were on August 31st, when I took this screen shot. It's a nice classic example of a data map. A visualisation of wind patterns. A key of wind speeds. Nothing particularly special. Until you see the realtime aspect ...

https://hint.fm/wind/ Slide 38

... and suddenly the map comes alive. It moves and almost breathes. As with a lot of today's map visualisations, it's oddly compelling and draws you in.

Slide 39

And now, to coin a Monty Python phrase, for something completely different. We're used to seeing maps in Geradus Mercator's map projections. The first maps we see, often at school, or in an atlas at home, tend to be in this projection. It's easy to forget that this is how maps have been projected since 1569. But if you've seen any of the stunning NASA images of our planet as seen from space, you'll probably have noticed that a Mercator map doesn't look like our planet does from space. The map is distorted to fit the projection. Antarctica is this long white smear along the bottom of the map. Greenland is around 60% bigger than it really is.

There's other map projections. Buckminster Fuller's Dymaxion map is one of my favourites. It makes a bit more sense if you rotate it through 90 degrees.

https://www.bfi.org/about-bucky/buckys-big-ideas/dymaxion-world/dymaxion-map Slide 40

There's no projection distortion. It shows our planet almost as a single island in a massive ocean. There's no splitting of the continents. As Buckminster Fuller put it "the maps we use still cause humanity to appear inherently disassociated, remote, self-interestedly preoccupied with the political concept of its got to be you or me; there is not enough for both".

https://www.westnet.com/~crywalt/unfold.html Slide 41

There's just one more map to go but before I get there, I wanted to look a little bit ahead to where maps might go if you could interact with them with more than just a finger on a touch screen or a mouse or trackpad on a laptop.

What about putting a boarding pass on a map and it would show you where in the airport your gate is. What about putting a mobile phone on the map and it would show you where you could charge it. What if you could put a credit card on a map and it would show you where a bureau de change or an ATM is? MIT's Media Lab have done just this with TaPuMa, the Tangible Public Map. Maybe this is the next generation of intelligent, interactive maps?

https://www.youtube.com/watch?v=l4bz9shk8UU Slide 42

So this is the part of a talk where it's traditional to start to wrap things up and to maybe pontificate where we go from here, or what's going to be happening in a year or so's time.

Sadly, I can't do that. As more and more data becomes available, more and more people are making maps from that data in ways we can't even think of right now. All I can say is that making maps is becoming more and more democratised and while we'll always need formal and authoritative maps, we also have the ability to make our own maps and that ability is becoming easier and easier with each passing month.

Slide 43

And as an example of this democratisation of the map in action, I'll leave you with my personal favourite map, again from Stamen, called Pretty Maps, which seems to be an excellent name as this is definitely a map and it's definitely pretty. Will this be my favourite map in a year's time ... only time, maps and data will tell.

Slide 44

I hope you've enjoyed seeing these maps, as much as I've enjoyed researching them. Thanks for listening.

Photo Credits: Eva-Lotta Lamm on Flickr.

Where You Are Isn't That Interesting But Where You Will Be Is

2012-07-24 14:55

Big Brother" and "company X is tracking me" as well. But lost in the rhetoric and hyperbole around this subject is a well hidden fact ... your current location isn't actually that interesting to anyone apart from yourself.

For most of the day we tend to be on the move so even if a service does know your location that fact becomes irrelevant almost immediately. Intrusive location based advertising is normally held up for inspection here but without context a location is just a set of longitude and latitude coordinates, coordinates that are out of date and no longer relevant almost as soon as they've been detected.

Maybe a location based service I use does want to target me with location based ads, but for example, if I'm on my irregular commute from the suburbs to the centre of London on a train, I challenge anyone to find an ad, intrusive or not, that would be contextually relevant to me in sufficient detail that would warrant an advertiser paying out the not insignificant sums that such ad campaigns cost. Unless maybe, just maybe, it's an ad that offers me a viable alternative to SouthWestTrain's execrable and expensive train service, but that's just in the realms of fantasy.

Every once in a while the thorny topic of location privacy rears its ugly head, often in tandem with a new location based service or the discovery of what an existing one is really doing. There's often cries of "Big Brother" and "company X is tracking me" as well. But lost in the rhetoric and hyperbole around this subject is a well hidden fact ... your current location isn't actually that interesting to anyone apart from yourself.

For most of the day we tend to be on the move so even if a service does know your location that fact becomes irrelevant almost immediately. Intrusive location based advertising is normally held up for inspection here but without context a location is just a set of longitude and latitude coordinates, coordinates that are out of date and no longer relevant almost as soon as they've been detected.

Maybe a location based service I use does want to target me with location based ads, but for example, if I'm on my irregular commute from the suburbs to the centre of London on a train, I challenge anyone to find an ad, intrusive or not, that would be contextually relevant to me in sufficient detail that would warrant an advertiser paying out the not insignificant sums that such ad campaigns cost. Unless maybe, just maybe, it's an ad that offers me a viable alternative to SouthWestTrain's execrable and expensive train service, but that's just in the realms of fantasy.

Now it's true that if you gather enough data points you can start to infer some meaning from the resultant data set. You can probably determine the rough area where someone works and where they live based on their location at certain times of the day. But in today's connected world of the interwebs, with their social networks and uploaded photographs, that level of locational granularity can be inferred fairly easily without the need to explicitly track the location of an individual.

All of the above can be summed up as something like ...

Where you are right now isn't that interesting. Where you were is slightly more interesting. Where you will be is very interesting.

I'm sure I've said words to this effect before in a talk at a conference but try as I might I can't find a reference to back up this assertion.

What's even more interesting is that a recent research study at the UK's University of Birmingham took 200 volunteers who agreed to have their phones track them, added in the locations of their friends in their social graphs and produced an algorithm that was able to predict where a participant would be in 24 hours time, sometimes with accuracies of less than 20 meters and with an average accuracy of around 1000 meters. The full research paper makes for fascinating reading and shows that the real key to location technologies may not be where you currently are but may be much more about our predicability and daily routines for ourselves and our friends.

Now that's interesting.

Photo Credits: misspixels on Flickr.

Big (Location) Data vs. My (Location) Data

2012-07-04 14:00

Geodigital asking me if I'd like to talk at the AGI's Northern Conference and thus, after a brief bit of editing to remove the conspicuous Silicon Valley references, this talk relocated from San Francisco to Manchester. As per usual, the slide deck plus notes are below.

For a pleasant change, the guts of this talk didn't metamorphose oddly during the writing. Instead, it geolocated. This was originally planned to be my keynote talk at Social-Loco in San Francisco last month. But I wasn't able to make it to the Bay Area as planned for reasons too complex to go into here. Suffice to say, the slide deck languished unloved on my laptops hard drive, taking up 30 odd MB of storage and not really going anywhere.

Then I got an email from Stuart Mitchell at Geodigital asking me if I'd like to talk at the AGI's Northern Conference and thus, after a brief bit of editing to remove the conspicuous Silicon Valley references, this talk relocated from San Francisco to Manchester. As per usual, the slide deck plus notes are below.

[scribd id=100297709 key=key-15vmdecagp3xopiyihgt mode=list]

Slide 2

So, hello, I’m Gary. I’m a self-confessed map addict, a geo-technologist and a geographer. I’m Director of Places for Nokia’s Location and Commerce group. Prior to Nokia I led Yahoo’s Geotechnologies group in the United Kingdom. I’m a founder of the Location Forum, a co-founder of WhereCamp EU, I sit on the Council for the AGI, the UK’s Association for Geographic Information, I’m the chair of the W3G conference and I’m also a Fellow of the Royal Geographical Society.

Slide 3

There are URLs in this talk but this is the only URL in the entirety of this talk you might want to take a note of. Although if you go there right now, it'll 404 on you, later today or tomorrow, this is where this slide deck, my notes and all the links you'll be seeing will appear on my blog.

https://vtny.org/jT Slide 4

One of the things I love about writing a talk for a conference is how the things I hear and the things I read get mentally stored away and then, somehow, they start to draw together to form a semi-coherent narrative around the talk title that I inevitably gave to the conference organisers around 3 months prior. So it is with this talk, which in Sesame Street fashion, has been unknowingly brought to you by ...

Slide 5

Kellan Elliott-McCrea, previously at Flickr and Yahoo! and now at Etsy ...

Slide 6

Aaron Straup Cope, previously at Flickr and Stamen Design and now doing stuff at the Smithsonian ...

Slide 7

... and my children. No, really. This isn't just an excuse to put a photo of my family up on the screen behind me so you can all, hopefully, go "awww".

Slide 8

But before I get into anything to do with location data, big data, my data or anything interweb or social network related I want to try and frame the context of my thoughts by talking about communication, or to be more precise, the way in which we communicate. We are, politics and warfare aside, a social species and communicating with each other is something we do a lot of, although the manner in which we communicate has changed a lot.

A lot of our communication is both verbal and non-verbal and relies of face to face, person to person, proximity so that the verbal and non verbal approach comes together to express what we intend to say.

Slide 9

Some of our communication is written, the old fashioned way, using pen and paper, although a lot of commentators have called out the "death of the letter". Whether that's true or just good headline making hyperbole remains to be seen, but to be fair, I can't remember the last time I actually sat down and wrote a letter.

Slide 10

A lot of our communication is still verbal but via a phone, be that a land line or a mobile. We call and we text. A lot.

Slide 11

But be it talking face to face, texting someone or even writing an email, the intended audience is still narrow, person to person, or person to small audience.

But the interwebs have added to this sphere of communications and now we broadcast our thoughts, feelings and experiences, sometimes regardless of whether we think anyone will see this, let alone empathise or communicate back. A lot of this broadcasting has a location context, be it explicit via a geotag or implicit through mentions of a place or some other geographical construct.

Slide 12

While we still talk, meet, engage and sometimes broadcast, like I'm doing right now, this human-to-human interaction has been augmented, maybe complimented by electronic communications.

Slide 13

We're as likely to post a Tweet on Twitter or a status on Facebook or Google+ or another social network as we are to speak face to face.

Slide 14

And because this type of communiqué is electronic, that means it generates data as we go. Today we generate lots of data, big data, on a daily basis. It's probably not unfair to say that there's data being generated in this very auditorium, right now, as I'm saying this.

Slide 15

Some of this data is implicit. A by-product of what we're doing. Whether it's our cell phones loosely mapping out where we are, not a privacy invasion I hasten to add, but the simple way in which cellular networks work, but that's a topic for another talk on another day, or our GPS navigation, be it built into our car or our smartphone, providing anonymised traffic data probes to show where freeway congestion is, we don't consciously set out to generate this data. It's a by product of what we're doing.

Slide 16

But a lot of this data is very much explicit. We type out a status update on our phone, our tablet, our laptop and we tap or click on the button that says "go" or "submit" or we take a photo, maybe add an image filter or a comment and tap or click the button that says "share" or "upload".

Slide 17

By doing this we're explicitly communicating, explicitly broadcasting and sharing with our friend, family, followers and the interwebs in general ... and in doing so, we're playing our part in generating more and more data.

Slide 18

And generate it we do. Lots of it. We call it big data, but massive data would be a more accurate definition of it. Whilst our own individual contributions to big data may not be that big, when you put it all together it's part of an ever growing corpus of big data and there's companies that both provide the means for us to broadcast and share this data as well as, hopefully, providing a means of revenue for them to enable them to keep doing this. The amounts that get generated each day is almost too much for us to think about and comprehend. Once a number gets that big, we can't really deal with it. We know it's a big number but what that actually represents is hard for us to get our head around.

Slide 19

So let's look at just a small sample of what gets generated on a daily basis from the social big data, communicating, sharing and broadcasting services I tend to use, if not on a daily basis then at least on a weekly basis. I Tweet and update my Facebook status at least once a day, sometimes up to 20 times a day. I check-in to places on Foursquare at least 10 times a day and take and upload photos to Instagram and Facebook around 3 times a week. That's just my contribution, think how many people are doing the same thing to get to the sort of volumes you can see on the slide behind me.

Slide 20

But how long will this continue? Remember the people I spoke about right at the start of this talk, some 16 slides back? It's time to bring them into the picture. Firstly, my children, although this applies equally to pretty much all children. Remember when you were a child? The summer vacation was endless. The skies were always blue and the sun was always out (remember, I'm from the UK where Summer and sun do not always go together, in fact it was pouring down with rain as I wrote this at home last week). And just like the summer vacation was endless, so were your parents and the people around you, they were eternal and would always be there. Remember feeling like that? But then the inevitable happened. We grew up and we discovered, often the hard way, that the summer wasn't endless and that almost everything is finite.

Slide 21

Social networks and social location networks aren't finite either. They get born, if they're lucky they grow and then at some time or other they ... stop. If it's a social network you don't use then it doesn't really bother us much.

Slide 22

But if it's a network you've shared a lot of content through, what happens then? A lot of people, myself included, immediately get into "I want my data back" mode.

Slide 23

But is it your data. Of course it is. You made it. You composed that Tweet. You shared that link. You took that photo. You were at that place you check-in at. Of course it's your data.

But there's a point to be made here. You may have created that data, you may own that data, but the copy of that data in that social network is just that. It's a copy. It's not necessarily "your" data and because most of us don't preserve what we send up into the cloud on its way to our social networks, you may have created it, but the copy in the cloud isn't necessarily yours.

Slide 24

It's an easy mistake to make. I may be a geo-technologist and many more things besides, but I am not a lawyer, and apart from the lawyers in the room, more of you aren't and most of the people who use social networks aren't lawyers either, unless it's DeferoLaw, which is a social network for the legal profession.

Slide 25

We see phrases such as “you retain your rights” …

Slide 26

... like "you own the content posted" ...

Slide 27

... and "you always own your information" and immediately the subtleties and complexities of data ownership, licensing, copyright and intellectual property are cast aside. We say to ourselves, "it's my data dammit, I own it, I want it".

Slide 28

And it's this belief that we really are lawyers in our spare time that makes people think that somehow the data they've shared via a social network is physically theirs, rather than a bit for bit perfect copy that we've licensed to that social network. We forget for a moment that we're using that social network as a cloud based backup, in some cases the only backup, of our creations and we mutter darkly about "holding my data hostage".

Slide 29

The blunt, and often harsh reality, is the age old adage that "you get what you pay for". If you pay, you're probably a customer. If you're using something for "free" (and I say free in very large italics and inverted commas here), then you're probably, unknowingly or unwittingly, the product. Harsh. But fair. It's our content that the social networks monetize and that allows them to keep their servers and disk storage up and running. You might have seen that previous slide with the Tech Crunch post and be thinking "ah, but Flickr Pro is chargeable and if my subscription lapses I can't get my photos back". That's actually not really true, if not particularly simple, but bear with me for a few more slides.

Slide 30

Now let's forget "big data" for a moment and think about "your data" instead. Actually, let's think about "my data" for a moment. As of last week, my social media footprint on Twitter, Foursquare, Instagram and Flickr looked something like this. Facebook's numbers would be up there too, but I'll get to that in a moment.

Now in the grand scheme of things, in the massive numbers thrown about around about "big data" this is but a drop in the ocean. But ...

Slide 31

I created these check-ins, status updates, tweets and photos. They're important to me. Very important to me.

Slide 32

And as Aaron Cope pointed our earlier this year, my small, insignificant contribution to big data is part of my own, very subjective, very personal, history.

Slide 33

As I may have mentioned before, I'm a geo-technologist and a high percentage of my explicit big data contribution has a geo or location component to it. I'd like to map our where I checked-in, I'd like to see where I was when I Tweeted or what photos I took at a particular location. Some of this "mappyness" already exists in some of the big data stores where my contributions live, but not all of it, it's far too niche and personal for that. But it's still important to me.

Slide 34

Remember, in 99% of the social networks I use, I'm not the customer, I'm contributing to the product. But how do my regularly used social networks fare here. Regardless of whether I own the data I put up there, how easy is it to get a copy of?

Slide 35

Firstly, what about a one click solution? Can I go to a particular page on the web and click the big button which says "give me a copy of my data".

Slide 36

Facebook is the only one of my 5 social networks that does this. Well, it almost does this. At least I'm sure I used to be able to do this.

Slide 37

I can still request a download of my information. But it now only seems to give me my status updates since I enabled Timeline on my account, though I can still get all of my photos and messages since 2008. Rather than say that this doesn't work, I'll just file this under "needs future investigation" and move on.

Slide 38

Sometimes this lack of a one button download of contributed data is a deliberate decision on the part of a given social network. Sometimes, it's a hope that with an API, some enterprising developer will do this, but most of the time, that doesn't always happen.

Slide 39

So talking of APIs, surely the remaining social networks will have an API and let me knock up some code to get a copy of my data contributions. Surely?

Slide 40

Not all social networks do. An API tends to come after a social network's launch, if it comes at all, and often it doesn't let me do all that I want to do.

Slide 41

Thankfully, all the networks I used, with the exception of Twitter not only provide an API, but let me use that API to get my data. All of my data.

Slide 42

This is a good thing and meets the requirements for an API to meet what Kellan Elliot McCrea calls "minimal competence". He went on to say

"The ability to get out the data you put in is the bare minimum. All of it, at high fidelity, in a reasonable amount of time.

The bare minimum that you should be building, bare minimum that you should be using, and absolutely the bare minimum you should be looking for in tools you allow and encourage people who aren’t builders to use."

Slide 43

Kellan was behind Flickr's API and his sentiments are, to my mind, admirable.

Slide 44

Sadly, Twitter doesn't let me do this and fails the minimal competence test miserably. Deep in their API documentation I found the justification for this as being essential to ensure Twitter's stability and performance and leave it as an exercise to you the audience to work out what I think of this excuse.

Slide 45

The sad truth here is that when it comes to our own individual online data history, there's not always a willingness to make it easy for us to get copies of our history, if it's even on the radar at all.

Slide 46

But if we can't always get our data history back, maybe the solution is to make an archive of it before it goes in or keep that archive up to date as you go ... a personal digital archive or PDA (and not to be confused with personal electronic organisers, or PDAs, such as the Palm Pilot).

Slide 47

Thanks to web APIs and another social network, admittedly one for people who know how to code, a lot of this is already possible and the scope, range and functionality is growing by the day. The irony that I can build my own personal digital archive out of code found on another social network, which itself is built around a source code archival system is not lost on me either.

Slide 48

So, firstly, there's my own Instagram (and no, I'm not going to share the URL of where this lives I'm afraid. The idea here is that this is a personal archive, not a clone of a social network).

Slide 49

My own Instagram is called parallel-ogram. It's on GitHub; you can download it, configure it, run it. For free.

https://vtny.org/jQ Slide 50

Parallel-ogram works as well on my phone as it does on my laptop, showing me exactly what I've uploaded to Instagram. Indeed, it goes one step further than Instagram as currently there's no way to see what you've uploaded other than through their mobile app. Parallel-ogram doesn't allow me to take photos or upload them, at least not yet, but it does allow me to go back to the day I first uploaded a photo, grabs copies for me and twice a day it uses the Instagram API to see what I may have uploaded and quietly grabs a copy and stashes it away for me.

Slide 53

There's also my own archive of Foursquare ...

Slide 54

It's called privatesquare and it's also on GitHub

https://vtny.org/jR Slide 55

Like parallel-ogram, privatesquare quietly uses the Foursquare API to go back to my first check-in and twice a day quietly synchs my check-ins for me. I can go back and look at them, see maps of them and browse my check-in history. Unlike parallel-ogram, privatesquare also allows me to check-in, even if I don't want to share this with Foursquare. In short it allows me to use it both as an archive and also as a check-in tool, and if I want to use Foursquare's official mobile app, I can do that, safe and secure in the knowledge that privatesquare will keep itself up to date.

Slide 61

I take a lot of photos. Some of them go into Instagram. All of them go into Flickr. But I can archive Flickr as well.

Slide 62

It's called parallel-flickr, it also lives on GitHub and it's also filed under "something I really must install, configure and get running when I have some spare time".

https://vtny.org/jS Slide 63

So I have my own archives of Instagram, Flickr and Foursquare. I sort of have my own archive of Facebook. But what about my Tweets?

Slide 64

Well until Twitter decides that their site is stable enough to let me grab my Tweets through their archive, the next best solution is to archive by another means. I've put the RSS feed to my Tweet-stream into Google Reader, which helpfully never throws anything away. I did this a long time ago and I have almost all, but 100% all of my Tweets. Now all I need to do is write some code to read them from Google Reader and then get the Tweet data from Twitter, which then do allow via their API. Sadly, this is also filed under "something I must do when I have the time". It's not perfect, but then again, none of what I've discussed is, but it's a start and that's good enough for the time being.

Slide 65

Finally, you might have noticed the links in my slides look sort of like bitly links, only on the vtny.org domain. That's because I've been archiving my short links for a few years now

Slide 66

Using my own short URL archive and my own, self hosted, URL shortener. I just thought I'd mention that.

Slide 67

So, my big data contribution, my personal online history, is important to me. Yours might be important to you too. We're often told that we can't have our cake and eat it, but with the advent of the personal digital archive, maybe we can thanks to the enterprising people who create APIs in the first place and those who not only use these APIs but also put their code up for all the world to use, free of charge. Your online history may not be that important in the grand scheme of things, but it's your online history, it's personal, you made it. When social networks go the place where software goes to die, you might just want to preserve that personal history before the servers get powered off forever. Maybe the geeks will inherit the Earth after all.

Slide 68

Thank you for listening.

If You Live In The UK, You Need To Know About The Communications Data Bill

2012-06-19 10:36

On Thursday June 14th. 2012, Theresa May, the UK Secretary Of State published the draft Communications Data Bill. If you've been reading or watching the UK media you might well be aware of this. The bill is hugely controversial, not least because it requires all UK internet service providers to track and store for 12 months the details of every email sent within the UK, every website visited from within the UK and every use of a mobile phone within the UK. This is a huge undertaking and will gather an equally huge amount of data. It's also a costly undertaking, one that is ill conceived and impractical, one that is a massive invasion of our personal privacy and right to communicate with each other and one that is fundamentally undemocratic.

It's costly because the estimated price tag is £1.8bn over 10 years, a price tag that the country cannot afford given the current economic climate and the austerity measures which are being applied across all aspects of the United Kingdom. The estimated price tag is also just that, an estimate and the UK Home Office has already stated that the final figure is likely to be much higher.

It's ill conceived and impractical because the data collection and monitoring will be bypassed by those that the bill seeks to target; the terrorist, the paedophile and the organised criminal. As Conservative MP David Davis said recently

"The only people who will avoid this, avoid being covered by this, are the actual criminals because they are always around this. You use a pre-paid phone, you use an internet cafe to hack into somebody's wi-fi. You use what is (sic) called proxy servers, and those are just the easy ways. There are harder ways too and you know, actually, the 7/7 bombers went round it. Organised criminals go round it. Organised pedophile rings go round it. What this will catch is the innocent and the incompetent"

It's an invasion of privacy because the much maligned Human Rights act dictates that our right to a private life must be respected by the Government. Though Theresa May has been clear to point out that what will be recorded and monitored is the end points of our communications, email address, phone numbers, web URLs, not the content of those communications, you can build up an incredibly accurate picture of an individual's life, activities and movements without the need to see the content of communications. It's also an invasion of privacy because examination of this data would be able to be undertaken without the need for a warrant and thus for fair and impartial scrutiny.

It's undemocratic because already Theresa May has branded anyone who criticises this bill as a "conspiracy theorist", using the justification of "if you have nothing to hide you have nothing to fear". This criticism is not only offensive to those who would hold our government to scrutiny, but it flies in the face of existing evidence of leaks and abuse of personal data that the government and its agencies already hold on us. It also is a direct reversal of the coalition government's pre-election stance, when David Cameron, now the UK Prime Minister said "If we want to stop the state controlling us, we must confront this surveillance state”.

I'm not alone in thinking this. The UK's Guardian calls it an "online snooping scheme", InfoSecurity notes that the bill is "more intrusive than anything anywhere outside of China". Big Brother Watch says "we are all suspects now" and Big Brother Watch and Liberty comments that "it won’t matter if you have never got so much as a speeding fine, personal information about you will be stored just in case it may prove useful one day".

What you can do and what you should do is protest against this bill. Let your MP know that you're deeply concerned about this bill. Sign the 38 Degrees online petition against the bill. I don't consider myself politically active but I've done just this because I value my freedoms and my privacy. I think you should too. You can also draw attention to this through your website as I've done here on my blog and I've knocked up a quick and dirty WordPress plugin to do just that.

Now The Metropolitan Police Want Your Phone's Data

2012-05-21 14:57

As a relatively prolific user of social networks and social media I generate a fair amount of data. Whilst I'm wary of what the social networks do with the data I generate, I appreciate that there's no such thing as a free lunch and the data I generate contributes towards the revenue that keeps these services alive. There's an uneasy tension that exists between big data and my data. I applaud services which allow me to retain or get back the data I put into them; Facebook, I'm looking at you here. I frown in a disapproving manner at services that make it challenging to get my data back without recourse to some coding; Foursquare and Flickr, I'm looking at you here. I'm quietly furious, yet continue to use services which are valuable to me but make it downright impossible to get my data back; Twitter, I'm fixing you with my steely gaze here.

This is all data that I willingly generate and contribute. But I'm increasingly wary about data which is not willingly generated or contributed. The data that private corporations hold on me, such as credit ratings agencies and more and more, the data that my government and their agencies hold on me, that I either haven't willingly consented to or that is generated or aggregated without my knowledge.

It now seems that I need to add the police force of the city in which I live to the growing list of government agencies I'm wary of. As the BBC reports ...

The Metropolitan Police has implemented a system to extract mobile phone data from suspects held in custody.

The data includes call history, texts and contacts, and the BBC has learned that it will be retained regardless of whether any charges are brought.

What? Seriously? Really?

I can accept that if a crime has been committed, there's a strong argument for getting access to data on a mobile phone, if it's done with the correct authorisation and if it's needed in order to achieve a conviction. But keeping the data, regardless of whether charges are brought or not has to be a breach of privacy. That breach isn't just of the individual concerned, but of all the contact information for individuals that are on a phone and for the company who employs the suspect, who now has their privacy breached. Whilst history of calls, texts and contacts are mentioned, I fully expect the information obtained to cover email, work and personal email, as well, which would be even more cause for concern for companies in this country.

I'm sure the standard nothing to hide, nothing to fear adage will be rolled out to mollify concerns over this and we'll be told that we can trust our police force with this information that they hold. After all, our police officers would never illegally access information that they hold, just like our civil servants would never snoop on the private health and financial information that the government holds ... would they? Photo Credits: Steven Guzzardi on Flickr.

"Disk Utility Can't Repair This Disk"

2011-08-01 06:38

Quis backup ipsos backups?", as the Roman poet Juvenal didn't say but might have if they had had computers in the first century AD.

Like most geeks I pride myself on being able to maintain the computers I use on a daily basis. Just like real men don't eat quiche and real programmers don't use Pascal, real geeks don't call for professional help or technical support.

But then the day comes when one of your hard drives goes crunk, you go through all the tricks of the trade you know, you exhaust searching for possible solutions on the web and you realise that maybe, just maybe, while it's not time to eat quiche or starting coding in pascal, it's probably time to call for some professional help.

Like a lot of people, I've amassed a not inconsiderable amount of digital media over the years, in the form of apps, songs, movies and photos. Most of these live on my laptop and are religiously backed up with SuperDuper! and with Time Machine to external drives, with one of these drives holding the overspill. This aforementioned external drive had given solid, reliable service over the years but had started to act ... quirkily. Fearing a critical mass of bad sectors I decided now was a good time to backup my backups.

"Quis backup ipsos backups?", as the Roman poet Juvenal didn't say but might have if they had had computers in the first century AD.

Like most geeks I pride myself on being able to maintain the computers I use on a daily basis. Just like real men don't eat quiche and real programmers don't use Pascal, real geeks don't call for professional help or technical support.

But then the day comes when one of your hard drives goes crunk, you go through all the tricks of the trade you know, you exhaust searching for possible solutions on the web and you realise that maybe, just maybe, while it's not time to eat quiche or starting coding in pascal, it's probably time to call for some professional help.

Like a lot of people, I've amassed a not inconsiderable amount of digital media over the years, in the form of apps, songs, movies and photos. Most of these live on my laptop and are religiously backed up with SuperDuper! and with Time Machine to external drives, with one of these drives holding the overspill. This aforementioned external drive had given solid, reliable service over the years but had started to act ... quirkily. Fearing a critical mass of bad sectors I decided now was a good time to backup my backups.

And then it happened. Crunk. The disk died. So I fired up OS X's Disk Utility and verified the disk. It had ... issues. Time to repair the disk. So it chugged and it whirred and the progress bar progressed with glacial slowness until finally, several hours later, I saw the message I dreaded.

Disk Utility can't repair this disk. Back up as many of your files as possible, reformat the disk and restore your backed-up files.

Of course, it was probably my fault. Despite the number of bad sectors and other magnetic media glitches that accumulate over time on a disk drive, the drive itself had still been functioning; probably because I'd never actually tried to read from one of those bad patches recently. But in trying to backup the drive, I was pretty much accessing every sector on the drive with the resulting crunk being pretty inevitable.

So what to do? Most of my photos were already hosted on Flickr. A lot, but by no means all, of my music could theoretically be re-ripped from CD. But my backups of my iPhone and iPad were gone and let's not even begin to talk about the movies. It may only have been under 500 GB's worth of data, which is a drop in the ocean compared to today's multiple terabyte drives, but it was a lot of data to me and it represented a lot of time, effort and memories.

Maybe data recovery was possible? A quick online search for "mac data recovery" had my bank balance wincing in shock. This was going to be expensive, if it was possible at all. Most recovery firms charged to look at the drive and then charged to extract the data from the drive, with pricing being based on the number of files, not the capacity of the drive. Then I found Tierra Data Recovery. Fixed pricing, free analysis of whether the data could be recovered, free courier collection and payment only on successful recovery.

It seemed too good to be true. But a quick phone call, explaining the situation and Tom from Tierra, as he will now be known, calmly laid out my options. So the following day a courier collected my drive and took it to Scotland and a couple of days later I got an email from Tierra with the news that all of my data could be recovered for a little over £300.00, and after shipping a new drive to them, all of my data made its way from Scotland back to London.

Here in the UK we've become accustomed to being gouged by companies, to expecting poor or no customer service and to be treated like a cash cow. Which makes the speed and quality of the service provided by Tom and Gill at Tierra all the more unexpected and pleasing. I hope I never need the services of a data recovery company again, but if I ever do, Tom from Tierra will be getting my business again without a second thought. If you find yourself in this unenviable position, you should give Tom a call too. Photo Credits: ~inky and Sifter on Flickr.

Costa Rica And Nicaragua; A Border Dispute In The Age Of Web Maps

2010-11-08 14:01

middle of rivers or even, in extreme cases, almost driving off of the edge of a cliff.

But the strangest example of this sort of behaviour was in the first reports of recent events on the border of Costa Rica and Nicaragua that seemed to implicate Google Maps as justification for Nicaraguan troops crossing the border into Costa Rica and raising the Nicaraguan flag on Costa Rican territory. The dispute seems to hark back to the 1850's where the contested border between the two countries followed the course of the San Juan River, the course of which has since moved somewhat, as rivers are wont to do. Costa Rica asserts their sovereignty on the disputed land based on the 1850's arbitrated border which follows the course of the river and Nicaragua asserts theirs based on the fact that the river has moved so some land must be theirs.

The popular press and media likes nothing better to poke fun at people who seem to ignore their own senses and instead rely on their GPS sat-nav systems, which frequently results in people ending up in the middle of fields, in the middle of rivers or even, in extreme cases, almost driving off of the edge of a cliff.

But the strangest example of this sort of behaviour was in the first reports of recent events on the border of Costa Rica and Nicaragua that seemed to implicate Google Maps as justification for Nicaraguan troops crossing the border into Costa Rica and raising the Nicaraguan flag on Costa Rican territory. The dispute seems to hark back to the 1850's where the contested border between the two countries followed the course of the San Juan River, the course of which has since moved somewhat, as rivers are wont to do. Costa Rica asserts their sovereignty on the disputed land based on the 1850's arbitrated border which follows the course of the river and Nicaragua asserts theirs based on the fact that the river has moved so some land must be theirs.

The reference to Google Maps turns out to be a bit of a red herring as well, originating from an opportunistic sound bite rather than fact. Granted Google have based their data set on admittedly sparse data, some of it originating from the US State Department, which had got it wrong. But other mapping data vendors, who should know better and who at the time were having a great laugh at Google's expense on various forms of social media, turn out to be just as incorrect as Google's.

While this is probably the most extreme example of "but I found it on the internet so it must be true", the whole story is less about whose map is right, less about blaming map error on an online map and more about how some parts of the world are less well mapped than others. Not all map data is created equal.

The twists and turns of the story are best followed on the original post from Jonathan Crowe's excellent The Map Room blog and its follow up as well as an in-depth article on the subject from Ogle Earth.

Talking GeoBabel In Three Cities (And Then Retiring It)

2010-10-08 15:48

can I adapt, cannibalise or repurpose one of my other talks?". This sometimes works. If there's a theme which you haven't fully worked through it can serve you well.

But a conference audience is an odd beast; a percentage of which will be "the usual suspects". They've seen you talk before, maybe a few times. The usual suspects also tend to hang out on the conference Twitter back channel. Woe betide if you recycle a talk or even some slides too many times; comments such as "I'm sure I've seen that slide before" start to crop up. Far better to come up with new and fresh material each time.

You're invited to speak at a conference. Great. The organisers want a talk title and abstract and they want it pretty much immediately. Not so great; mind goes blank; what shall I talk about; help! With this in mind, my first thought is normally "can I adapt, cannibalise or repurpose one of my other talks?". This sometimes works. If there's a theme which you haven't fully worked through it can serve you well.

But a conference audience is an odd beast; a percentage of which will be "the usual suspects". They've seen you talk before, maybe a few times. The usual suspects also tend to hang out on the conference Twitter back channel. Woe betide if you recycle a talk or even some slides too many times; comments such as "I'm sure I've seen that slide before" start to crop up. Far better to come up with new and fresh material each time.

But sometimes you can get away with it and so it was with my theme of GeoBabel. Three conferences: the Society of Cartographers Summer School, The Location Business Summit USA, AGI GeoCommunity 2010. Three cities: Manchester, San Jose, Stratford-upon-Avon. Three audiences: cartographers, Silicon Valley geo-location business types, UK GIS business types.

I've written about GeoBabel before; it's the problem the location industry faces as we build more and more data sets which are fundamentally incompatible with each other. This incompatibility arises either due to differing unique geographic identifiers, where Heathrow Airport, for example, is found in each data set, with differing metadata and a different identifier, or due to different licensing schemes which don't allow data to be co-mingled. We now have more geographic data than before but each data set is locked away in its own silo, either intentionally or through misguided attempts to be open.

The slide deck, embedded above, is the one I used in San Jose. The ones for Manchester and for Stratford-upon-Avon are pretty much identical but are on SlideShare as well.

As another way of illustrating the problems of GeoBabel, I came up with what I've termed The Four Horseman Of The Geopocalypse. All very fin de siecle but it seemed to be understood and liked by the audience at each talk.

The first Horseman is not Pestilence but Data Silos. All of the different types of geographic data we have, international and national commercial data, national and crowd sourced open data, specialist and niche data and social network crowd sourced data each live in isolation to each other with the only common denominator being the geo-coordinates each data set's idea of a place has.

The second Horseman is not War but Licensing. Nowadays in the Web 2.0 community we're used to having access to data but we're not willing to pay for it. Licenses vary between closed commercial licenses and open licensing. But even in the open license world there are silos, with well meaning licenses becoming viral and attaching themselves to any derived work.

Which segues neatly to the third Horseman, who's not Famine but Derivation. Each time you create something from data, you're deriving a new work in the eyes of most licenses and that means the derived work often has the original license still attached to it. You do the work, but you don't own the work.

Finally, the fourth Horseman is not Death but Co-Mingling. There is no one single authoritative geographic data set, you need to find the ones which work for you and for your business or use case. That means you need to mingle the data sets and frequently the licenses you have for those data sets explicitly prohibit this.

But now after three outings, it's time to retire GeoBabel, for now at least, just as I retired my Theory Of Stuff earlier this year. That means I had to find a new theme to talk about at my next event, the Geospatial Specialist Group at the British Computer Society. But that's in my next post.

Photo Credits: Nick. J. Webb on Flickr.

Knocking Down (Geo Data's) Brick Walls

2010-08-10 16:33

Location Business Summit in San Jose. The interview is now up on the GoMo News site and is reproduced here with permission.

Earlier this week I was interviewed by Cian O'Sullivan for GoMo News as part of the run-up to the Location Business Summit in San Jose. The interview is now up on the GoMo News site and is reproduced here with permission.

Ovi Places: Mobile Navigation needs to knock down its brick walls

When Ovi Maps launched at the start of this year, it really shook up the navigation industry. The free software gave everyone with access to Nokia's Ovi Store a perfectly serviceable Personal Navigation Device (PND), completely for free. But Ovi Maps is just the first exposure of the Nokia branch called Ovi Places. Recently appointed Director of Ovi Places, Gary Gale, took some time to talk to GoMo News about the state of mobile navigation ahead of his appearance at the Location Business Summit, USA, 14-15 September, San Jose. Most people know about Ovi Maps, but a lot won't have heard about Ovi Places. What is it, exactly?

It's the slightly unglamorous name for a set of back-end systems that understand what people are looking for. Within the Ovi Maps client, on both mobile and internet, there's the ability to look for what the industry calls Points Of Interest - or POIs. But we prefer the term "places" - because POIs comes laden with preconceived baggage. Our colleagues in Japan consider anything that isn't nailed down as a POI, including bus stops, park benches or traffic lights. That can lead to too-much data, an overflow that can't be easily consumed. People tend to think of these kind of location and navigation services as a yellow pages business listings - which is certainly important for the classic LBS model of "where am I, and what's around me". But Ovi Places takes into account local information, colloquial information, landmarks and places you'd want to go to as a tourist. For example, where I am in the Nokia office in the middle of Berlin, we've got the really common tourist POIs showing up - like the Brandenburg Gate, for example - but Places also refers to an excellent restaurant in the courtyard below me, and a local coffee shop.

Where do you source that info? Are there Places fact finders or do you buy the info?

It comes from a variety of sources. Some of it comes from commercial data providers - this is actually one of the main reasons we acquired NAVTEQ, and why TomTom bought TeleAtlas. Digital mapping companies have a rich set of data above and beyond the normal PND stuff. But there are also a whole variety of specialist premium partners that we do deals with; we're talking about regional specialists that we talk to on a country-by-country basis in order to gain their local insight.

There is no "one true" source of data - you need to make a lot of partnerships to get the best local data available. At the moment, Ovi Places really only powers the Ovi Maps application. Are there plans for more services to exist under a Places umbrella?

At the moment, it's exposed only through Ovi Maps. For the future... I can't say anything specific, but watch this space! How do you plan to make mobile location more personal to the mobile user?

Actually, the mobile user is probably the easiest use case for navigation. Your device has lot of options available to it to determine your location. From there, services like Places can provide rich experiences. The key problem is whilst all of this is pretty much mainstream now, there is a "Bay Area bubble" where a lot of the products and services coming out seem to think your user will always have a smartphone, and will always have a GPS lock with an excellent data connection. That may be fine for San Francisco, and even Western Europe. Sometimes even areas you think would be well served are awful. I recently went on a trip to Calais - when I got off the ferry and the GPS took 15 mins to pick up a lock. So you have to realise that there can be patchy 3G data coverage in even highly developed countries, and then look at areas which have growing economies and even worse connections. There are places in Africa and Asia that won't have 3G data in the next 5 or 10 years. You mentioned that mobile users are the easy use cases - what would you consider to be a challenging case?

The challenges arise when you've got infrastructure problems. Consider some of the poster child location services, like Foursquare, Gowalla and Yelp. Lack of 3G data infrastructure doesn't appear to be factored into the business models for these companies. Try using one of them in Africa, or India, or Asia. The infrastructure isn't there to address these needs. The populace simply don't have access to these services. Is Places doing anything to address that problem?

We're looking at potential handsets that don't need a dedicated on-board GPS or AGPS. They don't need the typical app store economy. We're able to tap into cell tower triangulation, where local laws and legislation permits it. It may not be as accurate as a GPS lock, but it's better than nothing. Is that really important for a developing country? How worried is a resident really going to be about their location services.

I think the best answer to that is from an article by Dr. Tero Ojanperä (Executive Vice President of Services, Mobile Solutions, Nokia). He said that the target is less about producing a device that runs apps than it is about creating a really useful platform - it's more about producing a context-aware device, that gives you the best relevancy depending on the services available to it. "It's about devices that offer truly connected services and learn your habits so well that they can give you what you want". That means you have a service that will provide good services to every customer, no matter what the state of their local infrastructure is.

Last month I was at the GeoLoco conference in San Francisco, talking on a panel about the challenges the industry is facing. An audience member asked "what advice would the panellists give to someone who is trying to establish a foothold in location?" I felt my answer got the most responses, at least on the Twitter back-channel. which was "I come from Europe - don't forget that we exist! There is a market outside of North America that is different in its needs and infrastructure". Services like TeleAtlas and OpenStreetMap (OSM) make a lot of use of crowd-sourced info. Does Ovi Places allow for that?

Very much so. We already have this kind of functionality built into the newer handsets, allowing you to add corrections and updates while you are on location. Crowd sourcing is very much a part of this industry's future - but I don't think it's the panacea that people think it might be. It's a vital additional source, but not the best thing since sliced bread until; at least, not until the industry gets together and comes up with a way to verify and editorialise new info. It's a benevolent technological anarchy - because there's no formalised control over how you tag a place, a consumer has to accept that finding out how to use the data will take significant time and revenue investment. If your local authority is trying to map its assets, you want to make sure those assets are exactly where you claim - because taxation and revenue streams can be assessed on that. If you get that wrong, it will lead to the kind of bad press a local authority doesn't want. Especially if emergency services are trying to get to a specific street address - you need that data to be 100% accurate. What do you think the main challenges facing mobile navigation are?

I think there two main challenges.

First is the privacy angle. People don't quite understand what it is that they're giving up to use the latest LBS app. You need to make sure that people understand the value proposition on the table when they're giving up their location to gain relevance in their local search. The public as a whole needs to understand this. And it will probably be driven by tabloid headlines - some celebrity who gets divorced because a location service proves they weren't where they said they were. And it would be better if it didn't happen that way. I hope the Industry is open and transparent about it as much as possible. It will be to our detriment if we don't expose this kind of information, and something sensationalist does happen.

Second, there's a need for people to talk to one another. We're all building loads of very rich data sets - OSM is doing it, Facebook, Foursquare, government services, NAVTEQ - but at the moment, to unlock their potential, they need to talk to each other. The current licensing set up means location data is still stored in a series of vertical silos which aren't allowed to work with each other. And the actual industry moves so fast that even those who are involved in it find it hard to keep up with developments. So keeping the legal and licensing system up-to-date with it must be nightmarish. It's getting increasingly more difficult to get solid patents in this area - and patents being wielded by the patent troll houses are being used in a way they were never intended. In order to work around this, I think the future will have to be less about aggregating these data silos, and more about synchronising the end-point exposure. If you have an identifier in one data set that corresponds to an identifier in another data set, they can sync up and present a united service to the end user... without having to share protected data.

Gary Gale will be speaking at the Location Business Summit, 14-15 September, San Jose, where he'll be further addressing the issues surrounding the "silo problem" and licensing issues. Photo Credits: William Warby and Ajith Kumar on Flickr.

Service Suspended On The London Underground (API)

2010-07-02 15:44

Transport For London Tube API, the London Datastore blog sadly notes:

Owing to overwhelming demand by apps that use the service, the London Underground feed has had to be temporarily suspended. We hope to restore the service as soon as possible but this may take some days. We will keep everyone informed of progress towards a resolution.

In the meantime, if you want to see how it does looks when the API is up and running there's a video clip of Matthew Somerville's recent Science Day hack visualisation over on my Flickr photo and video stream.

Photo Credits: Martin Deutch on Flickr.

If you build it they will come. Or to put it another way, sometimes demand outstrips supply. After the phenomenal success of the Transport For London Tube API, the London Datastore blog sadly notes:

Owing to overwhelming demand by apps that use the service, the London Underground feed has had to be temporarily suspended. We hope to restore the service as soon as possible but this may take some days. We will keep everyone informed of progress towards a resolution.

In the meantime, if you want to see how it does looks when the API is up and running there's a video clip of Matthew Somerville's recent Science Day hack visualisation over on my Flickr photo and video stream.

Photo Credits: Martin Deutch on Flickr.

Where's My Tube Train? Ah, There's My Tube Train

2010-06-24 20:52

I wrote about Paul Clarke trying to solve the problem of where's my train; that there must be a definitive, raw source of real-time (train) information and that

I assert that train operators know where their assets are; it would be irresponsible if they didn't

Whilst the plethora of train operators that fragmented from the ashes of the old British Rail network haven't answered this challenge yet, Transport for London has, opening up just such data as part of the London Datastore API. In today's age of talented web mashup developers, if you release an API people will build things with it if the information is useful and interesting and that's just what Matthew Somerville of MySociety did at the recent Science Hack Day ... a (near) realtime map of the London Underground showing the movement of trains of all of the Tube lines. A screen grab wouldn't do it justice and it takes a while to load, so a video grab might help here.

Back in December of 2009, I wrote about Paul Clarke trying to solve the problem of where's my train; that there must be a definitive, raw source of real-time (train) information and that

I assert that train operators know where their assets are; it would be irresponsible if they didn't

Whilst the plethora of train operators that fragmented from the ashes of the old British Rail network haven't answered this challenge yet, Transport for London has, opening up just such data as part of the London Datastore API. In today's age of talented web mashup developers, if you release an API people will build things with it if the information is useful and interesting and that's just what Matthew Somerville of MySociety did at the recent Science Hack Day ... a (near) realtime map of the London Underground showing the movement of trains of all of the Tube lines. A screen grab wouldn't do it justice and it takes a while to load, so a video grab might help here.

Coming down the escalators at Waterloo and want to know whether to head for the Bakerloo or the Northern Line to take you north of the river? Now you can tell which line has a northbound train closest to Waterloo.

Want to see just how close the gap is between Leicester Square and Covent Garden on the Piccadilly Line really is? Now you can.

Of course, this doesn't solve every problem ... 1. If you're on the escalators at Waterloo how do you get 3G data coverage to view this mashup on your phone as Transport for London still haven't manage to achieve cellular coverage underground, unlike Amsterdam, Berlin and other cities? 2. The site will probably be the target of a tutting campaign from the Health and Safely police insisting that such a visualisation will cause people to run for the train and of course, they might trip and hurt themselves. 3. If you're at the top of the escalator and the train is in the station, now, right this very minute now, how do you get down to the platforms quickly?

Whilst I can't answer the first two of these questions, this publicity stunt from Volkswagon at Berlin's Alexanderplatz U-Bahn station might just hold the solution for the third question ... a slide!

When Maps and Data Collide They Produce ... Art?

2010-06-01 19:45

It's instead an image from the Geotagger's World Atlas but it's still unintentionally beautiful.

Last month I wrote that a map says as much about the fears, hopes, dreams and prejudices of its target audience as it does about the relationship of places on the surface of the Earth. With the benefit of hindsight I think I was only half way right.

Sometimes a map becomes more than just a spatial representation and becomes something else.

Sometimes a data visualisation becomes more than just the underlying data and almost takes on a life of its own.

When these two things meet or collide the results can be spectacularly compelling and produce, unintentionally ... art? Look at the image below ... filigree lace work? Crochet for the deranged of mind? Silk for the sociopath? Macrame for the mad? Sadly none of the above.

It's instead an image from the Geotagger's World Atlas but it's still unintentionally beautiful.

The maps are ordered by the number of pictures taken in the central cluster of each one. This is a little unfair to aggressively polycentric cities like Tokyo and Los Angeles, which probably get lower placement than they really deserve because there are gaps where no one took any pictures. The central cluster of each map is not necessarily in the center of each image, because the image bounds are chosen to include as many geotagged locations as possible near the central cluster. All the maps are to the same scale, chosen to be just large enough for the central New York cluster to fit. The photo locations come from the public Flickr and Picasa search APIs.

I could look and stare at the all the images in Eric's Flickr set for hours. Correction, I have stared at the images for hours. Photo Credits: Eric Fischer on Flickr.

Reaching The Limits Of Unlimited

2010-05-06 09:34

Consider for a moment the word unlimited; it's an adjective and, if you'll pardon the condescension, it means the following:

not limited; unrestricted; unconfined
boundless; infinite; vast
without any qualification or exception; unconditional

Except in the world of mobile data or mobile broadband, where unlimited means, in a vaguely disturbing twisted, inverted, doublespeak sort of way, the exact opposite.

Vodafone, my current UK mobile provider, helpful tells me that I have unlimited data, subject to their fair use policy which promptly redefines unlimited as very much limited indeed and your limit is 5GB per month. That's a lot of data. Even being the compulsive photo uploader, web browser, Foursquare and Gowalla check in, Twitter and Facebook poster and checker that I am, I'm hard pressed to go above 500MB per month let alone 5GB.

So I was both vastly amused and somewhat shocked when this text arrived on my iPhone on the way home from work last night.

A quick call to Vodafone luckily cleared this up as being a glitch in their billing systems and I would not, as stated be charged, nor had I gotten anywhere near the 5GB limit of unlimited.

I found the whole process rather amusing in hindsight but shouldn't the mobile companies either come clean about what unlimited really means or just don't sell unlimited data as a concept at all and just sell a, 5GB in my case, data limit?

Reclaim and Own Your Short URLs

2010-03-03 11:05

bit.ly and tinyurl.com. These free services take a long URL such as this post - /2010/03/03/reclaim-and-own-your-short-urls - and compresses them down to a much more manageable shorterned version - https://bit.ly/aG1RBx or https://tinyurl.com/ylaodny.

There are many reasons to like the use of URL shorteners such as bit.ly and tinyurl.com. These free services take a long URL such as this post - /2010/03/03/reclaim-and-own-your-short-urls - and compresses them down to a much more manageable shorterned version - https://bit.ly/aG1RBx or https://tinyurl.com/ylaodny.

They increase link sharing; the vast majority of social networking sites use 140 characters as the maximum size for an update, using the full version of a URL you're sharing reduces the amount of space for you to put your own thoughts into the update. Just compare the full URL /2010/03/03/reclaim-and-own-your-short-urls at 65 characters against https://bit.ly/aG1RBx at 21 characters.

They can track and yield click and referrer information; the information that bit.ly provides is so useful, showing live clicks, geographic and referrer information amongst others.

But almost a year ago, Delicious founder and ex-Yahoo! Joshua Schachter made some pretty compelling arguments against short URLs:

The worst problem is that shortening services add another layer of indirection to an already creaky system. A regular hyperlink implicates a browser, its DNS resolver, the publisher's DNS server, and the publisher's website. With a shortening service, you're adding something that acts like a third DNS resolver.

...

But the biggest burden falls on the clicker, the person who follows the links. The extra layer of indirection slows down browsing with additional DNS lookups and server hits. A new and potentially unreliable middleman now sits between the link and its destination. And the long-term archivability of the hyperlink now depends on the health of a third party.

Or to put it another way, you no longer own your links or the data clicks that those links yield. If the service dies, your links break, pure and simple, and that does happen, as the demise of the original tr.im and cli.gs services show.

But there is a way to take all the benefit that short URLs offer and keep ownership of your links and all the data that clicks on those links will give you and that's to run your own URL shortening service, which is precisely what I've done with vtny.org which is running the YOURLS code behind the scenes. This gives me all the benefits and metrics that other URL shorteners provide but with the added and crucial benefit that I now own the links and the data they generate, in this case via the vtny.org/4 short URL.

Photo credit: playerx and revrev on Flickr

Geographic and Transport Data; a Tale of Capricousness, Whimsy and Downright Insanity

2009-12-11 23:38

there's no such thing as a free lunch". So stuff that costs is good and stuff that's free isn't. But normal rules don't apply here.

The industry I work in thrives on data; we consume loads of the stuff and in turn we generate petabytes of it. I'm talking about data in general, not the geographic, mapping or place data that I usually write about.But the longer I work in the Internet industry the more convinced I become that, as an industry, we need to get our act together. How else to explain the bizarre, rapidly changing and capricious nature of how we gain access to, use, pay, don't pay and disseminate data?We're socially conditioned to assume that free does not equate to good, hence the adage "there's no such thing as a free lunch". So stuff that costs is good and stuff that's free isn't. But normal rules don't apply here.

Let's take geographic data; I'm on home ground here so this should be relatively straightforward.The proprietary data vendors, Navteq, TeleAtlas and others, charge for their data and limit what you can and can't do with it. OpenStreetMap on the other hand charges nothing for its' data and only places limits on the data to protect the data by way of the Creative Commons Attribution Share Alike license.So naturally the data you pay for should be good and the data you don't pay for should be ... less than good. Naturally.Except OpenStreetMap data isn't less than good. UCL's Muki Haklay summed this up neatly as "How good is OpenStreetMap? Good enough" at the OpenStreetMap conference in Amsterdam this year. Conversely, the proprietary data vendors don't always get it right. One data vendor, who will remain anonymous, shipped a release of data with wildly incorrect centroids, the lat/long coordinate which represents the nominal centre of a place, which meant that amongst others, Covent Garden ended up being centred on Holborn Underground Station. This isn't an isolated incident.On the one hand, the City of Vancouver in British Columbia makes its data, all of its data, free and open. On the other hand, the City of Tempe in Arizona decides to charge a "fair approximation of market value" for its data, which as James Fee recently discovered means that you'll need to cough up $100,000 to use it commercially.In San Francisco, BART, the Bay Area Rapid Transit, makes their data which includes train times freely available and taking a refreshingly prosaic approach to accessibility and licensing.Getting an API key: Psyche: you don't need one. We're opting for "open" without a lot of strings attached. Just follow our simple License Agreement, give our customers good information and don't hog resources. If that doesn't work for you, we can certainly manage usage with keys and write more terms and conditions. But who wants that?Here in the UK TFL, Transport for London, give you some data for free but not the train times and for overground trains the Association of Train Operating Companies (pdf link) value this data at a staggering £27,430 per yearAnd elsewhere in the world, other operators are closing down people who want to use this data, in New York, in Berlin, in New South Wales and we can't really seem to work out who owns the data and whether there's intellectual property being infringed or a public service being undertaken.... and don't even talk about the British postal code data was closed, was then going to be opened up but now isn't. Apparently.With all the data we consume and emit, we spend a lot of time and effort evangelising APIs and web services that use it. But as an industry we really need to start to act clearly and consistently in order to be taken seriously and in order for the Internet industry to realise the potential that we all think it's capable of. Posted via email from Gary's Posterous

The (Geo) Data Dichotomy Dilemma

2009-11-16 18:30

FreeOurData.org.uk and other pleas, before the Internet itself, things used to be so much simpler for geo data. You were either an end user and accessed the data as a map or you were a GIS Professional and accessed the data via a (frequently very expensive and very specialised) Geographical Information System. But now we have geo data, lots of geo data, some of it free, some of it far from free, both in terms of usage and cost and a fundamental problem has replaced the paucity of data.

Everyone wants free, open, high quality geo data and no one wants to pay for it. But it's not quite that simple. The recent acquisitions of Tele Atlas and Navteq, the two big global geo data providers, by TomTom and Nokia respectively show the inherent value in owning data. But owning the data isn't enough any more as the market for licensing the data is a shrinking one, despite the phenomenal growth of the satnav market, both in car and on mobile handsets. Why is the market shrinking? Because no one wants to pay for it, at least directly. TomTom, primarily a hardware vendor, are differentiating into the software and data market, seems to be concentrating on the PND usage of the data, although we've yet to see how the outlay necessary to acquire Tele Atlas coupled with the overall economic downturn will effect their overall 2009 earnings. Their Q1 2009 report somewhat dryly notes that "market conditions were challenging" and that "we are making clear progress with the transformation of Tele Atlas into a focused business to business digital content and services production company". There may be other aspirations at play here but for now at least, the company is keeping quiet.

Before Web 2.0, before mashups, before FreeOurData.org.uk and other pleas, before the Internet itself, things used to be so much simpler for geo data. You were either an end user and accessed the data as a map or you were a GIS Professional and accessed the data via a (frequently very expensive and very specialised) Geographical Information System. But now we have geo data, lots of geo data, some of it free, some of it far from free, both in terms of usage and cost and a fundamental problem has replaced the paucity of data.

Everyone wants free, open, high quality geo data and no one wants to pay for it. But it's not quite that simple. The recent acquisitions of Tele Atlas and Navteq, the two big global geo data providers, by TomTom and Nokia respectively show the inherent value in owning data. But owning the data isn't enough any more as the market for licensing the data is a shrinking one, despite the phenomenal growth of the satnav market, both in car and on mobile handsets. Why is the market shrinking? Because no one wants to pay for it, at least directly. TomTom, primarily a hardware vendor, are differentiating into the software and data market, seems to be concentrating on the PND usage of the data, although we've yet to see how the outlay necessary to acquire Tele Atlas coupled with the overall economic downturn will effect their overall 2009 earnings. Their Q1 2009 report somewhat dryly notes that "market conditions were challenging" and that "we are making clear progress with the transformation of Tele Atlas into a focused business to business digital content and services production company". There may be other aspirations at play here but for now at least, the company is keeping quiet.

Nokia, also primarily a hardware vendor in the form of mobile and cellular handsets, are also moving away from their roots and into a wider market, hopefully in an attempt to stop the encroachment of upstarts such as HTC, Apple and RIM into Nokia's traditionally strong smartphone heartland. Again, Nokia has yet to make a public play into this arena but all the composite elements are in place to enable this to happen. Taking the opposite route, Google, which started off as a software player are now moving to being a player in the data market by gathering high quality geo and mapping data under the smokescreen of gathering Street View. This has allowed them to gather sufficient data to supplant Tele Atlas as a data provider, at least in the Continental United States. All three companies are either making or have the prospect of making determined plays in the location space but all three of them have ways of leveraging the value inherent in their data. Google has their unique users, their search index and a vast amount of advertising inventory; TomTom their satnav customers; Nokia their handset customers, albeit one level removed with the Mobile Network Operators as an uneasy partner and intermediary. So what of the open data providers? It's important to remember here that open doesn't always mean free, it means the ability to create derived works and to use the data in ways that the originator may not have immediately foreseen. True, a lot of open data is free, but even then it's the Free Software Foundation's definition of the word. "Free (software) is a matter of liberty, not price. To understand the concept, you should think of free as in free speech, not as in free beer." The poster child of open geo data is OpenStreetMap, the "free editable map of the world". Founded in 2004 by Steve Coast, OSM has enjoyed phenomenal growth in users and in contributions of data that can be used anywhere and by anyone and which espouses the values of free as in speech and as in beer. As with all community or crowd sourced collaborative projects, OSM's challenge is to sustain that growth and once complete coverage of a region is reached, in keeping that coverage fresh, current and valid. We'll leave aside that fact that complete coverage is an extremely subjective concept and means many things to many people. Traditionally strongest in urban regions, one of OSM's other key challenges is to match the expectations of their user community who consume that data rather than those who create it. Both internationalisation of the data and expansion out of the urban conurbations will potentially prove challenging in the years to come. That's not to say OSM isn't a significant player in this space and the quality of the data, though varying and in some places duplicated, is for the majority of use cases, good enough. This was backed up by research undertaken by Muki Haklay of UCL which answered the perennial question of "how good is OSM data" with a pithy "good enough". Attempts to capitalise on and monetize the success and data corpus of OSM through the Venture Capital funded Cloudmade have yet to deliver on the promise and with the exception of a set of APIs, Cloudmade has announced the loss of their OpenStreetMap Community Ambassadors and the closure of their London office. All of which lends credence to the fact that simply owning the data isn't enough. So how to solve the dichotomy of geo data? Everyone wants it but no one's willing to pay for it with the exception of the big players, the Googles, the Yahoos and the Microsofts of the world and control of the proprietary data sources has centralised into TomTom and Nokia, both of whom are well placed to capitalise on their data assets but who haven't yet delivered on that promise. Maybe the answer is twofold. Firstly develop an open attribution model whereby the provenance of an atom of data can be tagged and preserved; this would remove a lot of the prohibitions on creating derived works at the original data provenance could still be maintained. Secondly allow limited usage of proprietary data at varying levels of granularity, accuracy and currency, thus creating a freemium model for the data and stimulate developer involvement in donating data to the community as a whole. It's too early to see whether this will come to pass or whether an already tight hold on the data will become tighter still.

Posted via email from Gary's Posterous

O2 in Positive Customer Service Shock?

2009-10-07 04:11

" throws up such gems as "O2 customer service consists of PAY UP OR ELSE" and "O2's customer service has to be the poorest I have ever come across".

O2, the UK Telefonica brand and soon-to-be-loosing-the-iPhone-exclusivity-to-just-about-anyone mobile operator, have a reputation which is, to be honest, just a little bit crap. Their coverage in the rural wilds of Central London, especially around Soho and Covent Garden, seems to be scaled for a single user and a web searchfor "o2 customer service problems" throws up such gems as "O2 customer service consists of PAY UP OR ELSE" and "O2's customer service has to be the poorest I have ever come across".

So we'll leave aside for one moment the fact that I have to pay an additional £20.00 for a measly 10MB of data when abroad via O2's Data Abroad 10 bolt on and accept that I ordered this to be added to my account so I could use data on my iPhone when in the US for this week's Open Hack NYC.The first mailed response from O2 didn't inspire confidence."Hi, Thanks for getting in touch. We'll look into your query and get back to you as quickly as we can, normally within 24 hours."So I waited and less than 24 hours later I got this"Good Morning Gary. Thanks for emailing us about adding the 10Mb Data Roaming Bolt On to your account.Gary, you'll be pleased to know that I've added the 10Mb Data Roaming Bolt On to your account effective from your next bill onwards (10 October 2009). You'll be charged £17.02 excluding VAT (Value Added Tax) per month for this Bolt On.If you want to add the above Bolt On on a different date, please reply to this email and we'll help you further."Data roaming on; WIN. Data roaming on from the date of my next bill and after the event in New York; FAIL.So I asked them, nicely."I'm having to travel at very short notice so I really need this up and running from my first day out of the country which is this Wednesday, October 7th. Can the bolt on start date be brought forward to this day?"That automated reply came back again"Hi, Thanks for getting in touch. We'll look into your query and get back to you as quickly as we can, normally within 24 hours."I'd expected a cut-and-paste response that they could only start services such as this on the first day of a new monthly bill, which basically means minimal work for them and maximum inconvenience for the customer. Then this morning I got this, which was emphatically not what I was expecting."Good Evening Gary. Thanks for emailing us as you want to pre-phone your Bolt On start date. I've pre phoned your Bolt On start date to 07 October 2009 as requested by you. Important - When you email us please provide: your date of birth, postcode and mobile number as it helps us answer your query faster"So fair play to you O2; I'm not entirely sure what pre-phoning is and a bit surprised that you expect me to provide personal data including my date of birth and postal code in every email, but I went into this dialogue with you with zero expectation of success and you pleasantly surprised me. Now if we can just fix that "No Service" in Central London ...Posted via email from Gary's Posterous

Deliciousness: data, licensing, WordPress autosaves, cheese in space and lots of Nutella

2009-07-31 14:15

Ed Parsons, my opposite number at Google, wrote a great blog post on the knots that data licensing can tie you up in and why you end up paying more for a leased digital version than you do for the physical paper version.

WordPress started bugging me about an auto-saved version of a blog post I didn't want to keep but couldn't get rid of. Turns out there's no way to do this from the WordPress dashboard but some MySQL hackery did the trick.

"I am, and am VERY badly affected by being in close proximity to WiFi and other microwave transmission sources. Not that I’d expect you or anyone else who isn’t adversely affected to believe me". The rest of the story on the Daily Telegraph blog is priceless.

Ofcom confirmed what anyone with the UK ADSL line already knows, that the average UK broadband speed is just over half of what's being advertised and paid for.

A US highway exit sign got every word misspelled, apart from the word "exit".

Forget putting men on Mars or getting the Space Shuttle working; we put cheese into space, tracked it, lost it and found it again. Makes you proud to be British.

Someone likes Nutella. A lot.

And finally, if your iPhone gets a text message containing a single square character. Turn it off. Turn it off now.

More intriguing, interesting and just plain bonkers stuff from the information hose pipe we call the internet:

Starting off with a serious note, Ed Parsons, my opposite number at Google, wrote a great blog post on the knots that data licensing can tie you up in and why you end up paying more for a leased digital version than you do for the physical paper version.
WordPress started bugging me about an auto-saved version of a blog post I didn't want to keep but couldn't get rid of. Turns out there's no way to do this from the WordPress dashboard but some MySQL hackery did the trick.
"I am, and am VERY badly affected by being in close proximity to WiFi and other microwave transmission sources. Not that I’d expect you or anyone else who isn’t adversely affected to believe me". The rest of the story on the Daily Telegraph blog is priceless.
Ofcom confirmed what anyone with the UK ADSL line already knows, that the average UK broadband speed is just over half of what's being advertised and paid for.
A US highway exit sign got every word misspelled, apart from the word "exit".
Forget putting men on Mars or getting the Space Shuttle working; we put cheese into space, tracked it, lost it and found it again. Makes you proud to be British.
Someone likes Nutella. A lot.
And finally, if your iPhone gets a text message containing a single square character. Turn it off. Turn it off now.

Gary Gale

Welcome to B2* ... The New Reality Of The Mapping Industry

The Challenge Of Open

The Quest For The London Flood Map