Big data in small doses


I was recently fortunate enough to see Eric Schmidt (Google CEO) give a talk at the Hay Festival which takes place in a small town on the English/Welsh border. As I already work in Hay, the culture-clash of a multi-billionaire techno-giant visiting a rural idyll roughly 20 miles from the middle of nowhere was amplified somewhat and the event left me mulling a lot of things over. Often in the middle of the night when I should be asleep.

I became briefly obsessed with Big Data and noticed a problem (or opportunity depending on your personality type) which I’m not sure has been addressed.

I’ve not blogged before (and haven’t written anything remotely substantial since graduating Uni 5 years ago, so this may be a bumpy ride) but I thought writing some of this stuff down might be interesting to other people and also serve to clear space in my head for other things.


Big Eric was talking about (among many other things) the way technology is changing the world. Events like the Arab Spring were touched on and ideas like digital digital States, but the bit that stuck with me was where he talked about digital identities, citizenship and profiling.

The concept of an online identity was taken to extremes during the talk, with Schmidt commenting that some people are posting sonograms online with baby names – effectively generating a digital identity for the child before it’s even born.*

* Incidentally he spoke about this as if he found it slightly shocking, possibly forgetting this Google advert which kind of endorses pre-natal Googling.

Later during a Q&A session, the concept was taken to another extreme by an audience member who refused to create any kind of digital identity on the grounds that the internet was a “tool for anarchy and amorality”. Schmidt pointed out that due to aggregated profiling and statistical analysis it was almost impossible for anyone to not have some sort of digital identity (a digital shadow?).

So now we’re at a point where everyone who’s ever used Google has some sort of online profile and, according to Schmidt, we now create as much information in two days as we did from the dawn of civilization up until 2003. Add those two things together and people are inevitably starting to find ways to use the huge amounts of data to make predictions, track trends and sell you stuff.

The ways people are using big data range from mundane to ineffective and incredibly successful, but are largely working from the top down – starting with the data at the top and working backwards from that to the people at the bottom.

For example, financial services are plotting trends using big data to see what’s got the most potential for making a fuckton of money (fuckton = 100 shitloads) then deciding which things to invest in, and advertisers are plotting patterns of association to see what you’re most likely to want to buy next then trying to sell you said stuff.

This has two problems: firstly they’re making predictions which will inherently never be 100% accurate (and often fall far short of it) and secondly they’re often alienating the people the data’s mined from. The end result of this is that every time I buy a pair of socks from Amazon (for example) I’m immediately recommended 20 different pairs of socks which firstly I have no need for since I’ve just bought socks and secondly makes me resent Amazon for constantly trying to upsell to me.

Is there a way, then, that allows people to harness this big data to empower, inform or better themselves? For example, imagine an online alternative to meeting someone at the supermarket where profile-matched singletons are introduced to you along with that big HBO boxset you’re looking at so you can meet up and watch it together. Or where a spell of fantastic weather prompts a notification to get off your computer, go outside and mingle with other people in popular spots (as canvassed from things like Foursquare check-ins & mobile browsing data).

We The Data are doing some really interesting stuff in this regard, but there must be lots of possibilities like this where focusing on things from an individual’s perspective can allow for some really positive uses of big data in small doses.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: