Common Crawl is a not-for-profit web crawler service that makes the data collected open for all to use. A post on the MIT Technology Review blog speculates how the initiative might spawn the next Google.
One of the problems with Big Data is that it’s held mainly by large corporations and government agencies, both of which have the tendency to keep their data private on that basis that information is power and power means money.
We see this in the business models of Facebook, Google and many of Silicon Valley’s startups; the information garnered about users is as, if not more so, valuable as an utility from the product.
Initiatives like Common Crawl tilt the balance somewhat back towards consumers, citizens, and smaller businesses.
How well Common Crawl and other similar initiatives fare remains to be seen – Wikileaks was a good example of how such projects can flare out, collapse under the weight of egos or be harrassed by corporatist interests.
In search, Google are open to disruption as they tweak their results to suit initiatives like Google Plus. During the company’s earnings call earlier this week Larry Page spoke of the challenges of staying focused on the opportunities that matter, it may well be the company is more distracted from its core business than it should be.
Whether Common Crawl disrupts Google is up to history, it could just as well be a couple of kids called Sergei and Larry with a smart idea.
The imperative now though is to try and keep as much public data available for everyone to use and not lock it away for the privileged few. That will let the future Googles develop while making our societies more fairer and open.