Big data and the new phenomenon open data are closely related but they're not the same. Open data brings a perspective that can make big data more useful, more democratic, and less threatening.
While big data is defined by size, open data is defined by its use. Big data is the term used to describe very large, complex, rapidly-changing datasets. But those judgments are subjective and dependent on technology: today's big data may not seem so big in a few years when data analysis and computing technology improve.
Open data is accessible public data that people, companies, and organisations can use to launch new ventures, analyse patterns and trends, make data-driven decisions, and solve complex problems. All definitions of open data include two basic features: the data must be publicly available for anyone to use, and it must be licensed in a way that allows for its reuse. Open data should also be relatively easy to use, although there are gradations of "openness". And there's general agreement that open data should be available free of charge or at minimal cost.
The relationship between big data and open data
This Venn diagram maps the relationship between big data and open data, and how they relate to the broad concept of open government. There are a few important points to note.
Big data that's not open is not democratic: Section one of the diagram includes all kinds of big data that is kept from the public – like the data that large retailers hold on their customers, or national security data like that collected by the NSA. This kind of big data gives an advantage to the people who control it but may disempower the rest of us. It's this kind of big data that has become most controversial.
Open data doesn't have to be big data to matter: Modest amounts of data, as shown in section four, can have a big impact when it is made public. Data from local governments, for example, can help citizens participate in local budgeting, choose healthcare, analyse the quality of local services, or build apps that help people navigate public transport.
Big, open data doesn't have to come from government: This is shown in section three. More and more scientists are sharing their research in astronomy, genomics, and other areas in a new, collaborative research model. Other researchers are using big data collected from social media – most of which is open to the public – to analyse public opinion and market trends.
But, when the government turns big data into open data, it's especially powerful: Government agencies have the capacity and funds to gather very large amounts of data (such as the US examples in section six), and opening up those datasets can have major economic benefits. I now direct the Open Data 500 study at the GovLab at New York University. We've found 500 examples of US-based companies that are building their businesses on open government data, and much of that data is big data as well. We're now planning to work with Open Data Institute to replicate our study in the UK, and I expect to find the same pattern.
Applying open data principles to big data can help solve some of the difficult issues that big data has raised. The biggest threat to public wellbeing is the risk that private, personal data can be collected and used as big data in ways the subjects of the data – namely, all of us – may not want or approve of. Paradoxically, opening up this sensitive data, in a specific and controlled way, may actually make it more secure.
The problem now is not only that government agencies and some businesses are collecting personal data about all of us; it's also that we as individuals don't know what's being collected and don't have access to the information about ourselves. If we knew more, we could control more. The UK government's midata initiative, which has encouraged businesses to share customer records with customers themselves, is part of the solution. Similar US programmes, such as Blue Button for health records and Green Button for energy usage data, are also having a positive effect.
Both big data and open data can transform business, government, and society – and a combination of the two is especially potent. Big data gives us unprecedented power to understand, analyse, and ultimately change the world we live in. Open data ensures that power will be shared – and that the world we change will, with luck, become a fairer and more democratic one.
Joel Gurin is senior advisor at GovLab at New York University, where he directs the Open Data 500 study. He is author of Open Data Now.
What government can do with big data – livechat
Want your say? Email us at public.leaders@theguardian.com.
Join the Public Leaders Network for more comment, analysis and job opportunities, direct to your inbox. Follow us on twitter via @Guardianpublic.
Comments (…)
Sign in or create your Guardian account to join the discussion