Zika virus in Rio de Janeiro, Brazil: the city government used digital dialogues to decide how to address the disease outbreak. Credit: Dado Galdieri/Bloomberg/Getty

The election of a politically inexperienced president in the United States, Britain's vote to leave the European Union and the initial rejection of a peace deal in a Colombian referendum to end an armed conflict all signal dissatisfaction with the political status quo. Yet citizens have few opportunities to influence government decisions beyond the ballot box.

“This is a time when almost every aspect of government can be improved,” Geoff Mulgan, chief of Nesta, a UK charity that aims to foster innovation and digital democracy1, has said. It is time to work out how, together.

Last year, my students at the Governance Lab at New York University designed a process to help four governments — the city government of Rio de Janeiro in Brazil and the national governments of Argentina, Colombia and Panama — to obtain expert advice about the global Zika outbreak. Our project, called Smarter Crowdsourcing, broke down the outbreak into actionable problems, such as the accumulation of standing water leading to the breeding of more infected mosquitoes. Then we organized 6 online dialogues with 100 experts from 6 continents to gather knowledge, experiences and advice. Three months on, these governments are beginning to implement what they learned. For example, Rio and Argentina have started social media 'listening' initiatives to learn how the public perceives the disease.

Listening and crowdsourcing approaches can make governments more agile in responding to problems. Whether the issue is public health, global warming or prison reform, governments struggle to identify and implement new approaches quickly. When car pioneer Henry Ford wanted to innovate, he shut down and retooled his factories. Governments do not have that luxury.

Applied research

Technology is already changing the way public institutions make decisions. Governments at every level are using 'big data' to pinpoint or predict the incidence of crime, heart attack and foodborne illness. Expert networking platforms — online directories of people and their skills, such as NovaGob.org in Spain — are helping to match civil servants who have the relevant expertise with those who need the know-how.

To get beyond conventional democratic models of representation or referendum, and, above all, to improve learning in the civil service, we must build on these pockets of promise and evolve. That requires knowledge of what works and when. But there is a dearth of research on the impact of technology on public institutions. One reason is a lack of suitable research methods. Many academics prefer virtual labs with simulated conditions that are far from realistic. Field experiments have long been used to evaluate the choice between two policies. But much less attention is paid to how public organizations might operate differently with new technologies.

The perfect must not be the enemy of the good. Even when it is impractical to create a control group and run parallel interventions in the same institution, comparisons can yield insights. For instance, one could compare the effect of using citizen commenting on legislative proposals in the Brazilian parliament with similar practices in the Finnish parliament.

Of course, some leaders have little interest in advancing more than their own power. But many more politicians and public servants are keen to use research-based evidence to guide how they use technology to govern in the public interest.

The MacArthur Foundation in Chicago, Illinois, has started funding a research network — a dozen academics and public servants — to study the possibilities of using new technology to govern more transparently and in partnership with citizens (see www.opening-governance.org). More collaboration among universities and across disciplines is needed. New research platforms — such as the Open Governance Research Exchange, developed by the Governance Lab, the UK-based non-profit mySociety and the World Bank — can offer pathways for sharing research findings and co-creating methodologies.

Colombia's government polled its citizens using paper ballots on its peace deal. Credit: Mario Tama/Getty

Five priorities

These are the key areas in need of research.

Data-driven decision-making. Computable information can improve governance. So it is imperative to do more systematic research to guide investment in new data-rich platforms and policies.

Through data analysis, policymakers can understand the past performance of policies and services — their efficiency and their disparate impact on different populations. For example, in the United Kingdom, studies of a unique birth cohort of 70,000 people since the Second World War have generated more than 6,000 academic papers and led to an overhaul of medical support during pregnancy and childbirth2.

And better data can help to predict policy outcomes. Chicago's city government, for example, created an algorithm to predict food-safety violations. This increased the effectiveness of its inspections by 25%.

But it is the exception not the rule in the public sector to use advanced analytics, not the rule. Even when algorithmic approaches are adopted, such as to measure the risk of reoffending, outcomes are rarely evaluated. Few people working for government have the data-science skills needed to conduct such research.

Open government data. Many nations collect and publish government information in freely reusable forms. The impact of these data on solving public problems needs study. In the United States, data from universities and transport authorities have been transformed into apps to help the public make more-informed decisions about their university education and their commutes to work. And open data can promote civil rights. For example, civil-rights lawyers in Zanesville, Ohio, used data released by utility companies to uncover a 50-year pattern of racial discrimination in water-service provision3. But more research is needed on the impact of open data on governing and problem solving.

For example, many governments assume that, by helping consumers to make more effective choices, the collection and disclosure of information on energy efficiency or mobile-phone tariffs lets them regulate industry with little administrative burden. But under what circumstances are disclosures more effective than command-and-control regulations? This is a testable research question.

The potential public good of companies giving consumers access to the data they each generate also needs researching. The Small Data Lab at Cornell Tech in New York City is investigating what happens when online supermarkets open up purchase data to individuals, and to researchers on their behalf. The team is testing tools that use such information to 'nudge' people towards healthier eating, with personalized coaching derived from their data.

Responsible data use. To move towards evidence-based decision-making, citizens must probe how to make use of administrative data collected about them by the government. The London-based non-profit organization New Philanthropy Capital has been testing a model for a secure 'data lab'. The lab accepts outside requests to use UK Ministry of Justice administrative data about criminals who reoffend to measure the effectiveness of social-service programmes. It next wants to test the approach with Britain's National Health Service to use health and social-care data to evaluate programme effectiveness while safeguarding privacy.

Research is urgently needed on the impact of algorithms on public life — of the kind being done by Lee Rainie at the Pew Research Center in Washington DC, who is studying both the ways in which algorithms might be used to solve societal problems and their potential for misuse. Questions about the impact of data-driven decision-making on civil liberties go beyond the usual issues of surveillance and privacy. There are ethical implications for the 'digital invisibles' — people on whom data are not collected. Justin Longo, a open-government researcher at the University of Regina in Canada, has found that people who aren't represented in the big-data world may be subject to misguided interventions and biased policy.

Citizen engagement. Whether obtaining diverse public input through the Internet improves the legitimacy and efficacy of governing processes is another hypothesis in need of testing. Crowdsourcing and open innovation have been used in the public sector, but the practices are not well institutionalized. There is a dearth of research on how public organizations engage with civil servants and the public online, and thus a lack of insight into how to design successful online engagement processes and institutions.

What are the best ways to devise participatory opportunities at each stage of public decision-making, from problem solving to policy evaluation? At Arizona State University, the Center for Policy Informatics is partnering with the city of Phoenix to test and develop ways of including the expertise, experience and priorities of citizens in urban planning. Approaches range from sophisticated computer models to coloured pieces of paper scaled to the sizes of budget proposals.

Crowdsourcing environmental data holds great promise, as citizen scientists worldwide are doing in the SciStarter, CrowdCrafting and Safecast online communities. There are myriad natural experiments on citizen participation in lawmaking, from Brazil to Canada and France. But research lags behind and lessons are not being learned.

Incentives. To design participatory governing processes for the digital age, researchers must dig into the age-old question of human motivation. There is a well-developed literature on crowdsourcing in business and science, but there is too little understanding of what drives different kinds of individuals to take part in online policy consultations and what motivates governments to run them. Research led by Karim Lakhani at Harvard Business School in Boston, Massachusetts, suggests4 that people respond to intrinsic incentives (such as membership of a group) more than they do to extrinsic ones (such as the offer of an iPad). Companies may need other encouragements to share data that help solve public problems. At New York University, Stefaan Verhulst and Andrew Young study different ways of getting companies to do so. They are creating and studying such data-sharing arrangements, what they call data collaboratives, in partnership with UNICEF, the United Nations' children's charity.

We also need to know more about who participates. To 'unmask the crowd', Tanja Aitamurto at Stanford University in California and colleagues have studied5 a crowdsourced law-reform initiative in Finland and found that it mostly involved educated professional males.

Into the wild

To accelerate research into how real-world institutions could improve through technology, I offer these short-term prescriptions.

First, more conferences in more disciplines should address how to introduce research, including experimental design, to study innovations in governing. Anita McGahan, who studies strategic management at Toronto University in Canada, made opening governance the theme of the 2015 annual conference of business-school researchers when she was president of the US-based Academy of Management, an association for management researchers. Henry Farrell, a political scientist at George Washington University in Washington DC, brings together computer and political scientists to develop a common understanding of collective problem solving.

Second, in addition to philanthropic investment, more agency budgets at every level of government should pay for research into agency operations. Although dedicated science agencies fund external research on innovation, almost no money goes towards addressing internal innovation.

Public expectations of what governments should deliver have risen.

Third, public officials need to know how and when to design experiments that will yield insight while protecting taxpayers' money. Policy agencies such as the UK Cabinet Office and the US Office of Management and Budget (OMB) should offer guidance and training on how to design responsible research experiments including randomized controlled trials.

Fourth, government bodies need streamlined ways of collaborating with academia, such as the authority to recruit and to create short-term fellowships. The Social and Behavioral Sciences Team, a cross-agency group that was part of former US president Barack Obama's administration, routinely collaborated with and brought in outside academic experts to complete rigorous evaluations of more than 30 trials of new policy interventions.

Fifth, where academics cannot be brought into government, agencies should push questions out. The United States has legal authority under the America Competes Act to award prizes to members of the public for solving difficult problems. US federal agencies have conducted such prize-based challenges more than 750 times on Challenge.gov, inviting the public to reduce casualties from runaway vehicles at military checkpoints or to compose a catchy public service announcement to increase handwashing and prevent the spread of influenza.

Finally, rules on ethical but efficient administration of research need to be clarified. For example, US regulations on human-subjects research exempt from ethical review any research on how government service and benefits are delivered — but make no mention of the study of governance innovations. The United States should streamline its Paperwork Reduction Act statute, which requires that agencies wishing to ask questions of the public submit to a lengthy approval process by the OMB.

In the face of increasingly complex challenges, rapid social change and technological innovation, governments must find new ways to do more with less. Despite declining tax revenues and deteriorating fiscal conditions, public expectations of what governments should deliver have risen. In every domain, governments need to innovate in how we respond to challenges.

It is not enough to experiment with new policies in the laboratory of democracy if we use the same beakers. We need to change the processes by which we make policy and deliver services for the public good. Empirical yet agile research in the wild is the route to knowing how.