Open Data Opens New Potential for International Development
September 21, 2011
In recent years, a revolution in data analytics has changed the way both public and private sector institutions share, manage, and analyze information – and it’s a revolution now reaching developing nations. In Timor-Leste, the Ministry of Finance has just made current, real-time national budget statistics available online. The government of Kenya recently launched a website providing public access to data from the ministries of Finance, Planning, Local Government, Health, Education, and Kenya’s National Bureaus of Statistics. For international development professionals and policymakers, working with “big” and “open” data to better understand the challenges less-developed communities face has never been more exciting than it is today.
Historically, The Asia Foundation and its local partners have invested heavily in the collection and dissemination of research data on the socio-political values, aspirations, and expectations of Asia’s diverse people. From some of the largest election surveys ever conducted in Indonesia, to comprehensive, nationwide business climate studies, to monthly rapid assessment polls in Bangladesh, to the broadest nation-wide public opinion survey conducted annually in Afghanistan, our efforts have responded to the paucity of information in certain contexts and the need to quickly collect, analyze, and disseminate to policymakers the findings of our work. But several key developments have created a flood of available data: from government statistics, to quality of life data, from spatial geographic information to internet usage throughout the developing world. As the volume of enterprise, development, and social information moves from the terabytes to the petabyte scale, development organizations must explore how best to share, manage, and analyze this information.
The first major trend is a technological one. In the 1980s, one terabyte of disk storage cost roughly $12 million dollars. Today, standard individual network drives store twice as much information at a cost of less than $100. This means that classes of data that were extremely expensive to store in the past are now easily preserved and exchanged on low-cost networks. As CPU costs have also fallen, the amount of information that technology systems can effectively manage has skyrocketed. And that’s good news because some incredible advances in how development professionals, governments, and citizens share and exchange data are happening right now.
Launched in early 2010, the World Bank Open Data Initiative is one example of the tremendous potential inherent in the “open data” movement. The Open Data Initiative has become a central hub for accessing free data sets of over 1,200 carefully curated World Bank indicators via the web, World Bank applications, and APIs (Application Programming Interface). This is an important step in opening data for public access and entrepreneurial energy, and it’s an initiative matched by organizations like the IMF, the Asian Development Bank, and many governments including the United States and the United Kingdom – not to mention Kenya and Timor-Leste.
A key step in “opening” data, and where it becomes an incredibly powerful tool for analysis and understanding, is in its inherent structure. The most universal form of structured data is a database which allows for technology systems to select specific pieces of information based on fields which explain to a computer what that piece of data actually is. Because computers can exchange data that is “structured,” researchers and developers can easily access that information to conduct rich analysis, mapping, data visualization, or even new application development. By converting data sets like those compiled by the World Bank to machine-readable formats, the ease of access, analysis, and sharing of that information rises exponentially.
When many actors, including governments, civil society organizations, donors, and the private sector, agree to make their data available in a consistent and structured fashion, the capacity to share and exchange data across any number of platforms and tools becomes transformative. However, we must also remember that open data initiatives will only work when the data collected is reliable, timely, and consistent. Biases introduced into large data sets through faulty methodologies by researchers can create as many problems as they seek to solve. This is particularly the case in the developing world, where capacities remain limited and strained as a result of scarce resources.
Despite potential problem areas, greater access to information allows researchers to conduct better analysis; policymakers to access more complete pictures of socio-economic trends; and entrepreneurs access to data that can be built into consumer-facing applications leveraging real-time public data. Open, structured data is fueling a revolution in how development professionals study social phenomenon, and at The Asia Foundation we’re working toward aligning our own research and analysis capabilities with this new movement. That means we will be incorporating tools like GIS and mapping tools into our work; contributing to open-source movements designed to enable the collection of data over SMS-enabled mobile phone networks; and integrating 3rd party data sets into our own research efforts.
In Afghanistan, for example, The Asia Foundation has conducted seven nation-wide surveys of the Afghan people dating back to 2004, and this work, collectively, has helped establish an accurate, long-term barometer of public opinion across Afghanistan. The 2011 survey captures the Afghan public’s perceptions of reconstruction, security, governance, and attitudes toward government and informal institutions, as well as the upcoming elections, the status of women, the role of Islam, and the impact of media. In support of the burgeoning “open data” movement, and in conjunction with the launch of the 2011 survey in mid-November, we intend to release these survey results as well as past survey findings in a machine-readable format, accessible to all and available for download. It’s never been a more exciting time to be involved in data-driven research, and it’s time to get smart about data.
John Karr is The Asia Foundation’s Director of Digital Media and William S. Cole is Senior Director of the Foundation’s Governance, Law, and Civil Society Programs. They can be reached at [email protected] and [email protected], respectively. The views and opinions expressed here are those of the individual author and not those of The Asia Foundation.
About our blog, In AsiaIn Asia is a weekly in-depth, in-country resource for readers who want to stay abreast of significant events and issues shaping Asia's development, hosted by The Asia Foundation. Drawing on the first-hand insight of over 70 renowned experts in over 20 countries, In Asia delivers concentrated analysis on issues affecting each region of Asia, as well as Foundation-produced reports and polls.
In Asia is posted and distributed every Wednesday evening, Pacific Time and is accessible via email and RSS. If you have any questions, please send an email to [email protected].
ContactFor questions about In Asia, or for our cross-post and re-use policy, please send an email to [email protected].
The Asia Foundation
465 California St., 9th Floor
San Francisco, CA 94104
PO Box 193223
San Francisco, CA 94119-3223
THE LATEST ACROSS ASIA
Tackling South Asia’s Water Crisis: Can Civil Society Help?
September 21, 2016
Laos: Reaping the Benefits of a New Era of International Cooperation
September 21, 2016
Asia Foundation Trustee Kathleen Stephens to Receive Distinguished Friend of Korea Award
September 20, 2016
Asia Foundation Launches Let’s Read! E-Books for Cambodian Children
September 19, 2016