Open Data is the idea that data should be freely available for everyone to access, use and republish as they wish, published without restrictions from copyright, patents or other mechanisms of control. Public sector information made available to the public as open data is termed ‘Open Government Data’.Governments and their contractors collect a vast quantity of high-quality data as part of their ordinary working activities. Typically this results in the state becoming a powerful data monopoly able to structure and homogenize the interactions between itself and its citizens. These one-sided interactions are expensive and unresponsive to citizens’ needs and can unnecessarily restrict government activities, as well.
Opening government data involves both policy and technical considerations. If governments’ data is made open, it can have huge potential benefits including:
- Transparency: In a well-functioning, democratic society citizens need to know what their government is doing. To do that, they must be able freely to access government data and information and to analyse and share that information with other citizens.
- Efficiency: Enabling better coordination and efficiency within government, by making data easier to find, analyse and combine across different departments and agencies.
- Innovation: In a digital age, data is a key resource for social and commercial activities. Everything from catching a bus to finding a doctor depends on access to information, much of which is created or held by government. By opening up data, government can help drive the creation of innovative business and services that deliver social and commercial value.
Where many public records laws and policies regulating the right to information have traditionally relied on reactive disclosure, meaning public information has to be requested before it is shared, a government fully engaged in open data is choosing to proactively disclose information - meaning public data is released as it is collected and before it is requested. Put another way, the vision of open data is for government information to be ‘open by default’.Open data also has a number of technical implications, with special consideration given to the particular formats chosen for data release. Open formats are those that are structured and non-proprietary, allowing the public and the government to extract maximum value from the information now and in the future.
Governments around the world cite many different reasons for starting open data initiatives, including increasing government transparency and accountability, catalysing the creation of new digital services and applications for citizens, unlocking the full economic potential of public information, and evolving current government services for anticipated future needs. Although much of this top-level government interest is new, there are many professions and communities engaged in dialogue, policy, and development around this issue, including from government officials, journalists, developers, transparency reformers, issue advocates, and interested citizens.
Examples in Practice
NASA hosted the world’s largest hackathon
The International Space Apps Challenge is an international mass collaboration focused on space exploration that takes place over 48-hours in cities around the world. The event embraces collaborative problem solving with a goal of producing relevant open-source solutions to address global needs applicable to both life on Earth and life in space. Themes are Earth Watch, Technology in Space, Human Spaceflight, Robotics and Asteroids. NASA is leading this global collaboration along with a number of government collaborators and 100+ local organizations.
Australia’s information commissioner is responsible for all information held by the government
According to the Australian Information Commissioner Act 2010, the information commissioner’s function is to report on any matter related to the collection, use, disclosure, management administration or storage of, or accessibility to, information held by the Government.
Belgium piloted a system to develop unique identifiers for addresses and streets
The Core Location Pilot ‘Interconnecting Belgian National and Regional Address Data’ analysed the possibility of developing unique identifiers for addresses and streets in Belgium.
Canada developed an Open Government License
Canada’s OGP Action Plan included include a specific commitment to developing a new Open Government Licence. The goal of this commitment was to remove restrictions on the reuse of published Government of Canada information (data, info, websites, and publications), aligning better with international best practices for licensing. Various licences in use to this point were inconsistent, contained restrictive clauses, and were written in language only a lawyer could understand.
Greece has set up a central catalogue of public data
Greece has set up Data.gov.gr, a central catalogue of public data which provides access to public databases from all Greek government bodies. The purpose of the website is to increase public access to high value, machine readable datasets, by cataloguing, indexing, storing and searching the public sector’s data and information, as well as providing web services to citizens and other information users.
Data.gov.gr was launched in September 2013 and Ministries will be responsible for ensuring the availability of datasets falling under their jurisdiction in the form of ‘open data’.
The website provides a mechanism for citizens to vote for the datasets they wish to see that are not yet publicly available. Citizens’ proposals are sent to the relevant public bodies and the datasets are made priorities for publication.
In Canada, a government wide initiative has been undertaken to identify all government data holdings
The Canadian Data Inventory Project is a government-wide stock-taking of federal data holdings. The inventory consists of the metadata on datasets held within government departments and will be linked, when possible, to specific key policy issues. Other metadata will include the title, subject area and subtopics, time and geographic coverage, data source and size, ownership and contact information and a description of the dataset. Together with this data inventory project, work is also being done to assess data needs and identify data gaps.
In Canada, the information commissioner investigates access to information complaints and advocates for more proactive disclosure within government
The Information Commissioner in Canada investigates complaints about federal institutions’ handling of access requests. The Information Officer also encourages federal institutions to disclose information, and continually makes the case for greater freedom of information through targeted initiatives, such as the annual Right to Know Week, and ongoing dialogue with Canadians, Parliament, federal and provincial institutions.
In Denmark, enabling free access to ‘basic data’ will save the government an estimated USD 45 million per year
As part of Denmark’s eGOVERNMENT strategy 2011-2015, individuals, public authorities and private businesses get free access to retrieve and use what in Denmark is called basic data. Basic data is fundamental information that is used by government for day to day administration and includes data about people, companies, addresses, land/properties and administrative geographic data, such as administrative and electoral boundaries.
Once the initiative has been completely phased in, in 2020, there are expected savings for the public sector of about USD 45 million per year as a result of lower administration costs. In particular the cost of buying data from other government organisations will be reduced.
Saving for the private sector, from no longer having to submit the same information to the public sector several times or spend time on buying public data, is estimated to be about USD 87 million per year. It is expected that real estate, insurance, finance, and telecommunications and geospatial companies in particular will benefit directly from open basic data.
In Indonesia, individuals have the right to pursue legal recourse if they are prevented from obtaining information under the freedom of information act
The Indonesian Public Information Disclosure Act from 2008 (Article 4) gives individuals the right to file a suit in court if they are obstructed from obtaining, or fails to obtain information for which they entitles.
In Israel the government worked with Google to make transit data more useful
Data.gov.il is a single window portal that aims to give access to public records data produced from information held by the government. The idea is to encourage creative use of the data by civil society and the larger public by providing tools for information presentation, standardized downloading and standard APIs.
When Google Israel tried to use this data to create a transit app they identified many inaccuracies. Google Israel ran its validation tools over the government database and made recommendations for improvement. “We found the e-Gov team very open to the process of validation and improvement. It took 6-10 months of iteration with the government until we were able to create the transit map for 25,000 bus stations and 53 train stations across the country”
In Japan, the Parliament’s National Library plans to digitise its entire collection to make available online
The National Diet Library of Japan, the country’s only national library, is on its way to full-text digitisation, which will make Japanese literary artefacts widely available and searchable online.
In Slovakia, government contracts are published online
Corruption in public procurement in Slovakia has been a long-standing problem. Problems in public procurement occur through the manipulation of tenders before or after the awarding of the bid, for example by limiting competition through setting unreasonable conditions, or not fully enforcing contracts, or changing them significantly after tendering. One key example was the ‘Notice Board Scandal,’ a high profile procurement scandal which took place in 2007 when the Ministry of Development published a tender request for construction services totalling 119.5 million Euros on a small notice board in the hallway inside the ministry building where few could see it. A firm that was known to have close ties to the head of the ruling party won the contract. More than a year after the fact, the scandal came to light and was invalidated by the Slovakian Office of Public Procurement.
Slovakian law was overhauled in response to the 2011 update of the EU Procurement Directives and the concern over high profile public procurement scandals. Major reforms included:
- The introduction of e-procurement, in which dissemination of tenders, tender documents, the submission of bids and the publication of notification of awards is done publicly through a single portal;
- The introduction of reverse auction mechanisms for procuring goods and services
- Mandatory publication of all public contracts on a centralized online government contract repository.
The Slovak Republic, Act No. 546/2010 Coll. supplementing Act No. 40/1964 requires all public contracts, with certain limited exceptions, to be published online. To avoid secret contracts, any unpublished contracts are declared to be unenforceable. The Government Public Procurement Office manages Slovakian procurement rules. This office publishes the official Public Procurement Journal, legal interpretation of the Public Procurement Act, maintains a register of procurement documents and operates the procurement portal, which is called EVO.
These reforms have made it significantly easier for organizations and people outside the government to access public data about procurement. Previously access tender documents and other procurement data was granted primarily through requests under Freedom of Information Laws (FOI). If requests were not granted and appeals could take so long that the information was no longer relevant. Now, through EVO all of that information is published including the contract itself and information on the bids received and the process of contracting.
The availability of data has lowered the barriers to entry for participating in the oversight process. This has enabled much greater public participation in uncovering suspect procurements. Before the reforms, exposing corruption relied on whistleblowers alerting journalists or watchdogs of suspicious proceedings. Now more tips come from local activists and individuals. For example teachers, highlighted information about large Ministry of Education procurements for flowers and alcohol and the Ministry of Finance conducted an audit.
Opening up the information makes it possible compare contractors and look at patterns across cities and institutions. Local people who may know who the mayor is friends with, and which local businesses have relationships with people of influence are able to spot issues that a researcher farther removed from the process might not be able to.
Despite these positive developments corruption in Slovakian public procurement remains a challenge. While the availability of data has been a positive step, there remain barriers to using it, and where there is no real threat of enforcement, transparency and disclosure are limited in their ability to bring about effective change. Researcher using the data say that rationalizing the manner in which the data is released and formatted would make it easier to use, reducing the cost of finding corruption and increasing the likelihood that improper processes will be uncovered. The NGOs TI-Slovakia and Fairplay Slovakia maintain an online Open Contracts website built off the procurement data scraped and structured from public sources. This portal visualizes procurement expenditures by procurers, suppliers, sectors and regions as well as provides downloadable structured procurement data in bulk. Having data available in these formats also enabled TI-Slovakia to conduct broader analyses than were previously possible.
Standardised data formats, bulk downloading, structured formats like XML would make it easier to use the data. Another key problem is that there are no data licenses attached to public data so it is not clear how it can be used. Additional information about pre-tender requirements would also be valuable.
Formal mechanisms and institutional means of sanction also need to be strengthened. The Sunlight Foundation concludes that “Most significantly, transparency alone, it appears, cannot change deeply ingrained corrupt practices in a short time span. Transparency can only highlight the problem, and provide tools for oversight and investigation. Enforcement mechanisms, both formal and informal, must be brought to bear to sanction those whose transgressions are revealed by transparency-enabled oversight.”
[Source: Sunlight Foundation case study]
In the U.S. city of San Francisco, the government’s open data policy calls for all departments to publish an inventory of data held
According to the San Francisco’s Open Data Policy, each City department must make reasonable efforts to make available all datasets under the department’s control. In addition, the policy calls for an Open Data Plan to be prepared and to include a summary description of all datasets under the control of each City department.
In the U.S. state of Utah, a Transparency Advisory Board acts as the oversight authority for the state’s open data policy
In the U.S. state of Utah, the Utah Transparency Advisory Board acts as the oversight authority for the state’s open data policy. The Board is comprised of six members knowledgeable about providing public access to public financial information. Its responsibilities include making public information more accessible through an information website transparent.utah.gov.
In the United States, the 2013 Open Data Policy called for federal agencies to perform an audit and create an index of datasets
The US Open Data Policy from 2013 require agencies to index all of their data (internally), to make a public list of all their public data, and requires all agencies to list all their data that can be made public.
In the US, the White House Office of Science and Technology Policy requires federal agencies that fund research make the results and underlying data available to the public
A memo Increasing Access to the Results of Federally Funded Scientific Research from the Office of Science and Technology Policy from February 2013 includes a requirement for all U.S. federal agencies that spend over $100 million annually supporting research and development to make the results of that (non-classified) research, as well as the underlying data, available to the public within a year of publication.
Many government’s FOI laws give citizens the right to sue the government over freedom of information requests
The U.S. Freedom of Information Act, for example, provides a person with the right to sue the government if he/she has had no response to an information request in a fixed period of time.
Moldova publishes government data through an Open Data portal
Moldova launched its Open Data portal www.date.gov.md in 2011. The portal brings together disparate and disconnected datasets published by various Ministries and central administration offices under a single umbrella platform. Following technical and financial support from the World Bank, the portal started off with publishing 67 datasets from five public agencies in education, healthcare, economics, finance and agriculture. The number of datasets has since increased rapidly to 672 datasets in mid-2013. New legal provisions for open government data in Moldova, based on European best practices, have been drafted alongside the work on the Open Data portal.
Moldova requires each Ministry to release new datasets
In April 2011, Moldova became one of the first countries in Eastern Europe to launch an Open Data portal. The portal, www.date.gov.md brings together datasets published by various Ministries and central administration offices. The Prime
Minister issued a directive which required Ministries to release three new datasets each month.
Some countries, such as Canada, have freedom of information ombudsmen with special legal enforcement powers
In Canada, the Information Commissioner investigates complaints about federal institutions’ handling of access to information requests and has strong investigative powers to assist her in mediating between dissatisfied information applicants and government institutions. As an ombudsperson, the Commissioner may not order a complaint to be resolved in a particular way, though she may refer a case to the Federal Court for resolution.
The Australian state of New South Wales opened up its draft open data policy to comments on the web
After consulting with stakeholders, the Australian state of New South Wales released a draft Open Data Policy in May 2013 for public comment. The public was given about a month to participate in a discussion forum on the website engage.haveyoursay.nsw.gov.au and provide input to the following questions: (1) Which ‘open data principles’ should be included in the policy?; (2) Which jurisdictions have the best open data policies and why?; and (3) What can governments do to promote open data in the public service?
The Brazilian ministry of finance publishes large amount of budget data in machine-readable formats
Brazil has developed a number of instruments for fiscal transparency. The most well known initiative perhaps is the transparency portal www.portaltransparencia.gov.br/, which was started in 2004 and which draws together information from across the federal government. The portal includes information on public revenues, budgets and spending as well as income and assets of public servants and information on procurement. It is updated daily and provides searchable, open access to the data, as well as offering specific sections focused on areas of particular interest such as the World Cup and the Olympics.
The portal aims to provide information which can be easily accessed and used. For example budget lines are tagged with popular names (for example ‘bolsa familia’ as well as the official title ‘Transferência de renda e apoio à família no acesso à saúde e a à educação’).The information in the budget portal is provided in the form of open, machine readable data, so that it can be used and reused by civil society, media and other organisations to create analysis, visualisations and tools to navigate the information.
As the practice of transparency has developed, so too has the legal framework. In 2012 Brazil established a new Transparency Law obliging all Brazilian public entities (executive, judiciary and legislative bodies at the federal, state and municipality levels, as well as in the federal district) to publish detailed budget data online in real time.The transparency initiative is led by the Office of the Comptroller General. Significant effort has been put into education and dialogue, both within and outside of government, to understand user information needs, to support users to use the data and to generate high quality information flows.
As part of its first OGP Action Plan Brazil has been providing more budget information on its transparency portal, improving data on social benefits and on contracts. In its second Action Plan there is a plan to rebuild the Transparency Portal completely, based on learning from the first ten years, to improve quality and usability and to connect budget information to datasets on actions.The federal government is now working to encourage states and municipalities to develop their own portals.
[Source: Based on a webinar presentation by Otavio Castro Neves, July 30 2013]
The City Council of Enschede passed a motion recognizing the importance of reusable open data and encouraging citizen participation
At a meeting in March 2011, the City Council of the Dutch city of Enschede passed the Motion Innovation and Agility with Open Data where it requested that data should be made open and freely available. The motion recognised that the value of the data held by the city could be enormously enhanced if it was available as open data, allowing citizens to use this data to contribute to economic and sustainable activities.
The city of Oakland drafted their open data policy using a collaborative method involving open, online comments and public meetings
The 2013 open data policy for the city of Oakland, California, was drafted using a collaborative method in which the public was invited to comment and participate directly in a public hearing.
The England and Wales Land Registry contains an inventory of all available data
The Land Registry is a UK government department established in 1862, responsible to the Secretary of State for Business, Innovation and Skills. It keeps and maintains the Land Register for England and Wales; one of the world’s largest property datasets.
A number of free datasets have been published since 2012 under the Open Government Licence (OGL) in CSV and Excel formats, and as of January 2013 as linked data. The Land registry is progressively adding more datasets, focusing on those it judges to be of wider public interest, information frequently requested and information that is not available from elsewhere. As part of this process the Land Registry released a dataset inventory to provide a transparent, comprehensive list of the data it holds, clearly outlining whether it can release it as open data, and inviting feedback to prioritise future data releases.
The Flemish government’s open data policy that makes open-by-default the standard
The Flemish government in Belgium approved an open data policy in 2011. The policy contains six main elements:(1) open data is the default option, and public digital availability the norm; (2) re-use of the data is allowed for any purpose; (3) open standards need to be used; (4) data is to be made available at the first source where possible; (5) integral approach to realizing open data, involving all levels of government; and (6) a central data warehouse on Flemish government’s own data is foreseen.
The government of Chile has published 1,003 datasets
These 1,003 datasets, published on the government website datos.gob.cl are ordered by ministry, categories, and popular tags.
The National Audit Office in the UK audited government data holdings and released a report on their findings to the public
The UK National Audit Office scrutinises public spending on behalf of Parliament and has statutory authority to report to Parliament on the economy, efficiency and effectiveness with which departments and other bodies have used their resources.
In 2012 the NAO carried out a cross-government review of the government’s progress in implementing transparency. This included a survey and interview of 16 departments and a review of central and local government data releases to assess compliance with Treasury guidance on departmental spending data, and progress towards implementation of the public data principles developed by the Public Sector Transparency Board.
The Senate of Italy publishes legislative data on its own open data portal
The Italian Senate publishes legislative data on its data portal dati.senato.it. The portal provides information for citizens, researchers and journalists to analyse and share information of what is being proposed, discussed and voted in the Senate. The data is released under Creative Commons (CC BY 3.0) for free reuse.
The U.S. Government has released its full list of published and unpublished data holdings
After a series of requests by the Sunlight Foundation, the U.S. Office of Management and Budget (OMB) announced in February 2015 that it would release its Enterprise Data Inventory (EDI). The EDI is a comprehensive inventory listing of agency data resources including public, restricted public and non-public datasets. The newly released EDI has a total of 624 datasets from 24 agencies of which 53 datasets are ‘restricted public’ and 30 are ‘non-public’. The Enterprise Data Inventory was developed as part of the May 2013 Memorandum on Open Data Policy-Managing Information as an Asset.
The UK government published guidelines on the use of unique resource identifiers
The UK Government website data.gov.uk provides guidance about publishing Linked Data/unique resource identifiers within the UK public sector.
The UK Prime Minster set out a broad commitment to open data in a letter to government departments
In May 2010, the UK Prime Minister, David Cameron, sent a letter to government departments on plans to open up government data. The letter contained the government’s initial transparency commitments in relation to central and local government spending transparency, and key government datasets, such as crime data. The Prime Minister sent an additional letter to Cabinet Ministers on transparency and open data in July 2011 which set out a new set of commitments. These commitments focused on improving data quality and data availability, primarily concerning the National Health Service, education and skills, criminal justice, transport, and government financial information.
The United States provides novel ways for publicly released data to be used by citizens to solve global challenges
The US National Aeronautics and Space Administration (NASA) have launched an International SpaceApp Challenge. In what turned out to be the largest hackathon in the world, the Challenge brought together space agencies and citizen technologists, scientists and developers to contribute to space exploration missions using publicly released data. The participants used NASA’s massive datasets and open source software technology to respond to over 50 challenges designed around software and hardware development, citizen science, and data visualisation. For this year’s event 770 solutions were submitted.
The SpaceApp Challenge provides a model of government-citizen collaboration that could be used for other transnational issues, such as the environment, defence or diplomacy.
The US city of Chicago is developing contract provisions to promote open data in technology related procurements
Chicago’s open data policy includes a provision requiring the chief data officer to work with the chief procurement officer to develop contract provisions to promote open data policies in technology-related procurements.
The US government established a public open source project to facilitate the implementation and evolution of its Open Data Policy
When the US Government established its Open Data Policy it recognised that technology moves much faster than policy ever could. Often when writing policy for technology, agencies are stuck with outdated methods as soon as they publish new policies.
It therefore established Project Open Data on GitHub, a popular code-sharing website used by the ‘open source’ community. Project Open Data is an online collection of code, tools, and case studies to help U.S. agencies adopt the Open Data Policy . Anyone – government employees, contractors, developers, the general public – can view and contribute by editing the content, adding new pages, or adding tools.
At the onset, the General Services Administration is providing a moderator role – giving oversight and support, but over time,the vision is that contributors both inside and outside of government can be empowered to take on additional leadership roles.
The US government has made a public commitment to open data
The US Government has made strong public commitment to opening data in recent years. For example, on his first day in Office, President Obama signed the Memorandum on Transparency and Open Government, instructing government agencies to take specific actions to implement the principles of transparency, participation, and collaboration. In addition, the Office of Management and Budget published an Open Government Directive in December 2009 obliging agencies to expand access to information by making it available online in open formats. Finally, the Memorandum on Open Data Policy from May 2013 established a framework to help institutionalise the principles of effective information management at each stage of the information’s life cycle to promote interoperability and openness.
The US White House requires predicable URLs for committee documents and data
In order to ensure that members of the public have easy access to U.S. legislation, the U.S. White House enacted the ‘Standards for the Electronic Posting of House and Committee Documents & Data’ in 2011. These standards included a requirement for files to be posted using permanent URL links.
Open government data
- Create public listings of government data, and audit data availability and management
- Proactively engage with and support data users
- Require that open data commitments apply to all organizations handling public data
- Create or appoint an oversight authority
- Establish new legal rights to empower the public