Towards understanding policy design through text-as-data approaches: The policy design annotations (POLIANNA) dataset

Despite the importance of ambitious policy action for addressing climate change, large and systematic assessments of public policies and their design are lacking as analysing text manually is labour-intensive and costly. In a project funded by the Swiss National Science Foundation and ETH Zurich, we made a first step in contributing to overcoming this bottleneck: We created POLIANNA, a training dataset of policy texts from the European Union (EU) that are annotated based on theoretical concepts of policy design, which can be used to develop supervised machine learning approaches for scaling policy analysis. The dataset consists of 20,577 annotated spans, drawn from 18 EU climate change mitigation and renewable energy policies and is published open source to facilitate follow-up projects. Concretely, we developed a novel coding scheme translating existing taxonomies of policy design elements to a method for annotating text spans that consist of one or several words. An extensive description of the coding scheme, the annotated corpus, and an analysis of inter-annotator agreement complements the training dataset. There are various potential applications that can build on POLIANNA. Primarily, we envision this database to be used for building tools that help with manual coding of policy texts by automatically proposing paragraphs containing relevant information. As fully automated assessment of policy design characteristics a long-term goal rather than an imminent reality, we understand our effort as being an important first step along this way, hoping to encourage practitioners and researchers alike to pick up elements of our framework in their upcoming endeavours.