Diagnostic Tools and Performance Indicators

From the introduction: "This paper examines a diagnostic tool developed for [United Nations Educational, Scientific and Cultural Organization] UNESCO to identify indicators that assess how the media can contribute to democracy. This project reviewed existing systems of assessment and generated a new, composite approach to analysing the media’s role.... This paper goes on to consider how this initially broad diagnostic tool could be used to drill down and identify more focused performance indicators that can assess the impact of specific media programmes." The paper is part of the publication of papers for a conference on "The Role of the News Media in the Governance Agenda", which was co-sponsored by the World Bank Communication for Governance and Accountability Program (CommGAP) and the Joan Shorenstein Center on the Press, Politics and Public Policy at the John F. Kennedy School of Government, Harvard University, Boston, United States (US).

The need for effective indicators for grant-based freedom of expression media projects is underscored in this document by donor support figures of US$1 billion a year from donors outside the US and an additional US$142 million spent by the US. In order to give context for this assessment tool, the paper describes the dominant debates on the media roles of facilitating communication, advocating, acting as watchdog, and offering symmetry to the "'natural asymmetry of information' between governors and governed, and between competing private agents." The document recognises a need for nurturing media in new or restored democracies and for standard setting, so that mass communication channels are free and independent of established interests, and access to media is widespread. Hence, independence and access are two of the starting points for measurement of media development. The paper also recognises a role for state intervention to guarantee pluralism and diversity, such as provision for public and community-based broadcasting, as well as training for capacity building among journalists and media managers and the development of professional associations. Additionally, there is a need for infrastructure capacity so that the means of communication is diverse and widely accessible.

UNESCO's exercise examined 26 initiatives studying communication in order to find a frequently-cited set of measurable indicators. The two most frequently cited are the IREX Media Sustainability Index and the annual Freedom of the Press Survey published by Freedom House. To develop its selection of indicators for a toolkit, UNESCO chose not to use a comparative method, but to assemble diagnostic tools, enabling countries and organisations to customise the indicators appropriately in order to identify gaps in media development that need strengthening. It chose 5 categories of media indicators:

• Category 1: A system of regulation and control conducive to freedom of expression, pluralism, and diversity of the media: existence of a legal, policy, and regulatory framework which protects and promotes freedom of expression and information, based on international best practice standards and developed in participation with civil society.
• Category 2: plurality and diversity of media, a level economic playing field, and transparency of ownership: the state actively promotes the development of the media sector in a manner which prevents undue concentration of ownership and ensures plurality and transparency of ownership and content across public, private, and community media.
• Category 3: media as a platform for democratic discourse: the media, within a prevailing climate of self-regulation and respect for the journalistic profession, reflect and represent the diversity of views and interests in society, including those of marginalised groups. There is a high level of information and media literacy.
• Category 4: professional capacity building and supporting institutions that underpin freedom of expression, pluralism, and diversity: media workers have access to professional training and development, both vocational and academic, at all stages of their career, and the media sector as a whole is both monitored and supported by professional associations and civil society organisations.
• Category 5: infrastructural capacity is sufficient to support independent and pluralistic media: the media sector is characterised by high or rising levels of public access, including among marginalised groups, and efficient use of technology to gather and distribute news and information, appropriate to the local context.

This set of indicators, with its subcategories, identifies those issues by country where change is most needed and where active donor intervention is likely to make the most significant impact and so is likely to be of most value in shaping donor or implementer intervention. But in itself, it does not provide a methodology that can help assess the impact of specific interventions - it does not represent a set of performance indicators. The author states that within the several levels of assessment discussed here, sets of indicators are under development. The first level is examining existing donor activities, which "rarely explicitly recognise in their strategies and programming the direct links between international obligations to protect freedom of expression and efforts to promote media development and democracy." The next level is assessing the impact of implementing organisations. The document recommends an approach to evaluation of projects that is ongoing, systematic, and built into internal project management. It recommends tying assessment to stated goals and objectives, programme activities, and deliverables, using three components - the congruence of the different levels, internal quality checks, and external verification. It recommends "light" external evaluation for quality control. "This approach to evaluation does not involve the production of generic performance indicators - rather it requires the development of a system of project management which ensures congruence between the different layers of the organisation and develops internal and external indicators appropriate to the deliverables." It concludes that evaluation is currently a work in progress: "Evaluation is an evolving tool and we must expect it to evolve and change shape as it encounters the real world."