What is this report about?
Environmental Standards Scotland (ESS) is committed to evaluating the impact of its work. Many factors influence environmental change, and ESS’ contribution is often indirect. As a result, we use a theory‑of‑change‑based approach to articulate and track how our work is expected to lead to improvements to environmental law and regulation and, ultimately to improved environmental outcomes.
ESS commissioned this independent report to:
- review whether its existing theory of change approach reflects best practice
- assess the longer-term impact of a previous ESS project through an evidence‑based case study, and
- develop a repeatable approach for future impact assessments
The case study examined ESS’ Air Quality Investigation Improvement Report, published in 2022. As one of ESS’ first major investigations, it provides an opportunity to consider the changes that have taken place since its publication.
Why should people be interested in this report?
As an independent public body, ESS must be able to demonstrate how its work contributes to better compliance with, and improved effectiveness of, environmental law in Scotland – and how that contributes to better environmental outcomes. This report supports that accountability.
Environmental improvements can take many years to become visible. Changes in governance, policy, organisational priorities and professional practice may happen first. Assessing these intermediate changes helps ESS understand whether its work is moving in the right direction, even when it is too early to identify a direct environmental effect. The assessment also identifies practical ways to strengthen the reach, reception and long-term influence of ESS’ work.
What areas of environmental law are relevant to this work?
This commissioned report focuses on how ESS assesses its own impact and is not directly linked to a specific piece of environmental law.
What have ForgeFront done on ESS’ behalf?
ForgeFront, in collaboration with the Edinburgh Climate Change Institute (ECCI), carried out the commission in three linked stages.
- Scoping and case study selection: Following a review and workshop, the AQIIR was selected because of its strategic importance, range of stakeholders and potential to show intermediate outcomes. The team also developed a selection method that ESS can use for future case studies.
- Review of ESS’ Theory of Change approach: More than 100 government, academic and public-sector sources were assessed to identify good practice. ESS’ guidance and training materials were then evaluated against this evidence.
- Impact Assessment: The team completed 16 in-depth stakeholder interviews, invited wider contributions through an online survey and independently analysed publicly available nitrogen dioxide (NO₂) monitoring data from across Scotland.
What was the result?
The review found that ESS already has a solid foundation for measuring impact and praised our theory of change guidance and training.
The review concluded that ESS’ air quality investigation acted as a catalyst for continued attention and action. Some local authority stakeholders said that it helped accelerate or improve Air Quality Action Plans, strengthen technical and administrative standards, increase accountability and raise the profile of air quality with senior leaders. The investigation also provided an independent and authoritative reference point for people seeking action on air pollution.
However, the report does not conclude that the Air Quality Investigation Improvement Report directly caused improvements in Scotland’s air quality. Some of ESS’ recommended changes were already under way, and other factors, particularly Low Emission Zones, has also influenced air quality.
The quantitative analysis found no national-level change in NO₂ that could be attributed to ESS. This is not unexpected, as only around three years of post-publication evidence was available for analysis. The report therefore presents ESS’ contribution to date mainly as supporting momentum and influencing governance, policy, priorities and organisational behaviour.
What happens next?
The report suggests light-touch refinements to ESS’ current theory of change approach, which will be incorporated into our standard operating procedures.
ESS plan to commission a further review of its impact in 2028-29.
Click here to read the report in full.
