Environmental Standards Scotland (ESS) is an independent body set up to ensure the effectiveness of environmental law, and prevent enforcement gaps arising from the UK leaving the European Union.
More about usOur aim is to publish as much information as we can about the work that we do.
In this section you will find various publications relating to our corporate governance, investigations and analytical work.
You will also find all minutes from our Board and Audit and Risk Committee meetings.
Our workOne of the ways in which we carry out our function is to investigate environmental concerns made to us.
In this section, find out what we can investigate, how to raise a concern and what cases we are currently working on.
Raise a concernKeep up to date with all the latest news about Environmental Standards Scotland (ESS) including latest publications, latest videos, blogs, investigation launches, calls for evidence and related information.
News
This guidance document has been produced to ensure consistent and transparent approaches to undertaking high quality analytical and monitoring work.
June 2026
Strategy and Analysis – Operational guidance for Analytical process – June 2026.pdf
Download pdf (1.52 MB)
This document may not be fully accessible.
ESS – Environmental Standards Scotland
ET – Executive Team
SA – Strategy and Analysis
ISC – Investigations Standards and Compliance
CSC – Corporate Services and Communications
DPSIR – Drivers, Pressures, State, Impact, Responses
GSS – Government Statistical Service
SIO – Senior Investigations Officer
eNGO – environmental non-governmental organisation
1.1 About the Guidance
Environmental Standards Scotland (ESS) is an independent public body, set up to ensure environmental laws and standards are adhered to in Scotland, replacing the European Union’s scrutiny and enforcement role after Brexit. The Strategy and Analysis (SA) team undertake a range of monitoring and analytical work to ensure that we are well informed about environmental performance and identify issues of potential concern.
We may consider, assess and review data on the quality of the environment in Scotland, keep under review implementation of any international obligation of the UK relating to environmental protection and follow developments in international environmental protection legislation.
1.2 How to use this guidance
The processes outlined here should guide all of ESS’ analytical work. It:
i) describes roles and responsibilities of individuals and teams
ii) sets out the expectations of different analytical products
iii) provides links to key documents and further advice/guidance
It is essential that the analysis we produce is of the highest quality and that we minimise the risk of errors. This process is therefore essential to the reputation and credibility of ESS.
The process is mainly aimed at staff in the SA team. However, it is important that other staff in ESS are also familiar with the process as they may commission analysis and/or utilise its outputs.
1.3 Review and updates to procedure
This procedure will be kept under review by the Head of SA with revisions to be incorporated bi-annually if required. Members of staff are encouraged to flag any suggestions/required changes which they believe can improve the way ESS operates.
2.1 General ESS Powers and Obligations
i) Delegated Authority
Only the Board is given statutory authority by the UK Withdrawal from the European Union (Continuity) (Scotland) Act 2021 (the 2021 Act); however, as the Board would not have the time to physically review or authorise every decision that requires to be taken, it has agreed to delegate areas of responsibility to ESS staff. In legal terms, this means the actions of a member of staff acting with delegated authority are the actions of the Board.
ii) Legal Framework
The Board draws its authority from the 2021 Act, which both enables the actions of the Board and limits its powers. Each section of this guidance sets out the parts of the 2021 Act that is relevant to our work. As a public body, ESS is also subject to both the Freedom of Information (Scotland) Act 2002 (FOISA), the Environmental Information (Scotland) Regulations 2004 (EIR) and the Data Protection Act 2018 (DPA).
2.2 ESS principles
The following principles drive all of ESS’s work, including our analytical work.
2.3 Standards and Principles for producing Analysis
In addition to the ESS principles listed above we draw on the guidance in the Analytical Quality Assurance (AQuA) Book and, for our quantitative analysis, the Code of Practice for Statistics.
AQuA Guidance
The Analytical Quality Assurance (AQuA) Book provides guidance on producing good quality analysis for government to support well informed decision-making that creates better outcomes.
Key principles of the AQuA book we apply within our analytical work include;
i) proportionality of response: the extent of the analytical Quality Assurance effort should be proportionate in response to the risks associated with the intended use of the analysis, including financial, legal, operational and reputational impacts. In addition, analysis that is frequently used to support a decision-making process (or the conclusions of which are then used in future work) may require a more comprehensive quality assurance response
ii) assurance throughout development: quality assurance considerations should be considered throughout the life cycle of the analysis and not just at the end. Effective communication is crucial when understanding the problem, designing the analytical approach, conducting the analysis and relaying the outputs
iii) verification and validation: analytical quality assurance is more than checking that the analysis is error-free and satisfies its specification (verification). It must also include checks that the analysis is appropriate, i.e. fit for the purpose for which it is being used (validation)
iv) analysis with RIGOUR: quality analysis needs to be Repeatable, Independent, Grounded in reality, Objective, have understood and managed Uncertainty, and the Results should address the initial question robustly. It is important to accept that uncertainty is inherent within the inputs and outputs of any piece of analysis. It is important to establish how much we can rely upon the analysis for a given problem.
2.4 Voluntary compliance with the code of practice for statistics
ESS follow the principles of intelligent transparency and are voluntary adopters of the Code of Practice for Statistics. Our Statement of Voluntary Application sets out our commitments to applying the principles of Trust, Quality and Value as well as our approach to errors and corrections. We will continue to use the code to guide our approach and will publish an updated statement on an annual basis to demonstrate our commitment to continuous improvement.
In house training materials are available and questions on our application of the Code of Practice should be directed to the Head of Data and Quantitative Analysis.
3.1
A theory of change is a way to explain how and why a desired change is expected to happen in a specific context, linking activities to outcomes and impacts. In an ESS context, a theory of change approach sets out the intended improvements to environmental law and regulation as a result of our work, the chain of events that is expected to result in improvements, any key assumptions that underpin our theory of change and the wider context. Each theory of change will also identify specific indicators that ESS will use to track improvements in areas of the environment in which ESS has acted.
3.2
In our Strategy 2026 – 2031, ESS commit that for each of our major investigations and analytical projects that make recommendations for improvement, we will prepare a theory of change. This theory of change will explain how our work is expected to contribute to improved environmental outcomes through improved compliance with and better implementation of effective environmental law. Theories of change will be published alongside our analytical reports.
3.3
In addition to these indicators, we will commission detailed case studies on a three-yearly basis to examine the long-term outcomes of our work and ESS’ contribution to changes in law, policy and regulation. The first of these case studies will be published in summer 2026.
3.4
More detailed guidance is available in our Theory of Change Standard Operating Procedure and in internal training materials.
The Strategy and Analysis team produce a range of analytical products as ilustrated in Figure 3.1 and the purpose and scope of each of these products is summarised in this section.
The default audience of analytical products is assumed to be interested and semi-informed on environmental topics. Mixed stakeholders including MSPs, public bodies, environmental non-governmental organisations (eNGOs).
Figure 1 Analytical products produced by S&A team. (Post-intervention analysis includes tracking outcomes specified in theories of change and long-term impact case studies)
4.1 Foundational work (in development)
Purpose: Broad review of high-level topic area (e.g. Resources, Nature, Climate Change or Water) to build understanding and to allow further prioritisation and scoping to inform analysis. The primary output is a prioritised list of potential topics for further scoping or analysis, though other outcomes are possible (see below).
Audience: Mixed. Analytical output is internal and normally unpublished, though we may produce outputs of evidence that has been synthesised. Summary of responses to a call for evidence is external and published.
Type of report: Short, summary report setting out prioritised list of potential topics for further scoping or detailed analysis.
This should include
While formal theories of change are developed alongside recommendations there should be consideration of how ESS is likely to be able to add value through changing behaviour or processes at an earlier stage of the work.
It is possible that foundational work will include a call for evidence to identify key areas of concern, support further engagement with relevant stakeholders, highlight gaps or synergies and support further prioritisation. Where a call for evidence takes place a summary of responses will be published.
Out of scope: Budget (unless foundational work includes commissioned out elements), communications – lines to take etc (we may wish to issue high-level communications on what we are doing for the topic). Updates on prioritised outputs can be provided to wider ESS staff at any point; however this is most likely to be appropriate when undertaking further scoping or detailed analysis (see relevant sections of guidance).
4.2 Scoping reports
Purpose: Review of topic leading to one of four possible outcomes (see below). If the outcome is that further analysis is required, scoping should identify the specific issues to focus on. Where foundational work on a broad topic area has taken place more detailed scoping may follow on one or more sub-topics. Topics for scoping may be identified through foundational work or other routes, and the focus will remain high-level. Scoping reports should allow for an assessment of the value that could be added by more detailed analysis and aid in the prioritisation of future work.
Audience: Internal. Unpublished.
Type of report: Short, summary report explaining why the topic is important (in a page with links to other sources) and then some visuals (e.g. legislation/policy map, DPSIR summary, data sources, trends). Set out next focus with information to explain why topics not included to manage risk. If proposing further analytical work this should include project specification/proposal to give a sense of what resource would be required. While formal theories of change are developed alongside recommendations there should be consideration of how ESS is likely to be able to add value through changing behaviour or processes at an earlier stage of the work.
Out of scope. Budget (unless scoping commissioned out), communications – lines to take etc. ESS’ legal advisor should be consulted at the outset and updates should be provided to Corporate Services and Communications (CSC) and ISC though quarterly C band meetings.
Key roles. Small project team to discuss key findings and next steps. Commissioner – One of C1 for Policy, Science or Quantitative Analysis. Analytical coordinator – mostly at B band.
Duration. 2-3 months.
Quality assurance and fact checking. Light touch quality assurance of references. No external fact-checking.
External engagement. Yes. To understand what stakeholders are concerned about and where the risks are. However, stakeholder views should not drive priorities unless supported by evidence.
Board/ET engagement. Updates on progress and conclusions provided through S&A update papers for information only. Decisions on next steps lie with Head of S&A.
Possible outcomes. 1) No further action. 2) Further detailed analysis of one or more (prioritised) topics. 3) Issue passed to investigations (occasional). 4) Issue raised by letter with Parliament or a public authority (occasional).
4.3 Legislative Rapid reviews
Purpose. Horizon scanning.
Audience. Internal. Unpublished.
Type of report. Legislative rapid review.
Out of scope. Budget (unless scoping commissioned out), communications – lines to take etc. Detailed legal advice generally not required but may be sought on more complex areas and updates should be provided through quarterly C band meetings.
Key roles. Commissioner – Head of Policy Analysis and Horizon Scanning. Analytical coordinator/lead analyst – B band from policy team.
Duration. 2-4 weeks.
Quality assurance and fact checking. Light touch quality assurance within policy team. No external fact-checking.
External engagement. Not normally required.
Board/ET engagement. Updates on progress provided through S&A update papers for information only.
Possible outcomes. None. To provide background/context for other analytical products.
4.4 Analytical reports
Purpose. In-depth analysis of a defined, narrow topic. Will always follow scoping work, and the area of environmental law and focus (compliance and/or effectiveness) should be clear.
Audience. External published.
Type of report. Analytical report, synthesising evidence from a range of sources (quantitative, science, policy) and reaching conclusions relating to compliance with and efficacy of environmental law.
Scope. Most will not require a budget, but the option should be available. Communications and legal advice should be sought at outset and then at key stages in the process.
Key roles. Full project team with regular meetings and input on key findings and conclusions. Commissioner – One of C1 for Policy, Science or Quantitative Analysis. Analytical coordinator/lead analyst – mostly at B3 level.
Duration. Varies but likely to be at least six months and often a year or more.
Quality assurance and fact checking. Full internal quality assurance approach. External fact-checking.
External engagement. Stakeholder engagement expected but timing of this may vary depending on the topic. Stakeholder views should not drive priorities unless supported by evidence.
Board/ET engagement. Detail of specification shared at start for information. Nominated Board members engaged at key points. Updates on progress and conclusions provided through S&A update papers to ET and Board. Role in agreeing final recommendations (if any).
Possible outcomes. 1) No further action. 2) Further or ongoing analysis of one or more topics (prioritised). 3) Recommendations made to public authorities. 3) Issue passed to investigations (occasional).
4.5 Consultation responses/call for evidence responses
Purpose. To provide high quality analysis to inform ESS’ position in response to a consultation or call for evidence.
Audience. External published.
Type of report. Consultation response.
Scope. Generally, does not require a budget. Legal and communications updates should be provided and advice sought as appropriate.
Key roles. Commissioner – Head of Policy Analysis and Horizon Scanning. Analysis delegated as required. Lead analyst and a second quality assurance (QA) role normally sufficient though more roles may be included if multi-team response.
Duration. 4-6 weeks.
Quality assurance and fact checking. Within-team quality assurance though may need fuller approach if consultation requires a multi-team response. No external fact-checking.
External engagement. Not normally required though often part of wider engagement on associated topic.
4.6 Investigation support
Purpose. To provide high quality, in-depth analysis of a narrow topic to support an investigation or pre-investigation.
Audience. Internal reports, potentially contributing to eventual external investigation report.
Type of report. Templates for analysis from individual teams e.g. quantitative analysis, scientific evidence review or policy analysis. Occasionally may be required to synthesise multi-team response.
Scope. Generally, does not require a budget unless commissioned out. Legal advice if appropriate.
Key roles. Commissioner – Head of Investigations, Standards and Compliance (ISC). Analytical coordinator – Identified within S&A by C2. Project team depends on complexity. If single team response, then informal team of commissioner, coordinator and assurer normally sufficient. Multi-team response may require more formal team.
Duration. Variable.
Quality assurance and fact checking. Within-team quality assurance though may need fuller approach if consultation requires a multi-team response. No external fact-checking.
External engagement. Not normally required although may feed into requests made of stakeholders by the investigations team.
Board/Executive Team(ET) engagement. As per ISC own ET/Board engagement on wider Investigation. Updates on progress and conclusions provided through S&A update papers to ET and Board.
4.7 Advice and Briefing notes
Purpose. To provide high quality analysis in response to Board/ET requests.
Audience. Generally internal reports but potential to be published in some cases.
Type of report. Likely to be based on templates for analysis from individual teams e.g. quantitative analysis, scientific evidence review or policy analysis. However, format may change depending on scope of request. Occasionally may be required to synthesise multi-team response. May feed into Board/ET papers.
Scope. Generally, does not require a budget. Legal advice should be sought if appropriate. Communications advice should be sought if to be published.
Key roles. Commissioner – any part of ESS. Analytical coordinator identified by Head of S&A. Need for project team depends on complexity. If single team response, then informal team of commissioner, coordinator and assurer normally sufficient. Multi-team response may require more formal team.
Duration. Variable.
Quality assurance and fact checking. Within-team quality assurance though may need fuller approach if requires a multi-team response. Generally, no external fact-checking but may be required if report intended for publication.
External engagement. Not normally required but depends on scope of request.
Board/ET engagement. Detail will vary and this should be agreed with Head of S&A.
4.8 Post-intervention analysis (in development)
As ESS matures, we anticipate the requirement for post-intervention analysis on previously analysed topics to see if the situation has changed.
This could involve
5.1 Analytical priorities
ESS priorities are determined by our five-year strategy which is approved by parliament. For 2026-2031 the focus of our new proactive work will be on four interconnected priorities;
The strategy and analysis team will conduct foundational work on each of these priorities to inform which sub-topics are taken forward to scoping analysis.
In addition to work on these priority areas, we will continue to monitor and follow-up on our previous scrutiny work, for example, on air and water pollution, soils and access to environmental justice to ensure that the necessary action is taken.
We may also choose to scrutinise thematic or cross-cutting issues that span more than one priority (for example, data gaps).
Where a piece of analysis has significant resource implications and/or entails significant reputational risk it may be necessary to agree the scope, focus and proposed approach to carrying out the work with the Executive Team.
5.2 Foundational work
Where a topic is new to ESS or where sufficient time has passed that we need to re-examine the evidence base foundational work will be carried out. This is a broad review of one of the priority topics defined by the strategy to develop a prioritised list of potential topics for further scoping or analysis. Foundational work is expected to take around 6-9 months (see Figure 2).
Figure 2 Illustrative timeline and steps for foundational work
5.3 Scoping analysis
A scoping project should proceed any larger analytical project and is expected to take around 2-3 months (See Figure 3).
A scoping project will lead to one of four possible outcomes; 1) a decision that no further work is currently required, 2) a referral to investigations, 3) a letter to parliament or a public body, or 4) initiation of a larger analytical project leading to an analytical report.
Figure 3 Illustrative timeline for scoping analysis
5.4 Steps in conducting detailed analysis
Every project should allow work up front to scope the work required and identify a realistic timeframe. The time taken to conduct detailed analysis will vary according to the specifics of the project, but plans should allow for the steps and rough timeframes outlined in Figure 4.
Engagement with nominated Board members will occur though a briefing meeting with the project team at the outset of the project and then a meeting to discuss conclusions with the project team and the Head of S&A prior to analytical sign off.
Figure 4 Illustrative timeline for detailed analytical report
5.5 Drafting final report for an analysis project
The analytical coordinator will write the final report, bringing together wider context and synthesising quality assured findings of each analysis strand.
The report should provide some information on methodology, strengths, limitations and uncertainty (and its sources). This can be described in qualitative terms e.g. moderate uncertainty, high confidence etc. It should include a statement of assurance from the analytical assurer in the QA record. See analytical assurer within Table 7.1 Roles and responsibilities within analytical projects.
The report will include conclusions and key findings that have been discussed and agreed with the project team.
It should use the ESS analysis report template and be subject to internal Quality Assurance of Analysis before sharing with stakeholders.
Recommendations will not be included in the version sent for external factchecking and it should be made clear in request that we’re seeking a fact check and comments on any misrepresentation but not wider commentary on the information or language used.
Following the external fact-check further quality assurance will be undertaken of any changes that result and their impact on key messages.
When sending for comments, ask for feedback in clear categories e.g. critical/factual/to note.
Make use of proof-reading checklist at appropriate points e.g. before sending to stakeholders/Board.
5.6 Development of recommendations, Theory of Change and key indicators to monitor
Key findings will be agreed between the project team and the commissioner and discussed with the nominated board members.
Recommendations will not be made until the report is finalised with quality assurance and fact-checking having been undertaken.
The S&A team leads, and the analytical coordinator will discuss and agree recommendations. This group will have the responsibility of ensuring that there is a sufficient threshold of evidence and that the issue is serious enough to warrant a recommendation.
Theories of change should be developed alongside the recommendation and outcome indicators for monitoring should be identified (see 5.8 monitoring following analysis).
Recommendations will be of the form: review/change/do. If the issue doesn’t fit within our review/change/do approach, this may indicate the need for further work to allow it to be passed to investigations.
Not all analysis projects will result in recommendations.
5.7 Indicative timeline to publication
As illustrated in Figure 4 once the report has been finalised with key findings and conclusions and external fact-checking, around three months should be allowed prior to publication. The activities which need to be completed during this time are outlined in Table 5.1.
Table 5.1: Steps to publication once report is finalised
| Activity | Who? | How long to allow |
| Development of recommendations, theories of change and outcome indicators | Agreed by project team and the commissioner along with other C band staff in S&A followed by discussion with nominated Board members, legal and communications leads. | 4 weeks |
| Approval by Board/nominated members of strategic risks and messaging (dependant on significance) | S&A paper to ET and then the Board | 3-4 weeks |
| Final sign-off by CEO and confirmation of planned publication date | S&A summary of changes to CEO | 2 weeks |
| News release and lines to take | CSC Communication lead with input from S&A leads | 2-4 weeks |
| Comms plan produced by Senior Communications Officer and discussed/agreed at Comms meeting | CSC Comms to lead with input from S&A leads | 2-3 weeks before publication |
| Communications meeting with CEO and Chair | CEO, Chair, CSC C2, Senior Comms Officer, S&A C2, project analytical coordinator | 2 weeks before publication |
| Proofreading and preparing for html | Proofreading to be carried out by a member of S&A who hasn’t been heavily involved in drafting the report. Conversion to html to be carried out by CSC. | 2 weeks |
5.8 Monitoring following analysis
Not all analytical topics will result in monitoring.
Where a topic does result in monitoring it should be clearly defined and related to a specific topic or action. It should also include details of what is being monitored, when and how often and by which function within S&A.
Recommendations will be of the form: review/change/do and will be supported by Theories of Change. Monitoring requirements for recommendations should be clearly set out at this stage and agreed as part of the development of these elements.
Note that this is non-statutory monitoring of developments or changes to policy, legislation or evidence related to analytical topics. This is different to any future statutory monitoring that ESS may be asked to undertake or monitoring following interventions made by the ISC team.
6.1 Within Strategy and Analysis
In addition to project team meetings (Table 7.2) we hold six-monthly analytical review meetings for the S&A team.
At the analytical review meetings all ongoing and planned projects will be discussed. This gives us the opportunity to share information across projects and across the three analytical teams (policy analysis, quantitative analysis and data, scientific analysis) including:
These meetings will also be used to review the analytical process as needed and forward work plan.
6.2 Requesting Quantitative, Scientific or Policy input to a project.
Where specific quantitative, scientific or policy input is required for a project this should be requested using the analysis request form. The form is intended to be used to maintain a consistent and agreed record of the request, including details of timescales. It is important to be clear at the outset about the expected timescales and deadlines, both from the requester and the supplier(s) of the information. If there is a statutory deadline, or the work is in relation to a consultation / committee response, this should be flagged at the earliest opportunity.
6.3 Information sharing across ESS
While this guidance is primarily aimed at the S&A team we work closely with ISC and CSC.
As part of this we hold quarterly ESS C band analytical review meetings. The purpose of these meetings is to;
6.4 Passing issues to ISC
ISC can commission work from S&A. Similarly, while undertaking analytical work S&A may identify issues which fall within the remit of ISC (Error! Reference source not found.).
Although this should be a consideration in every project team meeting, issues are most likely to be identified towards the end of the analysis and may require further work to evidence the problem. Decisions about which issues should be passed from SA to ISC should be discussed and agreed with the Head of SA and the Head of ISC.
6.5 Executive team and board progress updates
ET are provided with a paper fortnightly by the Head of S&A updating on progress and current timelines to completion. A progress paper is also provided for each Board meeting.
Key messages/recommendations from projects included in these papers at appropriate points in project.
The Head of S&A has the delegated authority to sign off analysis reports. However, there should first be a discussion with nominated Board members of conclusions and recommendations and, if significance factors are present, the report may need to be discussed at a Board meeting. See Sign-off for Analytical Products 7.4.
7.1 Analysis roles
Analysis roles are set out in Table 7.1. These roles mirror those set out in the aqua book.
Table 7.1 Roles and responsibilities within analytical projects
| Role | Description |
| Commissioner |
|
| Analytical coordinator |
|
| Analytical assurer |
|
| Lead analyst(s) |
|
| Investigations lead |
|
| CSC lead(s) |
|
7.2 Project teams
Not every role is required for every analysis project but as a minimum the commissioner, analytical assurer and analytical coordinator are required. For larger projects (e.g. leading to a publication) CSC leads (legal and communications) will need to be included in the project team. If any work is being externally commissioned CSC should be included for budget and procurement purposes. For analytical reports an investigations lead will be required.
The commissioner and coordinator should discuss from the outset which other roles will be needed and invite these to form a project team. At an early stage, an initiation meeting should be set up to ensure a common understanding of the problem/purpose, type of product required, research questions (if required), context, scope, limitations, complexities and outputs required.
The team should agree on documentation requirements, proportionate to scope of project. Project teams remain in place to the point of agreeing key findings and conclusions.
Example project teams[1]
7.3 Project team meetings
The type and frequency of these meetings will vary according to the product being developed. For detailed analysis projects with a published output, the project team should meet regularly and no less frequently than every three months. For other products, the frequency of meetings should be discussed and agreed at the outset. Table 7.2 shows a schematic for expected project team meetings for scoping and analysis projects. The analytical coordinator is responsible for keeping a record of meetings and updating a key decision log.
Table 7.2 Suggested key meetings within a project lifespan
| Meeting | Suggested agenda | For which products? |
| Initiation |
|
All |
| Progress meeting(s) |
|
Scoping, analysis, others if needed. |
| Key finding meeting(s) |
|
Scoping, analysis, others if needed. |
| Post fact-check meeting |
|
Analysis, others if needed. |
| Concluding meeting |
|
All |
| Recommendations meeting |
|
Analysis, others if needed. |
Every project team meeting should consider whether any issues have been identified with sufficient evidence to be passed to the ISC team for consideration. Even where a potential topic of investigation is identified, further analysis may be required before it can be passed over.
For every product, there should be a concluding project team meeting which considers lessons learned. These should then be shared with the wider S&A team. For larger projects CSC can help with a lessons learned meeting.
7.4 Analytical sign-off
In addition to the quality assurance and analytical assurer roles, the three analytical function leads in S&A (Head of Policy Analysis, Principle Scientific advisor and Head of Data and Quantitative Analysis) should sign off every product produced (if relevant to their areas of expertise).
For detailed analytical projects it is likely that they will be involved in project teams throughout. However, this is not necessarily the case and unlikely for other products.
Therefore, relevant S&A function leads should be involved at key points in analysis:
This is particularly important where the original analysis is undertaken by someone outwith that function and they should ensure to engage the relevant team leads.
When sending reports for comment, areas for the analytical team leads to comment on should be clearly flagged. This should cover both where detailed analysis is presented and other parts of the report where quantitative analysis/science/policy have been referenced.
[1] These project team examples pre-date the appointment of the ESS legal advisor and we would expect they should be included in the project team for analytical projects.
8.1 General principles
The lead analyst(s) on the project will deliver the analysis with RIGOUR in mind:
QA should be ongoing throughout the project. Regular meetings with analytical assurer are encouraged to keep them updated. Within-team QA should occur as analysis is produced with the analytical assurer kept in touch and adding to this when the report is produced.
8.2 Use of DPSIR Framework
The ‘Drivers, Pressures, State, Impact, Responses’ framework is a problem structuring method that can be used to help conceptualise, prioritise and communicate an area of focus for analytical work
The initial entry point for scoping work may be a particular state (e.g. statutory target has not been met), impact (e.g. loss of biodiversity, reduced carbon sequestration) or response (e.g. specific legislation) that ESS has decided to scrutinise.
By considering the other categories within the DPSIR framework, it will allow us to scrutinise whether, for example, a specific piece of legislation (‘response’) is effectively targeting the pressures that drive the environmental state or impact, or the extent and effectiveness of responses to particular environmental states or impacts.
A simple DPSIR diagram showing how the proposed prioritised area of focus fits within the broader context of a particular environmental issue can serve as a useful communication tool to summarise the prioritisation process and justification for what is in/out of scope for an analytical project.
We are developing detailed guidance and training on DPSIR.
8.3 Quantative analysis
ESS voluntarily apply the principles of the Code of Practice for Statistics and will consider Trust, Quality and Value at the start of any quantitative analysis. Permanent members of the quantitative analysis team are badged by the Government Statistical Service (GSS) and follow professional guidance and best practice.
There will be a clear agreement between the commissioner, the project team and the Head of Quantitative Analysis and Data on the purpose of data analysis. The data team will then carry out initial scoping of data availability, quality and relevance to the question. Following this, next steps will be agreed with the commissioner, the project team and the Head of S&A considering the value of the analysis and the resource requirement.
We will clearly outline the methods and assumptions that have been made in our analysis including any caveats on their use or interpretation. This will increase the transparency of our quantitative analysis and allow us to increase the value of our work by allowing other organisations to replicate our analysis. Wherever possible we will perform our analysis using open-source software and publish our code on our GITHUB page.
ESS are voluntarily applying the Code of Practice for statistics. As part of this we limit access to pre-release statistics to provide assurance of independence. This is applied in a proportionate manner in line with also upholding our principles of cooperation and ‘no surprises’ as listed in our MOU’s. A template for recording pre-publication access to analysis is available which includes acceptable reasons for access, and the expectations that should be clearly communicated to the recipient of pre-publication access.
8.4 Reviewing scientific evidence
The scientific analysis team can support analytical projects by providing scientific evidence reviews. These can be provided in one of three formats. depending on needs and available time;
Scientific evidence reviews will consider the credibility, transferability and dependability of the research using the following questions as a guide.
Credibility:
Transferability: This is an assessment of the context of the research and its methods and the ability to transfer research findings to the setting, group or geography being considered in ESS’ analysis, for example:
Dependability:
See also Table 9.2 Key steps for QA of scientific evidence reviews.
8.5 Policy analysis
Policy and legislative analysis outputs include:
These should normally be undertaken by the policy analysis team using standard templates with QA and sign-off within team.
See Table 9.3 Key steps for QA of policy analysis.
8.6 Commissioned analysis
Where analysis is to be commissioned externally early discussion with the Head of CSC should take place to agree the most appropriate way to source, potential timescales and resource implications.
Budget implications should be discussed with the Head of S&A initially and then (depending on cost) the CEO, ET and/or Board.
Where analysis is to be commissioned externally it is vital that the reason for external commissioning is clear and that the purpose and expected output from the work are made clear at the outset. The detailed methodology and timeline to deliver the product should be proposed by the contractor and agreed at the initiation meeting – but, first and foremost, ESS must be clear on what question(s) it is seeking answers to and the scope of the analysis.
9.1 Our general approach to quality assurance
Much of our analysis will involve synthesising, interpreting and translating to new contexts existing literature and assessing the applicability to a new situation rather than new primary research.
Communication of the results and the associated uncertainties and limitations is very important. The analytical assurer See (Table 7.1 Roles and responsibilities within analytical projects) must be content that the final report presents a true representation of the analysis undertaken and evidence reviewed.
Analytical quality assurance involves verifying and validating the analysis – i.e. that the analysis has been conducted as planned and that it is the right analysis. The scale and scope of these activities need to be proportionate to the purpose and constraints of the analysis.
9.2 Quantitative quality assurance
Our approach to quantitative quality assurance (QA) is proportionate to the novelty and complexity of the statistics presented.
Table 9.1 Key steps for QA of quantitative analysis
| Lead analyst(s) – throughout | Within profession QA – throughout | Analytical assurer – engaged throughout but adds further QA at report stage |
| Adherence to relevant standards (e.g. coding standards) | Checks that standards adhered to | Is assured that standards have been applied |
| Quality assures the datasets feeding into analysis using template and takes account of quality issues, limitations and uncertainty identified | Spot checks that agree with lead analyst assessment of quality | Checks that the final report includes information on quality, limitations and uncertainty and agrees with assessment |
| Undertakes QA throughout analysis* | Undertakes further QA checks of any data transformations, code and outputs | Reviews the analyst QA log for assurance |
| Maintains log of verification and validation checks undertaken, at what points in process and any resulting changes | Updates log with further QA undertaken, at what points in process and any resulting changes | Checks that the QA undertaken is appropriate to the complexity and risk of the analysis.** |
| Sets up detailed peer review if analysis is especially complex | Supports lead analyst to review and implement results of peer review | Checks that peer review has been undertaken and comments implemented |
| Documents the analysis (either in code and a brief summary in report or in methodology notes if more complex analysis undertaken) as it is undertaken | Checks the code and/or methodology notes produced | Checks with lead analyst that appropriate information has been produced |
We will clearly outline the methods and assumptions that have been made in our analysis including any caveats on their use or interpretation. This will increase the transparency of our quantitative analysis and allow us to increase the value of our work by allowing other organisations to replicate our analysis.
Wherever possible we will perform our analysis using open-source software and publish our code on our GITHUB page.
9.3 Quality assurance of quantitative evidence sources
ESS does not collect original data, so our quantitative analysis is based on either publicly available datasets or, occasionally, data that is provided with us directly by public bodies (generally through an Information Notice).
When scoping data for our analytical projects, our preference will be to use accredited official statistics if available and appropriate for our intended purpose.
If the most relevant evidence is from administrative data, then members of the data
team will assess the data for quality and bias. In doing this, we will draw on the Administrative Data Quality Assurance Toolkit and will consider:
Where data is provided to us by public bodies or by those making a representation to us, we will encourage them to also make this public to allow our analysis to be repeatable by others.
In all cases the sources of the data used in our analysis will be referenced in our publications, alongside our assessment of the quality of data, including uncertainty and bias in estimates and impacts on appropriate interpretation and use. Where we have any concerns about bias or uncertainty in the underlying data, we will provide details of this alongside figures.
9.4 Scientific evidence reviews quality assurance
Table 9.2 Key steps for QA of scientific evidence reviews
| Lead analyst(s) – throughout | Within profession QA – throughout |
| Checks that literature is robust, timely and relevant to the project | |
| Considers the uncertainty, risk, limitations and constraints of the research reviewed and the implications for applying the research to a new context | |
| Checks that it has been sourced from the agreed academic databases | |
| Keeps a record of any issues or concerns about the quality, relevance or lack of supporting evidence in each source | |
| Assesses the credibility, transferability and dependability of each source* | |
| Maintains a log of quality assurance undertaken, at what points in process and any resulting changes | Updates log with further QA undertaken, at what points in process and any resulting changes |
9.5 Quality assurance of policy analysis roles
Table 9.3 Key steps for QA of policy analysis
| Lead analyst(s) – throughout | Within profession QA – throughout | Analytical assurer – engaged throughout but adds further QA at report stage |
| Checks that sources are robust, timely and relevant to the project | Spot checks that agrees with lead analyst assessment of sources | Is assured that robust approach to evidence used has been applied |
| Refers to, updates, or where required develops, legislative rapid review on topic ensuring that it includes all relevant legislation and is up to date | Spot checks that agrees with lead analyst assessment of legislation and policy | Reviews the analyst and QA approach for assurance as required. |
| Undertakes QA throughout analysis | Undertakes further QA checks of key elements of work | Reviews the analyst and QA approach for assurance as required. |
| Keeps a record of any issues or concerns about policy, regulatory or legislative evidence used | Updates log with further QA undertaken, at what points in process and any resulting changes | Checks that the QA undertaken is appropriate to the complexity and risk of the analysis |
| Ensures that appropriately rigorous review of policy, regulatory and legislative developments has been undertaken to support the project through horizon scanning and targeted research | Spot checks that agrees with lead analyst assessment of legislation and policy | Reviews the analyst and QA approach for assurance as required |
9.6 Quality assurance of grey literature
ESS’ analysis of topics regularly considers grey literature as part of the evidence base. This includes a range of material that is produced outside of traditional academic (or commercial) publication routes such as reports, media articles, strategies, etc.
Many of ESS’ analytical projects will bring together and synthesise analysis from multi-teams into an overall report with additional background/contextual information and conclusions.
Whilst each of the individual products will be subject to within team quality assurance, we need to ensure that the analytical coordinator applies QA thinking while producing the final report before this is passed to the analytical assurer.
The coordinator should consider the sources used and make a reasonable assessment of their relevance and potential for bias. The assurer should check that they would reach the same conclusions from the same evidence.
Particular red flags would be conclusions/recommendations based on only one evidence source (where we should be confident it is good enough) and conclusions/ recommendations based only on grey literature from non-official sources.
Analysts should use the grey literature QA checklist to inform sources included.
9.7 Errors and Corrections
We have robust quality assurance processes in place to minimise the risk of errors occurring. However, even with these processes in place, occasional errors are still inevitable.
If a potential inaccuracy is spotted, either internally during follow up or suggested by a stakeholder, this should be shared with the Head of Strategy and analysis and the relevant analytical function leads (Principal Scientific Advisor, Head of Data and Quantitative Analysis, and Head of Policy Analysis). We will welcome and investigate any suggestions of inaccuracy in our publications and issue corrections if necessary.
Our error corrections policy sets out how we will handle errors in a timely, transparent, and consistent manner.