Feeding Back Results
Introduction
This guidance is designed to make the specification and procurement of ITS tools easier.
In order to do this, experiences and results from actual implementations have been presented. Evaluation of your ITS tools is useful to:
- Justify National & Local Government expenditure on the projects;
- Demonstrate the benefits of ITS tools (financial, socio-economic and environmental/ecological) ;
- Demonstrate best value;
- Achievement of policy goals; and
- Increase understanding of the impacts of ITS services.
However, if the results and experiences of authorities are not shared, then time and effort will be wasted and best practice will often not be adopted. This guidance provides assistance and a framework on evaluation which, if used, will enable the enhancement of this guidance through feedback received from authorities.
The guidance provided will ensure that results are easily understandable and comparable.
Authorities who wish to submit their results for inclusion in the toolkit can contact us using the ITS Toolkit Feedback Form.
Guidance on Monitoring, Evaluation and Reporting
Overview
The performance monitoring, evaluation and reporting of results recommended in this guide are designed to meet an organisational need for accountability and transparency, throughout the lifetime of projects and to demonstrate, particularly for schemes receiving financial support from the Government, that monies have been wisely spent. The recording of evaluation results and decisions made as a result provide valuable evidence to inform future scheme development and improve future decision making.
As highlighted within the ITS Tools Directory they can be used to demonstrate the benefits of individual applications, but can also be used to demonstrate the benefits of integrated approaches, where ITS may be deployed and operated across geographic and institutional boundaries.
The reporting of results in the way advocated by the guide will improve the collective body of knowledge available about the costs and benefits of deploying ITS and enhance the guide for the future.
The adoption of a standard set of criteria associated with impacts on the environment and the economic and social consequences of deployment are advocated. These should be divided into a number of sub-criteria and indicators, to reflect the wide variety of impacts possible from schemes of a different nature. These indicators will give decision makers a clear, systematic and consistent basis for selection between options.
It is of great benefit to the organisation to make the results of the evaluation widely available. This will foster best practice. The reporting of results for a wider audience will form a basis for assessing the true, longer term merits of the investment made in ITS. Evidence arising from applying the Planning Guide can then be used to build a continuous data set of up to date information on actual costs and benefits. This will strengthen the body of evidence available to those who apply the guidance in the future.
As well as the direct participants, “Customers” for the results of projects also include a broad audience of potential users of ITS. For them, it is important to know how the various systems and applications perform, how much they cost to buy and to operate, and what impacts they might produce if implemented locally.
This sort of knowledge is not directly available from any one project alone. Without appropriate synthesis of project results, decision makers (those who will manufacture, own, operate or use ITS systems in the future) will be limited in the evidence available to them on the relative advantages of comparable systems and on which investment may be the most appropriate for their particular environment and conditions and for their own policies/purposes.
It is important, therefore, that evaluation results are:
- Transparent;
- Easily understandable; and
- Can be compared easily with other results.
Performance indicators can be used which reflect criteria for success in achieving specific aims and objectives. It is important that performance indicators reflect the services delivered by the ITS tool rather than simply a reflection of the technology used. Where appropriate, common indicators are advocated, which can be used across projects, for example, lives saved. However, specific data sources and evaluation tools vary between projects.
Those organisations entering the market for the first time may wish to focus more on how to measure the performance of individual applications of ITS, while those authorities already extensively using ITS will most likely wish to focus more on how to measure overall performance on a strategic level, as well as on an individual application basis. This will include measurement of integrated legacy/ITS equipment.
Impacts can be expressed in a range of different ways, again reflecting the diverse nature of options. They may be expressed:
- qualitatively – using terms, such as low/medium/high to describe the main impacts;
- quantitatively – using numbers to measure the scale of impacts; or
- as a ranking system – using, say, a 7-point ordinal ranking.
Whichever approach is adopted for expressing an impact, care should be taken to ensure that undue prominence is not given to any one type of effect, or to those benefits expressed in monetary terms compared with those that cannot be measured in this way.
Assessment Frameworks are advocated as an aid to decision making. They are an excellent means of making appraisal processes transparent and giving decision makers a clear, systematic and consistent basis on which to found their decisions.
In developing assessment frameworks for such a purpose, due regard must be given to the need for:
- understanding the underlying problem that is to be addressed (identifying exact user needs);
- identifying the options available to the decision makers (the measures which might address the perceived problem);
- appraising options (to determine the extent to which they address the problem as cost effectively as possible); and
- understanding the criteria for options to represent good value for money (identifying appropriate indicators and the relative importance in both monetary and non-monetary terms).
Recommendations on monitoring and evaluation reflect closely the appraisal guidance within the Department for Transport’s web based Transport Assessment Guide (WEBTAG). The use of appraisal frameworks in the testing and appraisal of options is presented as part of a 15-step process within WEBTAG, leading to an implementation programme.
The advice in this Planning Guide is predicated on the basis of using such frameworks to develop monitoring and assessment programmes, reflecting the aims and objectives for deployment and recording and assessing project outcomes through appropriate indicators of success.
The level of detail required for evaluation in each project will vary, dependent on its exact nature, however, the proposed assessment process should still be relevant, even if only high-level measures of success are adopted.
The proposed approach is intended to be flexible, whilst, at the same time, ensuring that all aspects of monitoring and evaluation are considered, even if only a few aspects can be wholly adopted within each project. Evaluation requirements should be considered on an individual project basis, dependent, for example, on whether the project is primarily aimed at informing future widespread deployment, or concerned with the deployment itself.
Key Stages
Seven Key Stages have been identified which underpin the overall monitoring and evaluation process. They are as follows:
The Process in Practice
It is recommended that the following tasks be undertaken for all projects, particularly those applying the decision making process set out in Planning Guide, with documentation produced which records the work undertaken and project outcomes and provides contextual information to enable transferability of results to be assessed by others.
Task 1 - Define all the different user groups within the project;
The identification of user objectives and benefit expectations is a pre-requisite to determining appropriate assessment objectives. The DfT / Ertico publication ITS City Pioneers Planning Handbook can assist with this process.
Task 2 - Describe key characteristics of the ITS application(s):
- Name;
- Major technologies;
- Service(s) offered; and
- Comparable sites for evaluation, if any.
Task 3 - Define assessment objectives for the project.
The following assessment categories should be considered:
- Technical parameters of system performance;
- Impact (estimated effects of an application e.g. impact on safety for drivers);
- User acceptance of application (based on surveys, interviews);
- Socio-economic evaluation (estimates gain/losses compared to “do nothing”);
- Market assessment; and
- Financial assessment.
Impacts recommended for assessment are as follows:
- Environment;
- Safety;
- Economy;
- Accessibility; and
- Integration.
It may be that a reduction in the number of elements to be evaluated is appropriate on certain projects. For example, if the system is technically proven then a further assessment of reliability may be unnecessary. Likewise, if the impacts are well known then this may not be relevant, for example, in the case of provision of additional traffic detection on the network.
Thereafter there should be a review to consider whether the assessment objectives are applicable to the specific evaluation task in hand. If not, then alternative assessment objectives should be proposed by those undertaking the evaluation.
Task 4 - The expected impact of each scheme for each target group of users then needs to be recorded. These expected impacts are then tested within the assessment process.
Indicators should be selected for each assessment objective (a common set indicators across similar projects is desirable). Reference cases should be reviewed and monitored (for example, to compare an existing situation with expected outcomes and to compare actual impacts with those envisaged)
Task 5 - The assessment methodology must be formulated specifically for each project.
Consideration should be given to the following, for each individual project:
- Review of available data and information, prior to the consideration of bespoke data collection and analysis processes for evaluation purposes;
- Development of data collection processes, recognising that the ITS tool under consideration can, itself provide useful data ; and
- Statistical considerations, timing and integrity of assessment.
Task 6 - After stage 5 has been completed, the project’s specific before and after data should be collected, analysed and assessed using appropriate measures.
Task 7 - Finally the results must be reported for each project.
It is important to remember that those likely to benefit most from the results of the evaluation are those who may be considering implementing a similar suite of ITS tools to address their own, specific local problems. They may well be unfamiliar with the ITS tool implemented and the area in which it has been deployed. Similarly, the reader may well be unfamiliar with the general principles of evaluation and the parameters which impact on system performance. It is imperative, therefore, that results are clearly presented with the necessary supporting information to enable the ‘transferability’ of results to be assessed.
A key outcome, which will be facilitated by adoption of the recommendation in part, if not in total, will be to get evaluation to be a routine part of the investment process and to get results reported in a consistent way. This will, in the longer term strengthen guidance available to those making investment decisions and help facilitate the deployment of ITS, in the most cost effective manner.
Reporting Format
The following aspect of the evaluation should be considered when reporting results:
Key Evaluation Results
- An Executive Summary of key achievements resulting from the project
Description of the Problem
- Site
- Issues addressed
Description of the ITS Project
- Objectives (considering Accessibility, Safety, Efficiency, Integration, Environment)
- Systems and Technologies applied
- Status of Project
Evaluation
- Timing and type of evaluation
- Objectives for the evaluation (considering technical, impact, financial and socio-economic objectives)
- Research questions (those asked by key stakeholders)
- Study area for the evaluation (Describe the site for both the application and also the associated extent of the monitoring and information about any control sites.)
- Impacts measured
- Methods employed
The Impact of the Project
- Technical Performance (present the technical performance results)
- Results (present the results of any impact, socio-economic and financial assessments)
- Research questions answered
- Statistical Analysis
- Overall Assessment.
Transferability of Results
- Local issues which have impacted on what has been achieved
