Monitoring and evaluation

Evaluation - of success and failure - is vital to ensure that scarce resources are directed towards measures that are effective in reducing crime and improving passenger perceptions of personal security Most would acknowledge the truth of this, but few organisations do in practice evaluate what they do. They are therefore unable to demonstrate that an initiative or change in working practice is effective and that the money is well-spent.

This paper explains what is involved in monitoring and evaluation. It also explores some of the difficulties that may be encountered, giving some suggestions as to how these may be overcome.

What is monitoring and evaluation?

Monitoring what happens when you have introduced an initiative or change in working practice is essential if you are subsequently to evaluate its effect on personal security. Monitoring simply means regularly collecting information on how the initiative has been carried out. For example, you may monitor the number of safety leaflets distributed, the number of drivers trained in dealing with conflict, or the number and location of CCTV cameras installed at a station.

Monitoring can also mean, for example, regularly collecting and recording information on incidents both before and after a change in working practice or the introduction of an initiative. It may also mean recording information on changing patronage or passenger perceptions on a particular route or service. Monitoring should also include maintaining records on the costs of the initiative - both set-up and running costs. Collecting such information will enable you to measure and evaluate the success of an initiative.

Evaluation builds on the monitoring and provides an assessment of the effectiveness and impact of the initiative or change. It can tell you, for example, whether the leaflets you distributed have helped passengers to feel more secure, and whether fewer have now become the victims of crime. As a result of the training, do staff feel safer, have fewer been assaulted, and are they as a result more proactive in dealing with the public? Do the CCTV cameras make passengers feel safer, has there been a reduction in crime, and has their presence led to more detections?

Why is evaluation important?

Evaluation of an initiative or change in working practice will tell you:

whether you have achieved what you set out to do
eg. has vandalism of the stop or station decreased?
do more people use the service after dark than did before?
do they feel safer?
have relations between drivers and school children using the service improved?
whether resources (money and staff time) have been well spent
eg. what did the initiative cost?
what has been saved in terms of staff time, cleaning costs, or repair of vandalism?
what has been gained in terms of improved perceptions, increased use, positive publicity?
whether an initiative is capable and worthy of replication elsewhere
eg. what were the factors that contributed to its success?
are these peculiar to this context, or might they be found elsewhere?
what were the factors that stopped it being as successful as it might have been?
could these be avoided next time?

How to evaluate?

Be clear about what you want to achieve:	How will you know whether you have been successful if you are not clear about what you want to achieve? Be as precise as possible about your objectives. What is the crime you want to reduce and where is the problem greatest? Over what period will the initiative last?
Consider all the possible measures of success:	Do not restrict yourself to the crime figures, but consider other indicators of success. These might include passenger and staff perceptions, letters of complaint or appreciation, cleaning or repair costs, increase in revenue, less staff turnover, and newspaper coverage.
Build evaluation in at the beginning:	It is always harder to go back and measure once you have begun. Start by thinking about what you want to measure and build a baseline of data against which you will measure the success of the initiative.
Calculate the costs of crime and the costs of the initiative:	An initiative can only give good value for money if the costs of doing nothing are greater than the costs of intervening. Although often difficult to calculate, costs such as loss of patronage, loss of staff time through sickness, and administration of cleaning and repair should be included in your calculation.
Consider both experiences and perceptions:	Your initiative or change in working practice may have little impact on crime figures but may do much to improve people's perceptions of their personal security. For example, better lighting may not lead to a reduction in assaults (which may in any case be low), but may make a significant difference to how passengers feel. So, your evaluation should include not only statistical measures, for example of the number of incidents or changes in patronage, but also information on changes in people's views and perceptions. Often the latter is obtained through short surveys asking people - passengers and staff - what they think the effect of the initiative has had on their feelings.
Be aware of other changes that are taking place at the same time:	Other changes may be taking place which affect the outcome of your initiative. Crime figures generally may be falling, there may be a police initiative in the area which is deterring offenders, or this may be a time of year when there would in any case be a seasonal fluctuation (for example, during the school term compared with the summer holidays). Using a control group or area can be a useful way of providing a comparison. In evaluating the success of a specific initiative or change in working practice, it is important to assess whether, for example, the reduction of incidents over a period of time is because of these other factors or because of the action you are taking.

Monitoring and evaluation