Second Review of ANASE Stated Preference Results
Background
I was originally asked in an email from Eric Weinstein of the Department for Transport on 4th May 2007 to provide a review of the MVA Consultancy’s research conducted as part of the ANASE project.
The review was to address the Stated Preference (SP) modelling aimed at deriving monetary valuations of changes in noise as proxied by changes in aircraft movements. The documents I reviewed were:
- Willingness to Pay, Time of Day Relativities and Implications for Leq
- Appendix X – SP Model Development
- Appendix XX – SP Valuations and SEL
This review was supplied to MVA, along with reviews by Ian Bateman and Brett Day.
A subsequent meeting was held on September 11th 2007 with David Thompson, Chief Economist at the Department for Transport, to discuss our comments and various ways forward. As part of this, we requested that MVA indicate how they had addressed the points raised, including any disagreement with or correction of the points raised in the three reviews (analogous to the responses an author would be expected to provide to the comments obtained in peer review of a research paper submitted for publication).
One suggestion was that we would review the amendments that had been made by MVA in the intervening period and provide a further report. This review would bear in mind what further work might feasibly be conducted in the immediate future, largely based around the existing data, and also identify longer term research that is deemed worthwhile.
An email from Roy Strapp on October 4th 2007 supplied a revised report and a document from MVA setting out the actions in response to comments made. We were asked to:
…… undertake a ‘second review’ looking at the changes that have been made in order to report where you believe that these have been beneficial etc. Similarly if there are still areas where you feel uncertainties remain these should be highlighted including, where appropriate, any specific areas where you may feel that further work would be desirable.
Comments
On this occasion I have been provided with an executive summary. This really is a rather excellent account of the research conducted, accessible to the informed lay-person and conveying the considerable attention to detail and thoroughness that has been a feature of this study.
Nonetheless I am a little annoyed that our request for responses to our comments has only been treated with what can best be described as ‘lip-service’. I am quite prepared to accept that I could have been wrong with some points, or misunderstood some aspects of the research, whilst it could be that some issues could not be addressed without fresh research or further funding. Such reviews as these are now not uncommon, but I sometimes wonder what purpose they serve! It would benefit DfT to know if its reviewers had got some things wrong, but this situation leaves us in parts with different sets of views.
The consultants have done work on the impact of exposure to noise at home given the points raised in the reviews. Those who were apparently not at home are found to have significant and non-trivial valuations, sometimes in excess of those who expressed they were at home. There are various possible interpretations of this, as the consultants recognise, such as being in the vicinity of aircraft noise in the time period in question, being at home for some of the time period or on some days, or others in the household being at home. Equally there might be response bias at work, in the form of halo effects, whereby changes in movements at other time periods are implied, protest responses or misunderstanding the time period to which the SP relates.
It can be a contentious issue removing observations from the sample, but it might be illuminating to observe the absolute values and particularly the relative values by time period on the assumption that these people should have zero values. It might make little difference to the latter, or else it might make for a more reasonable distribution of values across the day since I still have concerns about the relative time of day values which might be used to distribute, say, an hedonic pricing value across the day, just as I had with the jumbo jet values stated in the previous review. I still find the premium for 1900-2300 to be low (and I am certainly not promoting any evidence of my own here) and wonder why 0300-0700 is so high other than for a bunch of insomniacs.
Other than this, there is as far as I can see little further to comment upon and the points raised in my previous review still apply.
Further Research
I was not particularly impressed with the consultant’s suggestions for further research. At the least, the Department has gone to the effort of having this work reviewed and these reviews imply further work. The consultants should have either accepted these ideas or in their responses stated that our suggestions for further work were flawed. As an example, I suggested that there would be interesting further analysis, in the short term, in the functional relationship between utility and the combination of sound level and number of events. This was acknowledged in one of the few MVA responses to my comments, yet it still does not enter the further research section.
In the meeting with DfT, we were asked to identify areas of further research and to indicate whether this might be regarded as follow-on work on the existing data or more substantive research over a longer time period. The analysis suggested in the previous paragraph would certainly enter the former category.
There were concerns about the CVM analysis raised by all reviewers. Some immediate follow-up work could improve upon the analysis, although it should be remembered that, in my understanding, the CVM was not offered as part of the research proposed by MVA.
More work is needed on the time of day effect. Suggestions for follow-up work have been suggested above. However, it is clear more work is needed to explore more fully the relationship between exposure and valuation, including probing why respondents who are apparently not exposed to noise apparently have a value for reducing it. Ratings of annoyance by time period could provide some indication of relative valuations to confirm results from SP work. Such longer term work should examine the issue of package effects and the effects of using such small time periods. These could be behind the large values obtained.
Analysis of package effects could most likely also allow respondents to trade-off aircraft movements at different times of day, thereby giving a different and possibly more accurate account of time of day relativities rather than relying on variations across individuals in valuations for a specific time period. It might also provide an explanation for the high valuations here obtained.
At the moment, different aircraft movements are offered and the noise levels are estimated. However, respondents do not actually have experience of the different noise levels and therefore it seems to me that the linking of valuations to physical noise levels is entirely artificial. Fundamental research is needed that explores how respondents react to different experienced (and subsequently measured) levels of noise. This seems to me a major but highly significant challenge, and I would regard this as a priority for longer term research.
Evidence indicates that gains and losses in money are regarded differently. But quite apart from this, some might not believe that a grant would be provided. This might exhibit itself in non-trading behaviour, but not necessarily, and it would be interesting to establish the extent to which ignoring the monetary gains was reducing the cost coefficient and hence inflating the valuations and thereby contributing to the implausibly large values obtained.
Such simple SP exercises, where the purpose of the study is clear, do offer an incentive to strategic bias, particularly in such a contentious area. In the longer term, means of reducing, overcoming or identifying this in this context are needed.
More research is needed into whether there are thresholds in valuations and the extent to which non-linearities are present, such as gains and losses in movements having different valuations and unit values depending upon the level of movements. This is linked to the suggestion of further important analysis of the relative impact of noise levels per event and number of noisy events. This has important implications for the design of SP experiments to address this issue.
