Project: Clarifying Qualitative Sub-Objectives in Appraisal C62

Reference: STP 14/5/28

Last update: 31/12/2009 15:11:27

Objectives

The objective is to identify and define qualitative sub-objectives, establish the general public's perception of these using psychology and sociological models and suggest methodologies for measuring the sub-objectives. The success of the models used and the selected methodologies will be measured and reported.

Description

The aim of the research is to raise the standards of appraisal for qualitative sub-objectives to the same high standard as that achieved for other aspects of GOMMMS (Guidance on the Methodology for Multi-modal Studies) . The work will seek to clarify the definition and appraisal of qualitative sub-objectives within GOMMMS. A number of psychology and sociological models will be used to study the perceptions of selected qualitative sub-objectives amongst transport users to ascertain a common understanding. This understanding will be used to work towards developing and piloting quantitative assessment criteria, which can be handled in the economic appraisal.

Contractor(s)

University of Leeds
Institute for Transport Studies, 38 University Road, Leeds, LS2 9JT
+44 (0)113 343 5325

Contract details

Cost to the Department: £47,053.00

Actual start date: 01 January 2003

Actual completion date: 13 September 2004

Publication(s)

New Horizons Research into Citizensí Understanding of Journey Quality: Implications for Appraisal
Author: John Nellthorp, Ann Jopson
Publication date: 01/08/2004
Source: ers@dft.gsi.gov.uk

Summary of results

  1. The introduction of the New Approach to Appraisal (NATA) in 1998 (DETR, 1998a)) represented a major improvement in transport project appraisal. The inclusion of qualitative objectives was a significant development in the appraisal process. Nevertheless, the definition and assessment of the qualitative objectives is not underpinned by decades of applied research in the appraisal context as the economic appraisal is. Thus, the qualitative sub-objectives are less well understood and defined, resulting in somewhat subjective assessment. As Guidance on the Methodology for Multi-modal Studies (GOMMMS) - the multi-modal application of NATA introduced in 2000 (DETR) - and now part of Transport Analysis Guidance (TAG) includes additional qualitative objectives and is used to appraise more multi-modal strategies and proposals, the objectivity and coverage of the appraisal process could be challenged over its comparatively subjective appraisal of qualitative sub-objectives. Coverage may be challenged if perceived aspects of a sub-objective are omitted from the appraisal. These issues are evident in the Evaluation of the Multi-Modal Study Process for DfT (Mackie et al, 2004). The research carried out in this study sought to clarify the definition of qualitative sub-objectives within NATA/TAG, as a preparation for future improvements to the appraisal process. A range of social science approaches were used to study the perceptions of selected qualitative sub-objectives amongst transport users to ascertain a common understanding.

    The ultimate aim of the research was to contribute to the process of raising the standards of appraisal for qualitative sub-objectives. To this end, the work sought to increase the evidence underpinning the definition and measurement of these sub-objectives. This should help lay the foundations for progress towards quantification through future research. The specific objectives of the work were:

    1. To identify qualitative sub-objectives for which the definition may need to be clarified to avoid inconsistent appraisal
    2. To establish the general public's perceptions of these sub-objectives using social-psychology approaches and thus suggest modifications to the definitions
    3. To suggest methodologies for measuring the sub-objectives based on the clarified definitions and suggest ways of integrating this information into NATA/TAG
    4. To review the success of the models used and methodologies selected in achieving the above objectives.

    When selecting qualitative sub-objectives to be studied under objective 1, the focus was on those related to citizens' perceptions of transport use, rather than those more concerned with externalities from transport.

    The research reported here is novel in that it uses social-psychology methods to analyse some of the qualitative sub-objectives in transport appraisal, with a view to gaining an improved understanding of their importance to citizens. The use of social-psychology to enhance appraisal development was an innovative development in both transport appraisal and transport psychology research.

    Social-psychology approaches were adopted to provide an understanding of concepts being considered that would not be provided by conventional choice experiment for example. In particular, perceived behavioural control (PBC) and subjective norm (peer pressure) influences on perceptions of journey quality were investigated. A key aim within the development of appraisal methodologies is the estimation of monetary values. In order to focus a WTP exercise or choice experiment in an appropriate way, however, it helps to have a clear idea of what attributes are important to the consumer, and how they interact. It is a part of the role of this study to provide this sort of clarification.

    Principles from the Theory of Planned Behaviour (TPB; Ajzen, 1988) - see Section 2.3.1 - were utilised as they can elicit an understanding that goes beyond willingness to pay or conventional market research, in that they can elicit perceptions grounded in the behavioural context. Attitudes, PBC and subjective norms are the three behavioural antecedents (mediated via intentions) defined by the TPB.

    For many of the qualitative sub-objectives, the issues in the appraisal method start right at the beginning - with the meaning and definition of the sub-objectives - do they relate to what matters to citizens, do they correspond to how citizens understand these aspects of transport? If not, we should not expect them to relate well to citizens' willingness-to-pay in economic appraisal. An important task in improving NATA/TAG in the short term is to use the evidence gained during this study - qualitative and quantitative - to help re-specify the sub-objectives in NATA/TAG, with an appropriate grounding in citizens' perception and understanding of what matters in transport provision.

    The work comprised of five key stages, an initial appraisal review, a transport user representatives workshop to select the sub-objectives to be studied, development of the methodology to be used in developing our understanding of the selected sub-objectives, focus groups and a large sample survey, followed by synthesis and report writing to draw our final conclusions. The method developed in this research draws heavily on a review of measuring quality in other areas, particularly satisfaction with wilderness experiences and quality of life. It became apparent through an initial review (Jopson, 2003) that these two areas account for much of the literature on measuring quality. The emphasis on satisfaction with wilderness experiences and quality of life has meant that the psychometric methods that dominate these fields had a significant influence on the proposed methodology. The psychometric methods used in both of these fields are grounded in social-psychology.

    As indicated, the work commenced with a review of the procedures and thinking that resulted in the definitions of qualitative sub-objectives in GOMMMS volume two. A wider literature review was also undertaken to establish how qualitative objectives are dealt with in other assessment frameworks in the transport field, and in related fields. Other sectors considered included forestry, flooding and coastal defences, renewable energy, health, and water. The review also included a historical aspect by examining past work to define and assess qualitative objectives. The key outcome of the review was a short list of qualitative sub-objectives to be discussed at the workshop that followed the review.

    Building on the review, the sub-objectives to be clarified were identified through discussions with transport user representatives, practitioners working in the field and DfT. Transport user representatives and others working in the field were consulted through the first of two workshops built into the project (see Appendix A for list of attendees). Key issues in selecting sub-objectives for analysis were; sub-objectives should be qualitative, be part of users' perceptions of the transport system, and be amenable to our research method. Transport user representatives were consulted to maintain the focus on users' perceptions. Practitioners involved with transport decision making in areas relating to the qualitative sub-objectives short listed for discussion at the first workshop were invited to shed light on which sub-objectives practitioners found it most difficult to assess objectively and impartially. Informal discussions were also held with researchers working in the areas of qualitative aspects of transport, appraisal, and transport decision making.

    The seminar was structured around two key elements: transport users' understanding of five sub-objectives concerned with overall journey quality, and discussion of the appraisal of those sub-objectives. The five sub-objectives discussed were:

    . Personal security
    . Journey ambience
    . Reliability
    . Access to the transport system, and
    . Interchange.

    Users (and potential users) appear not to perceive these five sub-criteria as separate independent variables. Instead, they perceive them as interacting, complementary influences on journey quality. For example, personal security becomes a much more serious issue when scheduled public transport fails to operate without warning, or is late, leaving citizens 'stranded' at - sometimes exposed - bus or tram stops, or rail stations. The combined effect of poor reliability and poor security can be more serious in users' perceptions than either of the effects in isolation. Consequently, it was decided that all five of the sub-objectives should be taken forward to the next stage of the research.

    The attributes of each sub-objective that makes up journey quality, which needed to be measured through a large sample household questionnaire survey, were identified through focus groups. These were based around describing and discussing journeys. Focus groups are widely used in social research and their use to discuss actual journeys at this stage of the research was particularly important since Hagman (2003) notes that individuals build a variety of meanings around car use [and presumably other modes as well], and that those grounded in experience tend to be more absolute, where as those that are mediated [e.g., via the media, or word of mouth] tend to be negotiable. The focus groups concentrated solely on a discussion of journey quality and its attributes to provide depth of analysis. Questionnaire wording was developed later through piloting.

    It was clear from analysis of the focus groups that information, service frequency, control and quality issues that currently fall under the ambience and transport economic efficiency sub-objectives are particularly important to transport users with regard to over all journey quality. The most important quality issues appear to be crowding, vehicle quality, cleanliness (of vehicles and waiting environments), lighting, and information. Reliability and personal security do not appear in this list as a clear cause and effect is identified between these issues and information and service frequency. If services are high frequency (every two to five minutes), and transport users have all the information that they need, then personal security is much less important, as door-to-door travel time and likelihood of 'looking lost' are minimised and thus, so is exposure to risk of personal security incidents. Similarly, risk of unreliable services is minimised when services are high frequency. However, in the current status quo, personal security and reliability are significant issues. In addition, excessive time spent waiting for buses is also perceived as a problem resulting from current low service frequency.

    With regard to access to the transport system, transport users felt that this was central to journey quality. However, there is a subtle difference between the perceived role of access to the transport system and the issues discussed in the previous paragraph in achieving high journey quality. The issues summarised above affect journey quality en route, whilst access to the system is a prerequisite. Accessibility and continuous design were discussed frequently with regard to access to the system, as well as in relation to other sub-objectives that affect journey quality en-route.

    Interchange is also interpreted in a similar way. Firstly, interchange was largely discussed in terms of integration, which was considered essential for full access to the transport system. Thus, interchange is essentially an element of access. In this way it is also a prerequisite of a high quality journey. A good example of this is a discussion at the London focus group, in which participants described the journey to Crouch End as poor quality because it required two interchanges, for which there is inadequate information to make the interchanges easily. Thus, Crouch End was described as a, "public transport dead zone" and potential journeys to Crouch End are suppressed due to the difficulty making them, that is to say, interchange is an inconvenience to be avoided.

    The analysis of the four focus groups thus makes it clear that in different ways the five sub-objectives studied here - journey ambience, reliability, interchange, access to the transport system and personal security - are all essential to journey quality. This qualitative analysis put us in a position to make interim suggestions regarding modifications to the appraisal process. It also made it clear to us that data on the perceptions of transport users affected by a real-world transport scheme would be essential to progressing the case study planned in the next stage of the work. This would allow us to compare journey quality attributes, that we had been told were important, with the more conventional measure of journey time savings. Originally this was to be a desk study based on a previously appraised scheme, however, the need for real data on transport users' perceptions meant a real world case study should be developed instead.

    The large sample household questionnaire survey was undertaken to corroborate the importance of the issues raised by the focus groups in relation to journey quality. In this way we were able to collect data to confirm or refute the full range of attributes of journey quality raised by the focus group participants. Further to this, the questionnaire was able to collect data grounded in the TPB as outlined previously. Given the importance focus group participants placed on control, this aspect of the questionnaire had become particularly important. The questionnaire also collected data pertaining to the A65 Quality Bus Initiative case study.

    This analysis of the large sample household survey has confirmed that the five sub-objectives considered in this research - personal security, reliability, journey ambience, interchange and access to the transport system - are all important to overall journey quality. The existence of interactions between these attributes of journey quality is also confirmed. This suggests that journey quality could be considered as an appraisal objective in its own right, although further investigation of packaging effects could be needed first. It is notable that reliability and personal security are frequently of most concern to respondents, whilst interchange particularly, and also access to the transport system, are lesser issues. This is likely to be because these issues are to an extent perceived as prerequisites of high journey quality. It also appears that journey quality becomes more important as people become older, and that easy-to-use public transport, that approximates the on-demand nature of private transport is important to people.

    Information, high frequency public transport services, and predictable public transport routes are also important, as is feeling in control, which is influenced by these three factors. This confirms the focus group findings regarding control. In terms of the third TPB concept - subjective norms - influence of the views of others on mode choice was most apparent in relation to personal security and reliability, especially for public transport users.

    Use of the TPB has provided a deeper understanding of the journey quality attributes and the interactions between different aspects of transport users' views about the attributes. This understanding has helped corroborate the key interactions identified during the focus groups. It has become apparent that perceptions of importance of overall journey quality are related to perceptions regarding PBC and subjective norms (peer pressure), as well as attitudes relating to the five attributes of journey quality. In terms of the conclusions regarding implications for appraisal, this means that the suggestions made are grounded in a deeper understanding of the general publics' views regarding journey quality, not just a basic assessment of whether x is good or bad. Particularly, we have been able to gauge the importance of control with more confidence, and build this into our suggestions.

    Overall the grounded approach that we have taken, has significantly improved our understanding of journey quality and its five key attributes. The use of a transport user representatives' workshop allowed us to identify the important issues from the perspective of those with an overview of the current status quo regarding journey quality and a number of experts. The focus groups then allowed us to anchor this understanding with perceptions held by the general public, which were corroborated by the large sample household questionnaire survey. The survey also allowed us to develop our understanding from a behavioural perspective and collect data on changes to be brought about by a real-world transport project to facilitate assessment of the relative importance of journey time savings versus journey quality attributes.

    Having developed an understanding of journey quality, we have been able to draw a number of conclusions regarding the implications for appraisal. These are reported in an accompanying report, "New Horizons Research into Citizens Understanding of Journey Quality: Implications for Appraisal." Further to this, it would now be desirable to develop econometric work to generate monetary values associated with journey quality. A priority evaluator approach could be appropriate as this can cope with a large range of attributes of journey quality relatively easily. Alternatively, a series of stated preference experiments could be developed to test willingness-to-pay in specific choice contexts. The choice of econometric approach would need to be considered carefully, but we now have a firm basis for considering which of the attributes of journey quality should be included in future econometric work.