DfT home   WebTAG home  Documents - Guidance documents 3. Expert - Unit 3.15.5 The Treatment of Uncertainty in Model Forecasting

Documents

Guidance documents - Expert

TAG unit 3.15: Forecasting Using Transport Models

There are five modules within this section:

3.15.1: Forecasting Using Transport Models

3.15.2: Use of TEMPRO Data

3.15.3: Forecasting and Sensitivity Tests for Public Transport Schemes

3.15.4: Rail Passenger Demand Forecasting Methodology

3.15.4d: Rail Passenger Demand Forecasting Methodology - Draft

3.15.5: The Treatment of Uncertainty in Model Forecasting

Unit 3.15.5: The Treatment of Uncertainty in Model Forecasting

April 2011

pdf icon Unit 3.15.5 (Adobe Acrobat - 79KB)

1. The Treatment of Uncertainty in Model Forecasting
   1.1 Introduction
   1.2 Background
   1.3 Data Assembly
   1.4 Establishing Scenarios
   1.5 Development of the Core Scenario
   1.6 Analysing Uncertainty
   1.7 Reporting
   1.8 Summary
2. Further Information
3. References
4. Document Provenance

1. The Treatment of Uncertainty in Model Forecasting

1.1 Introduction

1.1.1 This TAG Unit is focussed on the treatment of uncertainty in forecasting the impacts of a transport project, with particular reference to the use of transport models for that purpose. While the material in this Unit is primarily aimed at modellers, it will also be of relevance to those considering non-modelled (mainly environmental) impacts. Uncertainty also affects the estimation of monetised benefits based on forecast impacts and the estimation of project costs, but these issues are not covered here. The impact of uncertainty on the estimation of project costs is discussed in The Estimation and Treatment of Scheme Costs (TAG Unit 3.5.9).

1.1.2 Uncertainty in model forecasts can be broken down into that associated with model specification and measurement errors, and that which stems from an inaccurate view of the future, sometimes termed forecasting errors. The guidance in this TAG Unit is primarily aimed at dealing with forecasting errors, further discussion on dealing with model specification and measurement errors can be found in DMRB Volume 12 and Variable Demand Modelling - Detailed Stages (TAG Unit 3.10).

1.1.3 Uncertainty in forecasting derives from the possibility of more than one outcome occurring during the period being forecasted and the forecast materially differing under these different outcomes. This would be represented by an input, or several inputs, to the forecast differing in the different views of the future. These may be described in a scenario where the effect on inputs is quantified. Analysis can then look at the sensitivity of forecasts to the inputs.

1.1.4 In identifying possible outcomes, the analysis ought to be proportionate. The analysis suggested in this guidance expects some assessment of the likelihood of different situations. However, this assessment is qualitative, helping to gauge the necessity of scenario modelling and where in the analysis it should be undertaken. Primarily, this will provide the Department with information about the robustness of the business case for a scheme. It ought also to assist scheme design to consider the aspects of a scheme which improve its robustness to uncertainty.

1.1.5 The guidance in this TAG Unit provides a systematic analysis method for dealing with uncertainty in model forecasting and is equally applicable across all forms of assessment (whether road based, public transport or multi-modal). Although the guidance here generally discusses scheme appraisal, it can equally well be applied to strategy level analysis.

1.1.6 It is anticipated that the guidance provided in this Unit will lead to:

  • An improved treatment of local sources of uncertainty in project analysis;
  • Better forecasts of the most likely outcome with and without the project; and
  • Closer integration of the project within the wider transport and land-use planning context.

1.2 Background

1.2.1 In the past, uncertainty in road scheme appraisal has been dealt with by using a range that was defined by use of high and low growth forecasts of national road traffic. These national forecasts were used as a framework within which to develop local forecasts for use in appraisal. Where particularly significant effects were expected from different inputs, sensitivity tests were carried out to support the main analysis. This procedure ensured that national sources of uncertainty, due to factors such as national economic growth and fuel cost changes, were always taken into account in project appraisal.

1.2.2 However, evaluation has shown that there is considerable uncertainty in scheme appraisal forecasts stemming from planning and land-use issues as well as the timing and delivery of other transport schemes. These sources of uncertainty have not always been fully taken into account. Similarly, there are occasions where policy intentions at a strategic level have been signalled, which, over the appraisal period, could materially affect the trends assumed when a scheme is analysed. In this category would be schemes to manage transport demand. The analyst will need to collect information about the extent to which such uncertainties have been translated into actual measures which could affect forecasts for the particular analysis being undertaken. This guidance offers tools to analyse any forecasting uncertainty resulting, advising promoters how to assess and then use any relevant information to ensure a complete picture of forecast uncertainty is provided.

1.2.3 These tools then improve on past practice, which has focussed on the forecast 'do minimum' (that is, a future including only committed transport and land-use changes) as the basis for analysis. This fails to take account of planned (and often well defined) future transport and land-use changes that are not committed. The do minimum is therefore a view of the future that is deliberately conservative in terms of interventions - a valuable perspective but not one that the Department is prepared to put forward as its view of what is most likely to happen.

1.2.4 For these reasons the analyst carrying out scheme appraisal is now expected to determine, and justify, local traffic and patronage forecasts that reflect both national and local uncertainty to provide a range within which it is reasonable to plan.

1.2.5 It is anticipated that a 'core' scenario will be developed and that a range of sensitivity tests and/or alternative scenarios will also be developed to account for future uncertainty. Development of these scenarios and sensitivity tests should be based on a systematic analysis of the uncertainty associated with all the forecasting inputs that could affect the impact of the project being considered. In some cases, this will require the collection of additional information and data, consulting local and regional land-use, transport and other plans with the assistance of local planners. Sometimes, further analysis using existing modelling capabilities may be more appropriate. Account will need to be taken of any dependencies between factors as appropriate. Local planners can provide information to help identify these.

1.2.6 This form of scenario and sensitivity test development will need closer integration between transport planning and the policy context of a scheme. It will ensure that scheme development takes appropriate account of all policy objectives, helping projects work better with aspects of planning policy.

1.2.7 To define an appropriate core scenario and alternative scenarios and/or sensitivity tests, the following steps should be followed and are detailed in the following sections.

  • Assemble demographic, economic and other data for the study, drawing on the TEMPRO database and other material provided by the Department for Transport, together with material provided in regional and local planning documents (section 1.3).
  • Consider the sources of uncertainty and qualitatively assess their likelihood (section 1.4).
  • Establish a core scenario, based on national and local data and taking account of the uncertainty associated with various elements of that data (section 1.5).
  • Establish a range of alternative scenarios or sensitivity tests, where forecasts inputs are likely to differ from the core and materially impact on analysis results (section 1.6).
  • In the analysis of scheme options, the core scenario and alternative scenarios or sensitivity tests should form the basis for a full appraisal. The core scenario would be reported in the AST, with alternative scenarios being exceptions (section 1.7).

1.3 Data Assembly

1.3.1 Forecasts of demographic (population, employment, households) and car ownership factors have been prepared by the Department for input to the National Trip End Model (NTEM) and published in TEMPRO (www.tempro.org.uk). Details of the development of the TEMPRO forecasts and guidance and their use in project appraisal are given in Use of TEMPRO data (TAG Unit 3.15.2). The core scenario must be based on, and constrained to, these projections. Alternative scenarios or sensitivity tests should also be constrained to these projections in most cases, although it may be appropriate to relax this constraint in some circumstances.

1.3.2 The Department recognises that TEMPRO is unlikely to be completely up-to-date with respect to new developments. Nowhere in the derivation of the TEMPRO projections is there an explicit assumption that particular developments will or will not go ahead. Where a particular development proposal is likely to have a significant impact on the scheme being appraised, the trips from the zones affected should be modelled explicitly rather than taken from the TEMPRO data. Where this is done, the planning data should be controlled to the published TEMPRO projections, usually at district level.

1.3.3 A review of land-use, transport and other plans as appropriate (for example Regional Spatial Strategy, Regional Transport Strategy, Regional Development Strategy, Local Development Plan, Local Transport Plan, and so on), including discussion with key stakeholders (such as local planners, Highways Agency, and so on), will identify local inputs to the forecasting process. This will provide information about planned developments - their location, scale and so on. It will also provide information about local transport measures, ranging from 'soft factors' and demand management measures to major road and public transport investment proposals.

1.4 Establishing Scenarios

1.4.1 Decision making processes should be underpinned with the most robust available information. Where there is uncertainty about the robustness of the information, the analysis might seek to quantify this uncertainty to enable reasoned decisions to be made. The key issues in assessing uncertainty are:

  • The range of possible inputs;
  • The likelihood of each input; and
  • The interaction between different elements which affect inputs.

1.4.2 Although a range of outcomes for any one input to the appraisal may be relatively easy to determine, through modelled sensitivity, attaching detailed probabilities to these outcomes is much more problematic. Providing detailed correlation between different inputs to the appraisal is also very difficult since there is a wide range of causal and dependent relationships affecting costs, traffic levels / patronage, revenues and benefits of the scheme. For example, there may be linkages between low levels of traffic / patronage and the possibilities of reduced operating or maintenance costs.

1.4.3 The recommended process for assessment of uncertainty in inputs is detailed in the following sections.

Identifying Uncertainty and Assessing Likelihood

1.4.4 To analyse uncertainty the first step in the process is to create an uncertainty log. The process for creating an uncertainty log is not unlike that usually adopted in a quantified risk analysis for cost estimation - see The Estimation and Treatment of Scheme Costs (TAG Unit 3.5.9). The uncertainty log should highlight all the local and external uncertainties and factors likely to affect the traffic / patronage, revenues and delivery of scheme benefits. The uncertainty log will need to include those factors that have an individually minor effect, as the cumulative effect of minor factors may be a material consideration in the appraisal.

1.4.5 The uncertainty log should include an assessment of the uncertainty of each individual input by placing it into one of four categories. In deriving this uncertainty measure the analyst should use the table below as a starting point. Many of the uncertainties will be local in nature, so drawing on local knowledge and experience to reach a final categorisation will be important. Where analysis covers a wide geographical area, it is likely to be sufficient to focus on the area in the vicinity of the scheme being considered. The values derived should be justified with a short piece of explanatory text.

Classification of Future Inputs:

Probability of the Input Status
Near certain: The outcome will happen or there is a high probability that it will happen. - Intent announced by proponent to regulatory agencies.
- Approved development proposals.
- Projects under construction.
More than likely: The outcome is likely to happen but there is some uncertainty. - Submission of planning or consent application imminent.
- Development application within the consent process.
Reasonably foreseeable: The outcome may happen, but there is significant uncertainty. - Identified within a development plan.
- Not directly associated with the transport strategy/ scheme, but may occur if the strategy/scheme is implemented.
- Development conditional upon the transport strategy/scheme proceeding.
- Or, a committed policy goal, subject to tests (e.g. of deliverability) whose outcomes are subject to significant uncertainty.
Hypothetical: There is considerable uncertainty whether the outcome will ever happen. - Conjecture based upon currently available information.
- Discussed on a conceptual basis.
- One of a number of possible inputs in an initial consultation process.
- Or, a policy aspiration.

An example of part of an uncertainty log is given below:

Input Uncertainty Comments
Factors affecting underlying demand:
400 house development @ W location More than likely Land identified in local plan for housing provision. Application submitted to local planning authority.
Large Housing Development (10000+ houses) @ X location Hypothetical Identified as one of 5 locations by local authority for new town development. Part of initial consultation process prior to inclusion in structure plan.
Superstore @ Y location Reasonably foreseeable Currently speculative project - land-use identified in structure plan (fairly high uncertainty about timing and exact location).
Factors affecting supply for transport:
Increase in Rail capacity @ Z location Near certain Under construction.
Factors affecting cost of transport:
A local road pricing scheme Reasonably foreseeable Business case under preparation for funding of a scheme.

1.4.6 The development of an uncertainty log will normally start with a review of land-use, transport and other plans as appropriate (for example Regional Spatial Strategy, Regional Transport Strategy, Regional Development Strategy, Local Development Plan, Local Transport Plan, and so on). These plans will help identify the main inputs to the forecasting process. However, discussion with key stakeholders (such as local planners, Highways Agency, and so on) should be carried out to reach an agreed position on the likelihood of any given input.

1.4.7 The uncertainty log should also be the subject of wider consultation (to gain the views, for example, of the public, statutory bodies, non-government organisations and so on) when appropriate so that as broad a consensus as possible is reached on the uncertainty log assessments.

1.4.8 Any evidence used to arrive at the register assessment should be carefully recorded, and conclusions should be kept under review and revised as necessary.

1.4.9 The development of an uncertainty log will assist in defining the scenarios that will need to be tested and allow an understanding of the level of certainty and possible extraneous factors that may affect the core appraisal and hence the number of sensitivity tests that will be required.

Sources of Uncertainty

1.4.10 The source of uncertainty in inputs is most likely to be associated with a number of issues that could include:

  • Uncertainty about background trends used in the modelling such as future levels of GDP growth and fuel cost, both of which have a substantial impact on the rate of traffic growth (at least in the short term);
  • Political or commercial uncertainty as to whether individual large developments, or transport projects other than those being appraised, will go ahead;
  • Local economic or planning uncertainty, e.g. as to the success of local regeneration initiatives;
  • Other factors, such as the effect of local policy initiatives to influence travellers' mode choice.

1.4.11 These issues will affect the modelling in different and quite complex ways. However, analysts may find the following categories useful in exploring the possible categories.

Uncertainties over Demographic, Economic and Behavioural Trends

1.4.12 There is a range of inputs into forecasting where the local analyst is unlikely to be able to gauge either the likelihood or the impact to a sufficient degree. GDP growth, fuel price trends, vehicle efficiency changes and other national trends are generally assessed and reported at a national level, with scheme analysts using these results. In fact, most models will not be able to reflect these effects explicitly, relying on the results from national models underlying TEMPRO to do so. Their impact will usually best be reflected by using high and low reference traffic growth instead.

1.4.13 To deal with such uncertainty in highway models, it is expected that the analyst will explore scenarios using an appropriate range about the core scenario growth forecast of ±2.5% for traffic forecasts one year ahead of the model base year, rising with the square root of the number of years to ±15% for forecasts 36 years ahead. For example, if central growth over 16 years from the model base year is 20%, the range to examine should be ±2.5% x 4 = ±10% or 10% to 30%. The box below outlines how these ranges should be used when variable demand modelling is being used.

Implementing National Traffic Forecast Uncertainty in Variable Demand Models

National traffic forecast uncertainty ranges quoted are for traffic (vehicle-kilometre) growth. Therefore, when variable demand modelling is being used, the most appropriate approach to carrying out the necessary sensitivity tests is as follows:

  • Using the core scenario future year reference case matrix, iterate the demand and supply models to convergence in the usual way and extract the converged future year trip matrix;
  • Adjust this matrix, on a cell by cell basis, to reflect the range of uncertainty by taking the appropriate proportion of the model base year matrix and adding it to or subtracting it from the converged future year core scenario matrix*. For example, for a forecast year nine years from the base, add or subtract 7.5% of the base year matrix.
  • Using these adjusted matrices, iterate the demand and supply models to convergence in the usual way to provide the required future year sensitivity tests;
  • Compare the outturn estimates of vehicle-kilometre growth for the sensitivity tests with that for the core scenario to confirm that the sensitivity tests do provide the appropriate range about the core scenario. Note, however, that the outturn range may be significantly narrower than that input when considering a heavily congested network. This is acceptable, since the impact of uncertainty in national trends is likely to be muted in such conditions.

* To understand why this is correct, consider a matrix cell with value A in the base year matrix and B in the (fully converged) future year matrix. Central growth is, therefore, G=B/A. We wish to test variants based on growth Ghigh=G+U and Glow=G-U, where U is the range appropriate for the given future year. Thus, for the 'high' variant, we need to calculate the value Bhigh= Ghigh*A=(G+U)*A=B+U*A. Similarly, Blow=B-U*A.

1.4.14 In public transport models, where modelling more explicitly used variables such as GDP growth, the potential for more sophisticated scenario modelling might be explored.

Factors affecting supply for transport

1.4.15 There are a range of uncertainties in the forecast assumptions entering a model relating to the transport supply:

  • New road schemes/road improvements;
  • New passenger transport schemes/passenger transport improvements;
  • Road space reallocation (e.g. introduction of bus lanes);
  • New/improved cycle facilities;
  • New/improved pedestrian facilities;
  • Parking supply;
  • Park and ride schemes; and
  • Traffic management schemes.

1.4.16 Major infrastructure works and developments currently under construction are likely to have the highest degree of certainty attached. Proposals that are fully incorporated into the local planning process and are a high priority for the relevant local authority will have a higher likelihood than more speculative proposals. Longer term proposals or proposals identified for future consideration will have a lower likelihood. It is essential that the allocation of likelihoods to proposals be carried out in a way that is realistic and based on local knowledge, avoiding "optimism bias" as far as possible.

Factors affecting underlying demand

1.4.17 Evaluation of appraisal studies after the opening of schemes has identified how important correctly determining the factors affecting underlying demand is. Many of the drivers of transport demand would be estimated using trends from past behaviour and the nationally consistent local estimates provided by the Department in TEMPRO quantify these. However, local knowledge and published plans may highlight some further determinants and the uncertainties attached:

  • Location of developments;
  • Timing of developments;
  • Size of developments;
  • Nature of developments (office, retail, leisure, residential, etc); and
  • Soft factors (such as videoconferencing, travel plans, individualised marketing campaigns, Bus Quality Partnerships).

1.4.18 While it is important to take account of developments in the preparation of scenarios and sensitivity tests, it is important to ensure that, for the core scenario and for most alternative scenarios and/or sensitivity tests, the planning data at the modelled area level is controlled to the published TEMPRO projections.

Factors affecting the cost of transport

1.4.19 Factors affecting the cost of transport include: parking charges; public transport fares, and other costs that would be under the control of local policy. It is recognised that forecasts of the costs associated with transport are open to a number of uncertainties which, to some extent, can be modelled in a transport model.

1.5 Development of the Core Scenario

1.5.1 As discussed earlier, it is important to ensure that a project is fully integrated with other transport projects and policies and with land-use plans and policies.

1.5.2 Indeed, in many cases, a project will have emerged from an integrated planning process and thus be part of a wider package of land-use developments, transport projects and land-use and transport policies and other measures, including 'soft factors'. Under these circumstances, it is clear that appraisal of a project must take account of the effect of the other elements of the planned package.

1.5.3 However, it must be recognised that delivery of the planned package itself will be subject to uncertainty. To assume that the entire package will be delivered as planned would be very optimistic - it is very likely that some elements will be delivered late or not at all. On the other hand, to ignore the planned package (as in a 'do nothing' scenario), or to take account of only those which are clearly committed (as in a 'do minimum' scenario), would probably be unreasonably pessimistic.

1.5.4 This leads to the requirement to appraise the scheme against a 'core' representing those elements of the planned package that can reasonably expect to be delivered by the specified forecast year. It is important to note that this scenario must be based on planned proposals - analysts should not attempt to speculate on the introduction of proposals that have not been included in published plans. Published plans will be periodically updated, introducing new proposals (and, perhaps, dropping some), especially for more remote years. It is clear, therefore, that the core scenario cannot be regarded as the 'most likely' outcome - it can only represent the outcome that appears most likely given published plans.

1.5.5 Subject to these limitations, the core scenario should be:

  • unbiased (that is, as likely to be exceeded as undershot, on any relevant measure);
  • coherent and self-consistent (if X is in practice "highly likely" to be accompanied by Y, then X and Y should both be included);
  • free-standing (not dependent on other scenarios for its definition); and
  • realistic and plausible.

1.5.6 Construction of this scenario will inevitably involve some element of judgement. This judgement should be informed by the development of the uncertainty log outlined above. It is presumed that all the inputs categorised as 'near certain' will be included in the core scenario, it is also expected that those inputs categorised as 'more than likely' will be included. It is expected that inputs in the 'reasonably foreseeable' category will be excluded from the core scenario and it is presumed that no inputs categorised as 'hypothetical' will be included in the core scenario.

1.5.7 The core scenario should include assumptions on economic growth and other trends, see The Economy Objective (TAG Unit 3.5), that may influence transport demand and costs. These national factors should not normally be varied without very strong evidence.

Without-scheme Case

1.5.8 The without-scheme case in the core scenario should represent a realistic view of what is likely to happen in the absence of the scheme proposals. It will usually correspond to maintaining present transport facilities and implementing the more certain aspects of regional and local transport strategies. It will take into account forecast changes in population and land-use.

1.5.9 Forecasts of population, households and employment published by the Department in the TEMPRO database should be used as an overall control for demand forecasts usually at district level. The additional population, households or employment implied by explicit representation of land use proposals must be compensated for by reducing background growth at the appropriate geographical level - usually district level - see Use of TEMPRO data (TAG Unit 3.15.2) for further guidance. Land-use aspects should be based on the uncertainty log and should therefore fit within the framework set by regional planning guidance and local development plans. The planning data forecasts should be broadly consistent between neighbouring authorities and between different areas of the country. Where regional forecasts differ significantly from the forecasts in TEMPRO, sensitivity tests may be required - see below.

1.5.10 It is anticipated that local forecasts of demand will be developed from the inputs described above. It may however not always be practical to develop local demand forecasts for certain elements of demand, and where these are non-critical it may be appropriate to use national forecasts. An example of this might be use of the National Transport Model to provide growth factors for heavy goods vehicle forecasts.

1.5.11 The transport supply aspects of the without-scheme case should be based on the uncertainty log. However, there may be circumstances where it is clear that transport conditions without the project are such that further improvements to the transport system are likely. Where that is the case, these improvements should be included even if they weren't identified in the list of transport changes. However, this kind of without-scheme improvement should not involve large expenditures (up to say 20% of the proposed scheme cost). This would run the danger of severely distorting the appraisal. Where this is an issue, the improvements should be redefined as an alternative with-scheme case.

1.5.12 The without-scheme case will need to reflect trends in transport provision. For example, if public transport service improvements or changes in the real cost of fares can be identified, there may be a case for extending these trends into the future. Further discussion on the issue of forecasting public transport provision can be found in Forecasting and sensitivity tests for public transport schemes (TAG Unit 3.15.3).

With-scheme Case

1.5.13 The with-scheme case will need to build on the same base forecasts and conditions as those outlined in the without-scheme case.

1.5.14 In some circumstances, there may be uncertainty about the precise definition of a scheme option. Where this is the case, sensitivity analysis should be used to assess the impacts on the appraisal of variations from the basic scheme definition. These tests are required to ensure that the basic case represents the optimum configuration, that is, it provides the best solution, but also to determine the impact on the scheme's value for money of unforeseen changes.

1.5.15 Where the scheme or strategy includes elements that could be introduced separately, the full package should be appraised but the contribution of separable elements should be identified. This will need to be done using sensitivity tests isolating elements of the strategy that should identify clearly how each element contributes to the overall scheme.

1.6 Analysing Uncertainty

1.6.1 The appraisal must consider at least two sensitivity tests or alternative scenarios apart from the core scenario, to provide an understanding of the range of uncertainty around the core scenario results. Appropriate assumptions need to be drawn up with reference to the uncertainty log.

1.6.2 The number of possible sensitivity tests may be very large, and it is good practice to adopt a structured approach to keep the number down to a manageable level. One way to address this is to group input uncertainties together to form alternative scenarios - further advice on that is given below.

1.6.3 The sensitivity tests or alternative scenarios should then be assessed to provide appraisal information about the recommended proposals. This will give decision-makers an idea of the robustness of the proposal. If traffic levels/patronage and the resulting appraisal do not vary significantly between the sensitivity tests or alternative scenarios examined, this demonstrates that the project is not sensitive to external uncertainties. Conversely, if there are significant variations in patronage, the decision maker will have a clear view of the risks the project faces, and the analyst can explore ways of mitigating those risks.

1.6.4 If the economic case for the scheme and the design of the scheme are robust to the tests outlined in this section, then the initial Green Book requirement for sensitivity testing against forecast uncertainty will have been met. Conversely, if these tests suggest that the case for the scheme depends on one or more other schemes, land-use changes or policies happening or not happening, then further tests may be needed to identify more precisely the dependencies involved and the likelihood of the individual events. Such a result is not a failure - a successful appraisal is one that tells decision-makers what they need to know, including identification of dependencies.

Defining sensitivity tests

1.6.5 The considerations for deciding on sensitivity tests for any scheme are concerned with the scheme's robustness in the light of uncertainty in demand levels. In particular, they are likely to be:

  • Does it work - is congestion kept to a reasonable level - in the face of possible high demand?
  • Is it economically viable in the face of possible low demand?

1.6.6 To this end most schemes and strategies should be tested to incorporate national economic uncertainties by adopting low and high reference case traffic forecasts as discussed earlier or, for public transport schemes, variations in the national economic parameters themselves. These tests are considered most relevant as sensitivity tests applied to the core scenario, although it may be appropriate to test alternative scenarios in this way as well.

1.6.7 In most cases, further sensitivity tests will be required. These should be based on proposals included in the uncertainty log, discussed above. It may be appropriate to carry out sensitivity tests including land use and transport proposals that have been excluded from the core scenario. Similarly, it may be appropriate to carry out sensitivity tests excluding land use and transport proposals that are less than 'near certain' but that have been included in the core scenario. In some cases, sensitivity tests may be defined in terms of the scale of the proposal, rather than whether or not it will proceed. For example, a sensitivity test might explore the consequences of 50 houses being built on a site, rather than the planned 200.

1.6.8 Further sensitivity testing may also be necessary employing alternative parameter values, where scrutiny of the base year model suggests that there is significant uncertainty in values. Further discussion of these issues is given in DMRB Volume 12 and Variable Demand Modelling Detailed Stages (TAG Unit 3.10).

Land-use / Transport Interaction

1.6.9 In developing sensitivity tests or alternative scenarios for testing it is important to recognise the interactions between transport and land-use.

1.6.10 Thinking first about the extent to which land-use is dependent on transport, the assessment of likelihood of major land-use changes needs to recognise that such changes will fall into three categories.

  • Some will be considered to be dependent on the scheme being appraised
  • Some will be considered to be dependent on other proposed transport measures
  • Some will be independent.

Where in this case "dependent" can be interpreted as being more likely than not if the relevant transport measure is implemented (that is, categorised as 'near certain' or more than likely'), but less likely than not in its absence (that is, at best 'hypothetical').

1.6.11 Changing the transport assumptions in the sensitivity tests or alternative scenarios may therefore have associated land-use changes. For land use assumptions, these sensitivity tests or alternative scenarios should use the same threshold of likelihood as in the core scenario, but because of dependencies the set of proposed land-use developments which meets this criterion will change and may include some inputs that are categorised 'reasonably foreseeable'.

1.6.12 Where land-use is dependent on the scheme being appraised, this may cause difficulties for the economic appraisal of the scheme, but that is not a good reason for denying the existence of such dependency. It may not be possible to complete a full economic appraisal of such an outcome, but it is important to get an idea of the likely impact on scheme patronage. Further advice on economic appraisal is provided in The Economy Objective (TAG Unit 3.5).

1.6.13 Consider also the converse relationship, the dependence of transport improvements on factors (like land-use) which affect transport demand. If at the modelling stage it becomes clear that transport conditions under any scenario and demand level are such that further improvements to the transport system are likely they should be included at that stage, even if they weren't originally included in the scenario. Such additions have historically been described as do-minimum traffic management improvements.

Using alternative scenarios

1.6.14 In some circumstances it may be helpful to bring a range of sensitivity tests together to form alternative scenarios. When deriving alternative scenarios, it is important to take a coherent view of the outcomes, so that each scenario represents a realistic future. Clearly, the scenarios will often be tailored to the scheme being considered. But this should not lead to unbalanced or otherwise unrealistic scenarios.

1.6.15 Consideration needs to be given to the interaction between different inputs and any interdependence needs to be accounted for. This may lead to more than two alternative scenario tests to enable the proposals to be tested against a range of assumptions that aren't necessarily easily bundled together.

1.6.16 As for sensitivity tests, alternative scenarios should be based on proposals included in the uncertainty log, discussed above. Alternative scenarios may be developed in a number of different ways, including:

  • Scenarios based on the assumption that current plans are delivered either more or less completely than is assumed for the core scenario. This kind of scenario could be developed by raising or lowering the uncertainty threshold adopted when developing the core scenario. For example, one scenario could be based on the inclusion of only 'near certain' proposal, while the other could include all 'reasonably foreseeable' proposals.
  • Scenarios designed to lead to extremes of demand (traffic levels or patronage) on the scheme itself. For example, a scenario in which competing transport schemes are implemented, thus reducing demand, and another in which development occurs that generates increased demand on the scheme.
  • Scenarios that better represent the world-view of particular stakeholders.

1.7 Reporting

1.7.1 Each scenario or sensitivity test (as a minimum, the core scenario and two alternative scenarios or sensitivity tests) should form the basis for a full appraisal, including environmental and other impacts where appropriate. It is expected that the core scenario will be reported in the Appraisal Summary Table, with any exceptional outcomes of the uncertainty analysis also included. For example if a particular strategy has significantly different results in an alternative scenario that would affect the AST score in a particular category, then that information will be included as qualitative comments (but quantifying the difference) in the AST.

1.7.2 The results of the uncertainty analysis, both absolute and in terms of variations from the core analysis should be reported as part of the model forecasting report. This analysis would include a report of the uncertainty log, composition and likelihood of occurrence of each sensitivity test or alternative scenario assessed, and model results.

1.8 Summary

1.8.1 The methodology outlined above is expected to provide a consistent approach to the treatment of uncertainty that enables the analyst to allow for local variations in a systematic and transparent way.

1.8.2 The procedure will enable an assessment of the recommended proposals to a range of assumptions. Ideally the proposals will remain robust for all sensitivity tests and alernative scenarios, conversely, it may identify that the case for a particular transport measure is dependent on some other proposal, or on a particular local or national demographic or economic outcome. This information will be key to the decision making process.

1.8.3 The information on the uncertainty analysis should be reported, detailing the likelihood of the sensitivity tests or alternative scenarios tested and the appraisal results obtained. Any significant change to the results of the core analysis should be reported on the AST for the scheme.


2. Further Information

For information on: See: TAG Unit number:
General transport modelling Summary Advice on Modelling Unit 2.4
Variable demand modelling Variable Demand Modelling - Overview of Advice Unit 2.9
Public transport modelling Introduction to Forecasting Models for Major Public Transport Schemes Unit 2.10
Economic appraisal advice and trends in values of time and operating costs The Economy Objective Unit 3.5


3. References

Department of the Environment, Transport and the Regions. Design Manual for Roads and Bridges, Volume 12.


4. Document Provenance

This Document was produced in July 2004 to formalise the treatment of uncertainty in forecasting. It has been revised in April 2008.

This Unit became definitive guidance in April 2009.

Technical queries and comments on this TAG Unit should be referred to:

Integrated Transport Economic Appraisal (ITEA) Division
Department for Transport
Zone 3/08 Great Minster House
33 Horseferry Road
London SW1P 4DR
Email: itea@dft.gsi.gov.uk
Tel: 020 7944 6176
Fax: 020 7944 2198

Updated: April 2011