While cost-benefit analysis is a standard economic approach for appraising any proposed public investment projects, Jon Stern questions its robustness—particularly when applied to large projects covering many areas and/or markets—and what other uncertainties need to be examined.
Since the 1950s, cost-benefit analysis (CBA) has become the overwhelmingly dominant investment appraisal method for infrastructure and other major public sector investment projects. In June 2022, the World Bank website had 359,000 entries under this heading and the UK Government website listed almost 123,000 entries. However, evaluation reports show increased problems with CBA appraisals. In 2010, the World Bank published a review of CBA use on their projects and only 54% received an acceptable or good rating as opposed to 70% in the 1990s, even though the number of schemes appraised by CBA was much lower (25% as opposed to 70% in 1970). Not only did the review find a lack of transparency and an inherent optimism bias, it also found that these problems persisted even though they had been identified 20 years earlier.
CBA is a method of estimating the resource costs and benefits of any project. The first known CBA, on bridge building, was carried out in the 1840s by Jules Dupuit – a French engineer and economist. That paper generated no interest and only re-emerged in the 1950s. However, from 1950, US economists, led by Otto Eckstein developed modern CBA methods for the appraisal of water resource and flood management problems, which were published as annual Green Book reports. The CBA approach then became used for a growing range of public projects in the USA, the UK and elsewhere, particularly transport projects.
How are they used? CBA results are clearly advisory and non-determinative. Large numbers of proposed projects with positive CBAs are not proceeded with and a goodly number of projects with very low or negative CBAs are implemented for essentially political and other non-economic reasons. CBAs are, in practice, very largely a sifting tool to weed out economically-oriented public investments with low expected returns.
Two sets of institutions are involved in CBA appraisal:
- Central control and methodology setting institutions; and
- Government spending departments and agencies plus a wide range of consultancies, companies, lobby groups, etc. who prepare and submit CBAs in support of specific proposals.
Both the UK Treasury and the World Bank are in the first group. Firstly, they need to consider and approve or reject spending proposals submitted to them. Secondly, they need to create, monitor and update the CBA methodologies used by the entities submitting the CBAs. For the UK Treasury, the main aim of the methodology is to ensure that all spending departments and public agencies are treated in the same way and with the same degree of oversight. It is imperative that the Department for Business cannot claim that it is being treated unfairly relative to the Department for Education (or vice-versa). (For the World Bank Group, the same imperative applies to all the countries and to all the sectors on which the Bank lends or provides subsidies)[1].
The second group of institutions includes a wide range of entities, commercial and non-commercial, all of whom are making a case for a specific public expenditure (or aid) project or programme. They are therefore not impartial. The range of the resulting CBAs varies … and so does the quality. They range from the thorough and scholarly to the blatantly biased and distorted. However, they are all partial and all are trying to make a good case as to why their expenditure proposal should be adopted under the existing CBA methodology – or, failing that, why the methodology should be changed to accommodate it.
In economic theory terms, CBAs use a partial equilibrium approach. Hence, they are well-suited to appraisals for single small-medium sized projects (e.g. a new refuse facility in a town), but become increasingly problematic when applied to very large projects covering many areas and/or markets. HS2 is a good example of this. The main advocates of the scheme were the Midland and Northern cities who argued that they expected major additional economic activity in their areas as a result of HS2. However, Overman and Glaister have argued that London and the South could well be the main beneficiaries. A general equilibrium or multi-market CBA method was not available so these competing views could not be tested within the CBA – which was unable to make any significant contribution to that debate. Hence, as Martin Hurst has argued, CBA methods face major problems with mega projects like HS2 and airports.
CBA projections have come under increasing criticism in recent years. It has long been known that, in CBAs, forecasting benefits is much harder than forecasting costs, particularly if intangible benefits and externalities are important. However, recent work by Flyvbjerg and colleagues on a sample of 2,062 CBAs has shown that cost projections are regularly under-estimated – by 39% on average – and biased. The size of the biases has remained constant for over 80 years. In the UK, large cost underestimates have been a feature of major rail projects besides HS2. Benefit estimate problems continue.
In the UK, CBA practice is most associated with the Treasury Green Book. The 1997 Green Book still reads very much like a CBA manual, although by then, it was increasingly sprouting annexes on evaluation, discount rates, etc. However, in 2020, the Green Book was completely revamped with resource cost CBA playing a much smaller role. A Green Book investment appraisal was now to include (i) a strategic review, (ii) a commercial review, (iii) a financial review and (iv) a management review – with these given equal importance to the CBA. It seems as though the Flyvbjerg and other critiques of CBA empirical performance have played a major role in this respecification of the Green Book.
A 2010 survey by Robinson and Hamitt suggests that CBA is probably best viewed now “… as a pragmatic framework for collecting, organizing and evaluating relevant information [on public investment and major public sector expenditure programmes]”. Within this, they rightly emphasise the need for extensive examination of uncertainties including sensitivity, probabilistic and breakeven analysis. Such an approach – which I support – seems to be the new current consensus for CBAs, including the new UK Green Book. Will it prove lasting? – only time will tell.
[1] At this point, I should declare an interest. In the 1980s, I was a member of the HM Treasury Micro-economics unit on public investment appraisal and the Treasury Green Book, including CBAs. I also worked as a consultant to the World Bank from 1993-2006, although primarily on the economic regulation of infrastructure industries.
The views and opinions expressed in this post are those of the author(s) and not necessarily those of the Bennett Institute for Public Policy.
