abstract. Some members of Congress, the D.C. Circuit, and the legal academy are promoting a particular, abstract form of cost-benefit analysis for financial regulation: judicially enforced quantification. How would CBA work in practice, if applied to specific, important, representative rules, and what is the alternative? Detailed case studies of six rules—(1) disclosure rules under Sarbanes-Oxley section 404; (2) the SEC’s mutual fund governance reforms; (3) Basel III’s heightened capital requirements for banks; (4) the Volcker Rule; (5) the SEC’s cross-border swap proposals; and (6) the FSA’s mortgage reforms—show that precise, reliable, quantified CBA remains unfeasible. Quantified CBA of such rules can be no more than “guesstimated,” as it entails (a) causal inferences that are unreliable under standard regulatory conditions; (b) the use of problematic data; and/or (c) the same contestable, assumption-sensitive macroeconomic and/or political modeling used to make monetary policy, which even CBA advocates would exempt from CBA laws. Expert judgment remains an inevitable part of what advocates label “gold-standard” quantified CBA, because finance is central to the economy, is social and political, and is non-stationary. Judicial review of quantified CBA can be expected to do more to camouflage discretionary choices than to discipline agencies or promote democracy.

author. John F. Cogan Jr. Professor of Law and Economics, Harvard Law School. Thanks for helpful discussions—but no blame for the contents of this paper—should go to Stephen Ansolabehere, John Armour, Michael Barr, Ryan Bubb, John Campbell, Mark Cohen, Clarke Cooper, Jim Cox, Paul Davies, Mihir Desai, Nancy Doyle, Eilis Ferran, Jeff Frieden, Jeff Gordon, Howell Jackson, Robert Jackson, Louis Kaplow, Duncan Kennedy, Andrei Kirilenko, Bruce Kraus, Alex Lee, Craig Lewis, John Manning, Miguel de la Mano, Tom Merrill, Robert Plaze, Eric Posner, Connor Raso, Mark Roe, Paul Rose, Ava Scheibler, Hal Scott, Holger Spamann, Suraj Srinivasan, Matthew Stephenson, Larry Summers, Cass Sunstein, Meg Tahyar, Dan Tarullo, Adrian Vermeule, Chris Walker, Scott Westfahl, Glen Weyl, and Richard Zeckhauser, and to workshop participants at Harvard Law School, Harvard Business School, Columbia Law School, the Securities and Exchange Commission, the Commodity Futures Trading Commission, and the Public Company Accounting Oversight Board. Any errors are mine.

For disclosure of financial interests potentially relevant to this Article, see Faculty Disclosures re: Related Outside Interests and Activities, Harv. L. Sch., http://www.law.harvard.edu /faculty/COI/2012_Coates_John.html [http://perma.cc/M3VN-8K2P]; and Faculty Disclosures re: Related Outside Interests and Activities, Harv. L. Sch., http://www.law.harvard.edu/faculty/COI /2013_Coates_John.html [http://perma.cc/TTH6-LNFE].

Introduction

A movement is afoot to impose cost-benefit analysis (CBA) on financial regulation (CBA/FR).1 The housing and financial crises of 2008 led to the Dodd-Frank Act,2 which restructured the financial regulatory agencies, mandated more than 200 new rules, and required changes to many older rules.3 The sweep of regulatory change has reignited criticism for failure to base the changes on adequate CBA/FR.4 Bills have been introduced to provide explicit authority for the President to require CBA/FR from independent agencies,5 even as critics argue that existing law already requires the Securities and Exchange Commission (SEC) and the Commodity Futures Trading Commission (CFTC) to conduct a particular form of CBA/FR: judicially enforced quantification.6 One panel of the United States Court of Appeals for the District of Columbia Circuit, composed entirely of Republican-appointed judges, held that existing law requires the SEC to quantify the costs and benefits of its proposed rules,7 while another judge—appointed by President Obama to the D.C. Circuit—subsequently held that such quantification is not mandatory, at least when the SEC is required by statute to adopt a rule, and the benefits sought to be achieved are humanitarian and not economic in nature.8

This Article critiques efforts to impose judicially reviewed, quantified CBA on independent financial agencies, while also attempting both to explore how conceptual CBA/FR could lead to better policy and to advance the substantive project of quantitative CBA/FR itself. This combination of objectives represents a moderate stance, between the polar positions that often characterize debates over CBA; the Article neither rejects it utterly nor embraces it naively. Rather, the Article explores how CBA is likely to function in the near term as applied to financial regulation and assesses the costs and benefits of using CBA/FR. In other words, the Article begins to develop a CBA of CBA/FR itself. The results of the exploration not only call into question simplistic efforts to mandate CBA—particularly quantified CBA, and particularly when enforced through judicial review by generalist courts—but also should help those who favor economic analysis of law to appreciate how CBA might advance and clarify policy analysis of financial regulation, rather than retard or obscure it.

Part I analyzes CBA generally, noting that it (a) can be either a framework for policy analysis or a legal means to discipline agencies and (b) can consist of either conceptual analysis or efforts at quantification. Part I also briefly reviews CBA’s origins in U.S. legal history to show that it can be used to camouflage as well as to discipline, referring to the Taylor Rule to explain why even CBA’s advocates do not propose to require CBA for monetary policy. Often, CBA is defended in part on the grounds that supposed alternatives—such as expert discretionary judgment—are no better, and often worse, than CBA. In fact, Part I suggests, CBA may turn out not to be an alternative to reliance on judgment: instead, expert judgment is a core and necessary component of CBA, as it is for any process of assessing and adopting financial regulations.

Part II describes existing law relevant to CBA/FR and investigates ongoing efforts to promote quantified CBA/FR. Chief among these efforts has been a string of high-profile CBA cases over the last decade in which courts have struck down financial regulations. Part II critically assesses those cases, showing they have been poorly reasoned, premised on mistakes, inconsistent with precedent, and based on misunderstandings about what CBA/FR can reasonably be expected to do. Nevertheless, those decisions have fueled efforts in the agencies themselves to undertake more CBA/FR. More problematically, those cases have also fueled efforts in Congress to give courts an even more expanded role in enforcing a general mandate for the independent agencies to include quantified CBA in rulemaking.

Part III develops case studies of how quantified CBA/FR might be conducted on six significant and representative financial regulations, drawing on relevant academic research to outline the tasks that need to be tackled to conduct CBA/FR on those rules. The case studies show that quantified CBA/FR amounts to no more than “guesstimation,” entailing: (a) causal inferences that are unreliable under standard regulatory conditions; (b) the use of problematic data; and/or (c) the same kinds of contestable, assumption-sensitive macroeconomic or political modeling used to make monetary policy.

Part IV concludes by reviewing the implications of the case studies. Anyone who supports CBA should agree that CBA should be conducted only to the extent it passes its own test—that is, only if CBA itself will produce more benefits than costs. Perhaps surprisingly, given that CBA has been part of administrative law for decades, CBA of CBA has itself never been adequately conducted, leaving the first-stage choice of when to perform CBA/FR itself in the realm of judgment rather than science. Part IV begins the task of outlining a CBA of CBA, both generally and in the context of financial regulation. It argues that the benefits of CBA/FR have been low in the past and are likely to remain low in the near future, while its costs will depend on the precise institutional and legal context in which it is pursued.

CBA/FR’s benefits are likely to remain low because it is by definition about finance: finance is at the heart of the economy; is social and political; and is characterized by non-stationary relationships that exhibit secular change (that is, long-term structural changes). These features undermine the ability of science to precisely and reliably estimate the effects of financial regulations, even retrospectively. Whenever agencies face such sensitive and speculative forecasting abilities, quantified CBA is not capable of disciplining regulatory analysis. It will generate low benefits in the form of reduced agency costs (in part by counteracting cognitive biases) or increased transparency. Moreover, CBA/FR will produce costs: resources consumed, regulatory delay, diffusion of regulatory focus, and potential decreases in regulatory transparency—particularly if regulatory agencies and courts involved in reviewing agency action do not have strong incentives to be honest about the limits of the results.

At the same time, CBA/FR is a useful conceptual framework, and quantified CBA/FR is a worthy long-term research goal. Attempts to quantify may advance the research needed to achieve reliable, precise estimates, and this makes quantified CBA/FR a worthwhile project for agencies to pursue. But the current benefits of CBA/FR remain low, because their real effects remain far off in time; like any regulatory benefits, the benefits of CBA/FR should be discounted to present value.

Completing a full, quantified CBA of CBA would require evidence and new research methods: studies of the degree to which CBA results in better regulations or more transparency in the regulatory process, as well as quantified estimates of the costs—delay, confusion, camouflage, partisanship—that CBA can introduce. Until evidence is developed to illuminate when CBA/FR passes its own test, courts and secondary agencies (that is, agencies other than those charged with rulemaking responsibility) should have no role in second-guessing the choice of when to conduct CBA/FR, or the details of CBA/FR when it is used.9 Not only should new legal CBA/FR mandates be resisted as likely to worsen policy outcomes, but existing interpretations of the Administrative Procedure Act (APA) and financial agencies’ governing statutes should also be reversed. A safe harbor should be created to shelter the CBA/FR that the agencies choose to conduct, so as to reduce the influence of concentrated interests through litigation and of politically partisan but unaccountable judges on regulatory outcomes.10 In sum, CBA/FR remains a potentially valuable regulatory tool, but only if implemented with a light touch.

As reflected in Part IV, this Article’s critique of CBA/FR is not sweeping. Rather, it is focused on one specific institutional arrangement for CBA/FR: mandates (whether through new statutes or judicial interpretations of existing statutes) for judicially reviewed, quantified CBA. Other arrangements that include CBA—such as the use of conceptual CBA on a voluntary basis by independent agencies—are much more promising. In between are a wide variety of possible arrangements, such as interagency review of CBA/FR (whether conceptual or quantitative) by a separate agency, as is currently done for rulemakings by executive agencies. Each such arrangement deserves its own fact-specific analysis. For example, for any interagency process, one should ask: How much of the interagency dialogue would become part of the public record, available for use in a subsequent judicial challenge? What real resources could the alternative agency bring to bear on the discussion? Would that other agency face genuinely different incentives in evaluating a given regulation, and how much value would participation by such an agency add if included in pre-rulemaking discussions? How important is it to achieve uniformity on specific kinds of CBA inputs, and alternatively, how important is it to allow for flexibility in such inputs over time and across agencies? As a result of the complexity of these questions, the full range of possible alternative institutional arrangements is not analyzed in detail in this Article. However, some of the reasons offered as to why judicially reviewed quantitative CBA/FR may not satisfy a cost-benefit test may also extend to those other arrangements, and the analysis here should at least illuminate policy debates over those alternatives.

I. what do people mean by “cost-benefit analysis”?

The literature on cost-benefit analysis is voluminous and multi-disciplinary.11 Not surprisingly, writers often talk past one another when they discuss the topic. Three distinctions are often elided: whether by CBA one means policy analysis or law; whether by CBA one means a conceptual framework or quantification; and whether CBA is likely to camouflage or discipline regulation.12 In this Part, I begin by presenting a brief typology of CBA and conclude by sketching the alternatives to CBA.

A. Policy Versus Law

Lawyers instinctively understand the difference between a norm or a policy, on the one hand, and a law, on the other—even when that law tracks a norm or policy. They know, for example, that the effects of a law (assumed to be justiciable) requiring an agency to act reasonably will not simply equate to the actions that an agency, acting reasonably, would take. A requirement imposes a set of burdens on the agency that the demands of reason do not. Law introduces new agents into the picture—usually, courts. Those agents are no more perfect than others, and their decisions will be uncertain. Agencies subject to court oversight will anticipate judicial error (or bias).

A law will lead an agency to keep more careful track of what it does, and why, than reason on its own would do. Agencies will incur costs to keep track in this way, just as they will incur costs to defend decisions against court challenges. They will refrain from acting when the expected cost of a challenge and record keeping falls below the expected benefit of the action, discounted for the risk that the court will wrongly overturn the decision. These consequences arise from enforcement and oversight by courts. Law changes behavior even when a law on its face only requires what someone would try to do anyway.13

Lawyers also know that a law requiring an agency to act reasonably will sound innocuous to most non-lawyers: who could be against acting reasonably? Lawyers know that non-lawyers systematically underestimate enforcement costs and their effects. They know that a clever way to shape regulation is to propose a law that tracks a general norm, the enforcement of which will have predictable effects that are not intuitive to non-lawyers. The asymmetry in perceived effect will allow political gains at a lower political cost than a straightforward law mandating or forbidding regulation.

These themes play themselves out when lawyers discuss CBA with non-lawyers. Specifically, non-lawyers typically mean by CBA the conduct of cost-benefit analysis itself—whether by researchers, regulators, or courts. Lawyers sometimes use CBA in the same way, referring to a particular type of policy analysis. But lawyers also often mean by CBA a set of legal requirements aimed at inducing regulatory agencies to conduct CBA exclusively or as part of their policy analysis in choosing to adopt or change regulations. When lawmakers, for example, describe a proposed law as requiring CBA, many non-lawyers will think of CBA as policy analysis and, if they favor using CBA in policy analysis, will assume that the law is a good idea. They will effectively conflate CBA as policy analysis with CBA as legal requirement. As with a requirement of reasonableness, however, a requirement of CBA will predictably have effects that diverge from those that would arise if CBA were simply used as a routine part of an agency’s policy toolkit, without a legal requirement.14

B. Quantities (or Guesstimates) Versus Concepts

A second source of confusion arises even within CBA as policy analysis. Most advocates of CBA expect it to include quantification and monetization. This type of cost-benefit analysis—if supported by strong consensus theory, reliable research designs, and good, representative evidence—could properly be called quantifiedCBA,15 but—if supported only by weak, contested theory, unreliable research designs, or poor, unrepresentative evidence—better deserves the label guesstimated CBA. Robert W. Hahn and co-authors, for example, criticized executive agencies for failing to comply with Executive Orders requiring CBA,16 based on the authors’ assessment that

agencies only quantified net benefits—the dollar value of expected benefits minus expected costs—for 29 percent of the forty-eight rules [reviewed by the authors], even though the Executive Order directs agencies to show that the benefits of a regulation “justify” the costs. . . . Although agencies may present reasons not to quantify and monetize benefits and costs, . . . we believe they should be able to meet the requirements of the Executive Order for a majority of regulations.17

Expectations of quantification have found their way into legal decisions overturning financial regulations, as discussed in Part II. For example, in Chamber of Commerce v. SEC, the D.C. Circuit held that the SEC acted arbitrarily and capriciously for failing to undertake some effort to quantify the costs of the mutual fund governance rule changes it had adopted.18

Others accept—indeed, often make rhetorical show of conceding19—that quantification or monetization is not possible in some policy areas but nonetheless believe that CBA can function as a disciplined framework for specifying baselines and alternatives, for ensuring that (at least conceptually) both costs and benefits of a rule are considered, and for encouraging reliance on “evidence” rather than solely on intuitive judgment.20 These types of CBA are best distinguished from quantified or guesstimated CBA with the label conceptual CBA.

Transforming conceptual CBA into quantified CBA is not an all-or-nothing proposition. Some effects of a given rule might be reliably quantified and monetized, while others might not be. Some inputs to CBA may be quantified, for example, to “scope” the domain of a proposed rule—how many people, transactions, entities, and the like would be covered by the rule. But quantified CBA in its ideal form—which some of its advocates refer to as “complete” quantified CBA21—entails specification and quantification of all benefits and costs in a single, uniform bottom-line metric (typically, dollars) representing the net welfare effects of a proposed rule. Some CBA supporters acknowledge that such an idealized version will not be feasible in “some” instances and have conceded that in such instances a more limited CBA—guesstimated CBA—should not determine regulatory outcomes. For example, in a 1996 policy article in Science, Kenneth Arrow and ten other economists advocated CBA but were careful to note that

[b]enefits and costs of proposed policies should be quantified wherever possible. . . . In most instances, it should be possible to describe the effects of proposed policy changes in quantitative terms; however, not all impacts can be quantified, let alone be given a monetary value. Therefore, care should be taken to assure that quantitative factors do not dominate important qualitative factors in decision-making.22

Particularly difficult to quantify or monetize are non-market goods and externalities. In non-financial regulatory domains, non-market goods, such as life, health, beauty, and biodiversity, have proven difficult to monetize with any degree of precision and confidence.23

In financial regulation, relevant non-market goods include trust, investor confidence, liquidity, and the psychological consequences of unexpected financial losses.24 In non-financial regulation, measurement of externalities has proven difficult, not only because these externalities are often non-market goods, but also because simply specifying and estimating their size is challenging. Financial regulation poses equally if not more difficult problems in measuring externalities, in part because financial markets are tightly interconnected systems (hence the now mainstream phrase “systemic risk”), in which one party’s losses can be rapidly transmitted to multiple related parties.25 As explained in Part III and discussed further in Part IV, full quantification in CBA/FR is likely to be difficult because finance is at the heart of the economy, involves groups of people (firms, markets) interacting in complex, difficult-to-study ways, and is shaped by forces that change rapidly over time.

Short of full monetization, CBA can include efforts to estimate ranges of costs and benefits, to bound them, to conduct “threshold” analyses comparing a rule’s quantified costs to unquantifiable benefits (or vice versa), and, more generally, to use empirical methods and data to generate evidence relevant to quantified or conceptual CBA. While ranges, bounds, threshold analyses, and incomplete but relevant evidence may all be viewed as part of quantified CBA, they begin to move the final result of CBA toward guesstimation, leaving it a matter of judgment whether and how the results of CBA should influence decision making. For example, guidelines from the Office of Management and Budget (OMB) provide little help in determining how to conduct threshold analyses if important benefits and costs are both unquantifiable, simply suggesting that agencies “exercise professional judgment” in weighing unquantifiable elements in the CBA.26 This recommendation is hard to criticize. But it also suggests that there may be circumstances in which a feasible but partial quantification will not be cost-justified. For example, it may be the case that the quantifiable elements are likely (based on judgment) to be trivial relative to the unquantifiable elements. It may also be that partial quantification is costly, or otherwise will undermine the value of a conceptual CBA, by—for example—conveying a false degree of precision to a general audience.

One also can draw a distinction within CBA law—analogous to the one between conceptual and quantified CBA—between CBA mandates and CBA process,although this is not typical in prior CBA scholarship. CBA mandates consist of efforts to require agencies to conduct some or all elements of CBA policy—presumably because legislators believe agencies must be forced to conduct it. CBA mandates include laws subjecting the CBA policy analysis itself to review by another agency (such as the Office of Information and Regulatory Affairs (OIRA), a unit of OMB), or by courts (as in review of rules as “arbitrary” and “capricious” under the APA).27 The objectives of this review are to ensure that the agencies take statutory CBA mandates seriously and (in theory) to improve the quality of CBA analyses. CBA mandates encompass binding executive orders or other interagency guidelines that specify particular components of CBA policy analysis, such as discount rates, or methods to quantify benefits or costs, with the goal of achieving uniformity across governmental agencies.28 Finally, CBA mandates can be a component of regulation itself—that is, an agency could require a private actor to demonstrate that a new activity or product would have greater benefits than costs before it could be permissibly sold.29

CBA process, by contrast, includes requirements for agencies to publicly disclose any CBA they conduct, or the sources of their data, and to solicit public comment and feedback on their CBA analyses (as under the APA).30 CBA process laws can require agencies to discuss how they took comments into account in their final rulemaking decision, to present their CBAs in particular or standardized formats, or to include specific kinds of information, such as standard statistics or data analyses that bear on the reliability of the primary findings of a quantified CBA. Such indicators of reliability include, for example, confidence intervals, p-values, test statistics, correlation matrices, sensitivity analyses, and the results of “Monte Carlo” simulations. Such “soft law” requirements may be viewed as a means of enhancing the quality of the agencies’ decisions by encouraging deliberation and care, or as a means of increasing public understanding and the legitimacy of adopted rules. These process requirements can also have less desirable effects, however, including delay, regulatory inertia, ill-informed judicial second-guessing, creation of incentives for agencies to engage in CBA for show, and waste of regulatory resources.

Table 1.

dimensions of cost-benefit analysis

Putting the first two dimensions of CBA together, Table 1 illustrates the multiple meanings that apparently synonymous uses of “cost-benefit analysis” might have for different speakers or audiences. Table 1 suggests that it is possible to be an advocate for CBA/FR—whether conceptual or quantified—as a form of policy analysis without wanting to entangle it in the legal system; or that it is possible to favor efforts to quantify CBA/FR without wanting to mandate quantification. One might even be skeptical that CBA/FR law will have any effect at all.31 Alternatively, if CBA/FR has clear virtues as policy analysis, one might believe that those virtues would lead agencies to use it, at least sometimes, without being legally required to do so, just as private businesses adopt “best practices” on a voluntary basis. Likewise, one can favor CBA/FR process laws without agreeing that courts or any other agency should have any substantive role in evaluating or constraining the content of CBA/FR. Or one could imagine mandating that a second political agent (a specialized court or another agency) conduct the CBA/FR analysis itself; the analysis would then have to be used by the primary agencies as inputs into their rulemaking decisions, without necessarily adding other process requirements to CBA/FR law.

C. Camouflage Versus Discipline

A third dimension along which CBA can vary is the motive of the person using it—and, relatedly, its effects on third parties. The conventional, optimistic view of CBA advocates—generally assumed or asserted rather than supported with evidence32—is that CBA is an agency cost-control device, used by politically accountable representatives (Congress or the President) to discipline expert but less accountable agencies (made up of appointed bureaucrats) in their rulemaking efforts. In this view, CBA will improve the care that agencies exercise in deciding whether a possible rule change is good for society while limiting agencies’ ability to adopt welfare-reducing rules.33 CBA optimists tend to assume or assert that CBA will enhance public understanding of why regulations are adopted (increase transparency)34 and engage more people in the democratic process, potentially combating pernicious rent seeking by special interests.35 By specifying how a rule will produce benefits, by acknowledging the costs involved, and by encouraging the consideration of alternatives, CBA is expected to improve the allocation of governmental resources and reduce the drag of regulation on beneficial activities.36 Some but not all CBA optimists even assert that CBA can mitigate cognitive biases of regulators or the public.37

Despite having potential virtues, however, CBA can have a different, darker, or more complex mix of effects. It can provide camouflage, reducing the transparency of a rulemaking process.38 More disclosure does not always improve transparency, a point that (ironically) some CBA advocates have made strenuously when resisting disclosure rules for private actors.39 Beyond the indeterminate effects of CBA soft law on the ability of the public to monitor regulatory agencies, CBA can also be a tool of political struggle over the distribution of rents, and it can serve as a means to increase the power of unelected expert agents as a tactic in that struggle.40

The origins of CBA in the United States illustrate this set of possibilities. It is commonly asserted that Congress “initiated the use of CBA in 1936, when [it] ordered agencies to weigh the costs and benefits of projects designed for flood control,”41 permitting authorization of such projects only if “the benefits to whomsoever they accrue are in excess of the estimated costs.”42 This origin story fits the optimistic view of CBA outlined above, making it a mechanism used by elected and accountable representatives to control costs at a wayward agency. In fact, however, the use of CBA by the Army Corps of Engineers emerged earlier, on the initiative of the Corps itself, as described in Theodore M. Porter’s Trust in Numbers.43 In Porter’s telling, the first efforts at CBA occurred in 1902, with the creation of a board within the Corps; opponents of public works spending hoped that the board’s performance of cost-benefit analysis and its issuance of recommendations would “reduce opportunities for purely political choices.”44 Rather than ranking all projects based on CBA, which would have systematized project choice, the Corps chose to maintain flexibility, “recognizing, it seems, that congressional choice was the key to congressional favor.”45 Far from being a tool for the management of the Corps, CBA became a tool by some politicians and by the Corps to manipulate Congress.

The Corps had developed a “huge civilian labor force” prior to the 1936 Flood Control Act, which mandated strict CBA for new projects. That Act, too, Porter concludes, was not aimed at disciplining the Corps, but was “one of the heroic efforts of the United States Congress to control its own bad habits.”46 The Act’s requirements, and particularly the delay requirement, were viewed as a benefit, and not a necessary cost, of conducting CBA: “A preliminary examination and then a full survey, each running through several levels of Corps bureaucracy, required months or years, and could not be completed to satisfy the sudden whim of a legislator.”47 Far from reducing the power of the Corps, the regularization of the project approval process (and the implementation of CBA) enhanced it, because neither Congress nor the public exerted the effort needed to evaluate and assess the Corps’s numerically impressive but sometimes ad hoc analyses: “The numbers were almost never questioned.”48

If some members of Congress favored a particular outcome, they could attempt to “manage” the Corps by finding unorthodox benefits to “quantify” (or include in a guesstimated CBA). One local district’s engineer, faced with an unfavorable CBA report based solely on flood control benefits, “developed other benefits that he did not find . . . necessary to develop when he wrote his main report,” including benefits from downstream power, pollution abatement, and improved water supply.49 Over time, more benefits were guesstimated, and previously rejected projects were accepted.50 The result, in Porter’s view, was that “Corps economic methods [that is, its CBA] could not, by themselves, determine the outcome of an investigation.”51 This observation was particularly true when powerful interest groups, such as the utility and railroad industries, or other regulatory bodies, such as units of the Department of Agriculture or the Interior Department, opposed the Corps’s initial conclusions.52

In sum, CBA can in principle provide public-regarding benefits by disciplining agencies, increasing transparency, and enhancing the public’s engagement with the regulatory process. In theory, CBA can reduce agency costs associated with delegation by politically accountable lawmakers to expert but less accountable agencies. But CBA can have other effects beyond direct costs of the CBA itself. These effects include use of technically opaque analytics to (1) obscure the issues at play, (2) raise the risks for lawmakers to question regulators, (3) shift power from Congress to regulators, (4) hide rent seeking, and (5) favor factions in distributional struggles among lawmakers. One form of camouflage that seems likely to recur is the presentation of guesstimated CBA as quantified CBA—which potentially misleads the public by omitting significant information about the uncertainty, judgment, and sensitivity of particular numerical results in a CBA.

Depending on one’s assumptions about the alignment of agency interests with public interests, these effects may be costly or beneficial. But they should be kept in mind when evaluating a given type of CBA in a given context, and they suggest that CBA itself needs to be subject to CBA before being mandated through law. In Part IV, I sketch a third set of effects that CBA policy can have—stimulating innovation and inducing better regulation over time—that differs from both the disciplinary role touted by advocates of CBA law and the camouflaging role illustrated by the Corps’s history.

D. Alternatives to Quantified CBA/FR

CBA is sometimes promoted on the ground that there is no superior alternative.53 Leading proponents of CBA/FR in the United Kingdom, for example, acknowledge problems with CBA/FR and then argue these problems do “not . . . mean that the best course would be to fail altogether to deploy the techniques of economic analysis.”54 (One would hope not!) Yet viable alternatives exist.

In non-financial areas of regulation, agencies use feasibility analysis, which focuses on the technical capacity of private actors to comply with a proposed rule; this procedure pays some attention to costs rather than attempting to quantify the rule’s full range of costs and benefits.55 Another alternative is risk-risk analysis, in which the risk addressed by a rule is compared to risks that can be expected to arise as private actors respond to the rule.56 Another option (sometimes included as a component of CBA) is cost-effectiveness analysis, in which costs of different methods of achieving stipulated or assumed benefits are estimated and compared.57 Yet another, reflected in some important statutes relevant to financial regulation,58 is a flat ban on certain kinds of activities—that is, requirements that agencies enact and enforce mandatory rules regardless of what an agency’s CBA/FR might suggest about those rules’ net benefits.59

1. The “Alternative” of Expert Judgment

But the primary “alternative” to guesstimated CBA/FR is expert judgment, which typically includes at least some elements of conceptual CBA (whether or not expressed in writing) and can be elicited and deployed in a variety of ways. More precisely, however, expert judgment is not an “alternative,” but a necessary component of guesstimated or quantified CBA, as the Office of Management and Budget’s guidance on CBA (OMB Guidance) makes clear.60 When defenders of CBA argue that expert judgment may be—as it often is—flawed, they are also necessarily arguing that CBA is flawed. The question is not, then, “What is the alternative?” Rather, it is, “Is judgment being camouflaged as something it is not?” An honest acceptance of the central role of judgment in policymaking, whether or not decorated with guesstimated CBA, should lower the stakes in the fight over CBA law.

In the context of financial regulation, the judgment of regulatory staff is expert because the appointees of the financial agencies have generally spent their careers in and have developed specialized knowledge of finance, financial institutions, and financial markets.61 They have sharpened their intuitive sense of what kinds of regulations work and why—particularly relative to non-experts, such as generalist judges. Such intuitions can be disciplined and informed in ways other than through formal CBA, such as through discussions with other experts (within or outside an agency); case studies, surveys, and polls; retrospective evaluations; regulatory experiments that are deliberately adopted without specific predictions about how they will turn out; and other forms of assessment that are not part of quantified CBA/FR.62

The experience and expertise of financial regulators does not make them infallible: the 2008 financial crisis proves that regulators with expertise can lack judgment, particularly when the challenges they face are novel, as with shadow banking, over-the-counter derivatives, and (ironically) the complex and unanticipated effects of deregulation.63 More generally, in many domains, experts are no more capable of predicting certain kinds of complex events than non-experts.64 Nevertheless, in the realm of financial regulation, expert judgment has always played a central role in the setting of monetary policy. This brings us to the Taylor Rule.

2. Monetary Policy: A Limiting Example

To set the stage for case studies of rules in Part III, this section recognizes that even CBA/FR’s proponents do not advocate requiring CBA/FR for monetary policy.65 As will be seen, guesstimated CBA/FR of monetary policy would result in conceptual, theoretical, and empirical challenges identical to those that arise in the case studies reviewed in Part III. This fact raises the question of why, precisely, CBA/FR proponents believe a line should be drawn between rules for monetary policy and other financial regulations.

To think through how CBA/FR might in principle be applied to monetary policy, consider the Taylor Rule. That “rule” is a principle of monetary policy that stipulates how much the Federal Reserve (or any central bank) should change nominal interest rates in response to changes in prices, output, or other economic quantities. In particular, the Rule stipulates that for a percent increase in inflation, a central bank should raise interest rates by more than a percentage point.66 First proposed in its specifics by John Taylor in 1993, the Rule sought to reduce uncertainty, limit adaptive inefficiency, and increase credibility by avoiding frequent changes in monetary policy as a result of the exercise of discretion.67 The Federal Reserve, it should be emphasized, has never “promulgated” the Taylor Rule, nor has it adopted the Rule in any formal or public fashion.68 Nevertheless, the Rule does fairly characterize (as a first approximation) the monetary policy of the Federal Reserve for some of the years under Chairman Alan Greenspan.69

Suppose, counterfactually, a future Federal Reserve (or Congress) wanted to “adopt” the Taylor Rule—or any other rule for conducting monetary policy—in a formal fashion. Could the Rule be defended through CBA/FR? Only a few CBA/FR proponents suggest that it could, or should, be defended through CBA.70 The numerous proposed bills in Congress that would extend CBA to the independent agencies have all exempted monetary policy.71

Why is monetary policy exempt? Politics and political power play a role, of course: few politicians want to take on the Federal Reserve (even if a few have done so, particularly during the public outcry over the 2008 crisis).72 History and tradition also play a role: monetary policy in the United States has long been (by consensus) an exercise in discretionary judgment, and it involves balancing multiple goals—full employment, stable prices, and moderate long-term interest rates.73 Any strict rule to set monetary policy according to a full quantified CBA would have to reverse this tradition and implicitly choose a priority scheme for the goals; as a result, the rule would be (to return to politics) highly unlikely to achieve the supermajority support necessary to enact major legislation in the United States.

But policy, too, plays a role here. In a context of high empirical and theoretical uncertainty, multiple competing macroeconomic models have long coexisted to guide the achievement of monetary policy’s goals. However, these models are widely conceded to be contestable,74 and no one model has ever achieved anything close to a consensus among “mainstream” economists. For this reason, presumably, even the most rule-oriented members of the Federal Reserve have never seriously attempted to persuade the Board to tie its own hands by articulating publicly a “rule” that would eliminate the Board’s discretion to set interest rates.

Absent such hand-tying, there is no need to exempt monetary policy from the proposed CBA/FR laws. So why have they been exempted? Presumably because CBA/FR proponents recognize that there may be welfare-enhancing “rules” (in the sense of regularities in the exercise of discretion that might come within the legal definition of “rule” used in the APA)75 that can discipline regulators but cannot be reliably shown to satisfy a cost-benefit test. The idea that a “rule” in the general legal sense of the APA could be valuable without being first validated by quantified CBA/FR prevails across many domains of discretionary decision making: in an attempt to constrain itself, a corporate board of directors may decide to adopt rules about the situations in which it wants officers to present an investment to the board (instead of pursuing the investment on their own), but such self-imposed rules may not be defensible under any kind of quantitative framework. Rules, in other words, can be a part of the way that discretionary judgment is exercised. Rules can have value even if they cannot be supported by evidence showing that their quantifiable benefits exceed their quantifiable costs.

Indeed, CBA/FR’s strongest proponents concede that expert judgment is necessary because CBA/FR can only be as good as the expert judgment that informs it.76 Pro-CBA/FR bills pending in Congress exempt monetary policy, presumably for this reason, and there is no serious call for hard-wiring monetary decisions into legislation or regulation. While there are economists who believe that basing monetary policy on simpler rule-like elements may be a good idea, even they suggest that rule-like monetary policy be adopted as a matter of expert discretion by the Federal Reserve Board and be subject to discretionary exceptions.77

The question remaining, then, is whether discretionary judgment should be confined to monetary policy or whether it should remain available for financial regulation more broadly. Put differently, the question is whether quantified CBA/FR is itself actually an alternative to judgment, or whether it should be viewed as judgment camouflaged by numbers (“judgment in drag,” one might say, or less colorfully, “judgment in disguise”). To answer that question, a detailed analysis of what CBA/FR might look like is needed.

II. a critical assessment of judicial review of cba/fr

As noted at the outset, a movement is afoot to impose CBA/FR on financial regulation. This movement is flowing through a variety of channels. Interest groups and advocacy organizations have been promoting CBA/FR as both policy and law, and regulators themselves have been beefing up their quantitatively trained staffs. But one big force (perhaps the biggest) that is promoting the role of CBA/FR has been judicial activism—aggressive review of agency decisions by courts focused in large part on CBA. After reviewing statutes relevant to CBA/FR, this Part critically assesses recent cases in the D.C. Circuit that have overturned financial regulations in whole or in part because of what some judges have seen as inadequately quantified CBA/FR. This Part concludes with a summary of how this judicial activism has led some of the financial agencies to engage in more CBA/FR, and has amplified legislative efforts to promote CBA/FR through oversight and proposed legislation.

A. Existing CBA/FR Law

Formally, independent agencies78 such as the financial regulators are not subject to explicit CBA/FR law to the same extent as executive agencies, which have been required (by executive order since 1981 and by statute since 1995) to conduct CBA for new rules.79 Vice President George H.W. Bush requested in 1981 that the executive agencies comply with the CBA portions of the executive orders, and some of the financial agencies have at times voluntarily, if incompletely and inconsistently, done so.80 By contrast, in the United Kingdom, the two main financial regulatory agencies are required by statute to conduct quantified CBA/FR, unless in the opinion of the agencies the costs or benefits “cannot reasonably be estimated” or “it is not reasonably practicable to produce an estimate,” in which case the agency must publish its opinion and explain it.81

Three CBA-related statutes cover the independent agencies. The Paperwork Reduction Act (PRA) requires agencies to justify collection of information from the public, to minimize the burden of any information collection process, and to maximize the utility of information gathered.82 The Regulatory Flexibility Act (RFA) requires agencies to assess and consider alternatives to the burden of regulation on small entities.83 The Congressional Review Act (CRA) requires agencies to submit proposed rules—along with any CBA the agencies have conducted—to Congress and the Government Accountability Office (GAO).84 The statute requires the GAO to submit an assessment to Congress of any “major rule,” defined as any rule having an expected impact of $100 million or more.85

As a result of these statutes, independent agencies include some CBA-relevant information in rulemakings, the GAO has been submitting annual reports on CBA for major rules (including rulemakings by independent agencies), and the OMB has collected and reported on the GAO’s reports on an annual basis.86 Analyses under the PRA and the RFA represent only a subset of a full CBA—even of a full conceptual CBA—and the information in these reports is thin—generally indicating whether CBA was conducted, without regard to whether it was conceptual or quantified, extensive or brief, persuasive or perfunctory. Still, the PRA and RFA have generated information used to critique financial rules on CBA-related grounds, and the GAO’s and OMB’s reports have made the complete absence of voluntary CBA in many rulemakings by independent agencies more salient over time. Together, this information has fueled legislative, inter- and intra-agency, and interest group pressure on the financial regulatory agencies to do more on their own to conduct CBA, and has also led to a sharp increase in industry-funded court challenges to agency rulemakings on CBA/FR grounds.

B. A Critical Assessment of Judicial Review of CBA/FR

Despite the fact that CBA/FR is not clearly required of independent agencies, business trade groups have since 2000 invested significant time and resources to persuade court­­s—primarily the D.C. Circuit—to strike down a series of rules under the APA and under statutes that authorize financial regulation. Cited in internal CBA/FR guidance promulgated by the CFTC and the SEC, these decisions have clouded implementation of the Dodd-Frank Act, contributing significantly to the rulemaking delays under that law. These decisions have had an impact on the legislative process, as lawmakers, lobbyists, and the agencies themselves have noticed that rules receive different treatment depending on whether Congress has required the agencies to enact them or has given the agencies discretion and authority to act on their own.

The first in the recent string of judicial interventions was Chamber of Commerce v. SEC.87 In that decision, the D.C. Circuit held that the SEC failed to comply with the Investment Company Act (ICA). The ICA requires the SEC to “consider . . . whether [regulatory] action will promote efficiency, competition, and capital formation,”88 a requirement added to the SEC’s statutory mandates in 1996.89 As a result, according to the court, the SEC had also violated the APA.90 The rules in question—discussed in Part III.B—made exemptions under other rules conditional on mutual funds increasing their boards’ independence.

The specific CBA/FR-related failings to which the court pointed were two small parts of the SEC’s regulatory analysis. The first was that the SEC declined to quantify costs of requiring more independent directors because it did not know how funds would respond to the rule.91 This, the court replied, was no excuse, saying that the SEC could have determined “the range within which a fund’s cost of compliance [would] fall, depending on how it responds to the condition.”92 Presumably the court had in mind that the SEC could quantify costs of each possible response and guesstimate a range based on assumptions about how many funds would choose each option.

The second failing was similar, relating to a requirement that fund boards have an independent chair. There, the SEC declined to quantify costs of the newly independent chairs’ hiring staff because staffing would be discretionary and the SEC had no basis for knowing how many chairs would hire staff (or how many staff each chair would hire). Again, the court held the SEC needed to guesstimate this subset of costs by estimating the costs for an individual fund, an exercise that the court asserted (without further explanation) would be “pertinent” to an “assessment” of the requirement.93 But the only way that an individual fund cost estimate would be “pertinent” is if the SEC implicitly or explicitly made further assumptions about how many funds would incur those costs—even though the SEC explicitly noted that it had no reliable basis on which to build the assumptions, and the court offered no reason to doubt the SEC’s claim.94 The court’s analysis under the APA was nonexistent: because the SEC had not followed the ICA, the court reasoned, it had violated the APA.95

In sum, the court interpreted the requirement that the SEC “consider” a rule’s effects on “efficiency” to imply a very specific CBA/FR mandate—calling on the SEC to guesstimate the range of one of a rule’s costs, rather than merely identifying the type of cost imposed. The court’s interpretation of the ICA was based on no prior court decision96 and no legislative history. Nor is it implicit in the ICA’s words, as “efficiency” is frequently used as a qualitative and not exclusively quantitative concept.97 Nowhere did the court cite (much less discuss) Supreme Court precedent under the APA that had emphasized that courts should be highly deferential in reviewing an agency’s judgment under the “arbitrary and capricious” standard.98 Nor did it address precedents more generally admonishing courts to be mindful of the “complex nature of economic analysis” in deferring to agencies.99

Finally, the court never explained how a crude guesstimate of one conditional component of possible costs of a rule could meaningfully inform the public about the “efficiency” of the rule when the SEC had not quantified the benefits of the rule—and when the court did not suggest that the SEC try to do so, whether it could, or how it could if it tried. In other words, the court read general language in the ICA as if it required the SEC to comply with the Executive Orders requiring CBA/FR “to the extent feasible,”100 and then added an interpretive gloss on OMB Guidance that has little apparent virtue in improving public understanding of the rule. Whatever the merits of the SEC’s mutual fund rules—and there are reasons (noted in Part III.B) to suggest that the rules might not be a good idea on balance—the merits of the court’s decision evaluating the SEC’s rulemaking under the ICA and the APA are hardly compelling and do not appear to reflect any meaningful deference to SEC judgment on how to conduct CBA/FR.

Yet this decision was only the first of a rash of judicial interventions into the financial regulatory process, each opinion growing steadily less deferential, culminating in the 2011 case Business Roundtable v. SEC.101 In the seven years after Chamber of Commerce, the D.C. Circuit handed down six more similar decisions, striking down a range of SEC actions (representing one in seven of the SEC’s major rules over that period).102 The D.C. Circuit has struck down a rule requiring registration of hedge fund advisors under the Investment Advisors Act,103 a rule exempting broker-dealers from registration under that Act,104 an order affirming expulsion of an NASD-member firm,105 and a rule treating a new class of securities market-linked annuities as securities.106 The court also struck down the same mutual fund governance rules from Chamber of Commerce a second time: the SEC, with perhaps tactless speed, patched the guesstimated CBA/FR holes in its rulemaking analysis, only to have its rule struck down on new grounds.107 Since Chamber of Commerce, only one decision, National Ass’n of Manufacturers v. SEC, has upheld an SEC regulation.108 Another upheld a CFTC regulation,109 and another upheld a decision of the Office of Thrift Supervision against CBA/FR-related challenges.110

Three facts are worth noting about these decisions. First, a business or trade group initiated and funded each of the cases; so far, consumer and investor lobbies have been sitting out these court battles.111 One-sided use of litigation as a lobbying tactic is not typically a stable feature of enduring battles between interest groups over important regulations. Second, not all of the decisions strike down new regulations—one struck down a new exemption from a regulation, and one overturned an enforcement action. Together, these two facts should give pause to political entrepreneurs who seek to use CBA/FR as a way to attack regulation generally; these observations suggest that CBA/FR law can slow or stop deregulation as easily as it can slow or stop new regulation, particularly if consumer or investor advocates develop and fund their own CBA/FR litigation agendas. Third, each regulatory action (except the action involved in National Ass’n of Manufacturers) was taken pursuant to the SEC’s general statutory authority to use discretion to adopt regulations in support of the securities laws—and not pursuant to a mandate from Congress to do so. That the District Court in National Ass’n of Manufacturers112 distinguished the string of anti-SEC precedents on the ground that the Dodd-Frank Act mandated the rule in question reinforces this take-away. Under the current CBA/FR legal regime, regulatory agencies are well advised to seek statutory language that requires them to adopt rules or to enforce rule-like legal requirements via enforcement proceedings that are generally exempt from judicial review under the APA;113 it is inadvisable to seek language that promotes SEC discretion and authority in rulemaking based on the agency’s expertise. Judicial efforts to promote CBA/FR, in other words, have given expert agencies an incentive to ask an inexpert Congress to tie their hands with inflexible statutory commands.

The most notorious114 decision in this line of cases was Business Roundtable, which struck down an SEC rule requiring public companies to include in their annual proxy statements, under limited circumstances,115 information about (and the power to vote for) board nominees nominated by large shareholders rather than solely those nominated by the incumbent board.116 Despite the SEC’s having debated the issue for over a decade, having developed an extensive public record before adopting the rule, and having adopted the rule under the explicit authority and implicit direction of Congress in section 971 of the Dodd-Frank Act, a panel of the D.C. Circuit struck the rule down as “arbitrary and capricious.”117 According to the court, the twenty-five single-spaced pages devoted to cost-benefit and related analyses in the adopting release was inadequate under the APA and “failed . . . adequately to assess the economic effects of a new rule.”118 The D.C. Circuit presented no evidence that there is any available scientific technique for the SEC to “assess the economic effects” of the rule along the lines that the court seemed to think legally required—as when the court held that the SEC “relied upon insufficient empirical data when it concluded that Rule 14a-11 [would] improve board performance and increase shareholder value by facilitating the election of dissident shareholder nominees,”119 or when it held that the SEC had “arbitrarily ignored the effect of the final rule” because the SEC “does not address whether and to what extent Rule 14a-11 will take the place of traditional proxy contests.”120

Instead, as in Chamber of Commerce, the U.S. court with “[s]tatus [s]econd [o]nly to [the] Supreme Court”121 ignored precedents establishing a “deferential” standard of review under the APA and substituted its own judgment for that of the SEC in evaluating the existing research relevant to proxy contests. In Business Roundtable, the D.C. Circuit went so far as to characterize (without explanation) a peer-reviewed article published in the Journal of Financial Economics as “relatively unpersuasive.”122 Even the Chamber of Commerce decision had not gone so far, for while that decision invented an obligation for the SEC to use guesstimated CBA/FR on the cost side of its rulemaking, it also held that the SEC need only “determine as best it can the economic implications” of a rule;123 moreover, Chamber of Commerce nowhere suggested the SEC had to remain inert whenever quantified CBA/FR was simply unavailable. Hypocritically, it was Judge Ginsburg who penned the Business Roundtable decision, just two years after he joined the decision in Stilwell, where the same court held that the APA “imposes no general obligation on agencies to produce empirical evidence.”124

In sum, the D.C. Circuit’s new interpretations of the APA and statutes authorizing financial regulation have permitted panels to overturn regulatory changes on the ground that a court would conduct its guesstimated CBA differently than an agency would. Since guesstimated CBA/FR is unreliable and imprecise, no matter who conducts it, courts have no legitimate role to second-guess the agencies—even if the agencies are arbitrary in how they go about the guesstimated CBA/FR. Indeed, the state of CBA/FR is such that one can reasonably argue that all guesstimated CBA/FR of major financial regulations inevitably contains multiple arbitrary assumptions and judgments simply to allow for rough guesstimates to be made. Worse, the judges reviewing these guesstimates are political appointees tenured for life, and so—while often selected for political reasons—are immune from conventional forces of political accountability; nonetheless, they have been frequently partisan in their approach to CBA. Because the D.C. Circuit is roughly evenly split between Republican and Democratic appointees,125 the partisan-driven outcomes in CBA/FR cases are unpredictable and depend on a factor (which judges are chosen for a given case) that has nothing to do with the APA or any other law. The normative implications of this state of affairs are taken up in Part IV.

C. Congressional Oversight, Regulatory Initiatives, and Proposed Legislation

Elements of the legislative branch, as well as the financial agencies’ own initiatives, have reinforced the effect of judicial review of existing CBA-related mandates on the financial agencies’ organic statutes. While Congress has not mandated CBA for independent agencies, members of Congress, in coordination with minority commissioners of the CFTC and the SEC, have pressured the agencies to engage in CBA, both by attempting to pass legislation (discussed below) and with soft power, through hearings, information requests, and public criticism. In 1998, the GAO released a critique of current law for failing to improve CBA in agency rulemakings.126 As discussed in Part III.B, Fidelity Management in 2004 persuaded Congress to require the SEC to justify proposed rules by preparing a report on their potential benefits.127 In 2007, the House held hearings on the Sarbanes-Oxley Act, in which one witness (inaccurately) critiqued the SEC’s CBA/FR,128 a criticism echoed by members of Congress129 and, more recently, by Republican SEC Commissioner Daniel M. Gallagher.130

These pressures, along with the court decisions discussed above, have led financial agencies to conduct and publish more CBA/FR in recent years. OMB reports show that this increase in the use of CBA/FR began in the early 2000s. In September 2010, the CFTC’s General Counsel and Acting Chief Economist distributed a memo to the CFTC’s rulemaking teams noting that, while the CFTC’s authorizing statute does not require quantified CBA/FR, it does require the CFTC to consider costs and benefits, and that recent court decisions had been expanding the demands of CBA/FR law under the APA.131 As a result, the memo directed staff to provide summary CBA/FR in proposed rulemakings and to address conceptual CBA/FR in adopting releases.132

Despite these efforts, congressional pressures have only increased, potentially stimulated by the financial industry lobbies seeking to influence rulemaking under the Dodd-Frank Act. In 2013, Senator Mike Crapo (Republican of Idaho) pressed the heads of the major financial agencies to commit to “act on GAO’s recommendation to incorporate OMB’s guidance on [CBA] into your proposed and final rules [and] interpretive guidance.”133 Shortly thereafter, ten Senate Banking Committee members requested financial agency inspector generals to report on CBA under the Dodd-Frank Act, “in response to concerns raised by Commissioners at both the CFTC and the SEC” regarding economic analysis at the agencies.134 Also in 2011, Congress amended the Dodd-Frank Act to require the GAO to analyze the impact of regulations on the marketplace,135 and in November 2011, the GAO released a report on the financial agencies’ Dodd-Frank Act rulemakings, finding that “[a]lthough most of the federal financial regulators told us that they tried to follow [OMB guidance] in principle or spirit, their policies and procedures did not fully reflect OMB guidance on regulatory analysis.”136 While noting that “for 7 of . . . 10 regulations we reviewed, the agencies generally assessed benefits and costs of the alternative chosen,”137 the GAO was particularly critical of the financial agencies for not conducting quantifiedCBA/FR: “[O]ne of the seven benefit-cost analyses monetized the costs of the regulation, but the analysis did not monetize the benefits. None of the other analyses monetized either the benefits or costs, identified the type and timing of them, or expressed them in constant dollars.”138 Trade groups and political entrepreneurs have picked up these criticisms,139 as have members of Congress.140

In March 2012, SEC staff distributed its own internal CBA/FR guidance. The guidance cited “[r]ecent court decisions, reports of the [GAO] and the SEC’s . . . [OIG], and Congressional inquiries” that had “raised questions about . . . the [SEC’s] economic analysis in its rulemaking.”141 The SEC guidance noted “[n]o statute expressly requires” the SEC to “conduct a formal” CBA but that “SEC chairmen ha[d] informed Congress since at least the early 1980s—and as rulemaking releases since that time reflect—the [SEC] considers potential costs and benefits as a matter of good regulatory practice whenever it adopts rules.”142 The SEC guidance went on to set out “[s]ubstantive requirements” for CBA/FR, drawing on the CBA Executive Orders and the OMB Guidance.143 Rulemaking staff were directed to work with economists on the SEC’s staff to analyze which costs and benefits a rule might create, to quantify those that could be quantified, and to explain why others could not feasibly be quantified.

The bluntest form of congressional pressure has taken the form of bills that would mandate CBA/FR across the board. In June 2013, three Senators reintroduced the Independent Agency Regulatory Analysis Act.144 That bill would permit the President to order all independent agencies, including all of the financial regulatory agencies, to (among other things) conduct a CBA of any new “rule and, recognizing some costs and benefits are difficult to quantify, propose or adopt a rule only upon a reasoned determination that the benefits of the rule justify its costs.”145 The bill incorporates the definition of “rule” from the APA and excepts only rules of the Federal Reserve “relating to monetary policy.”146

In addition, for any “economically significant rule” (ESR), an independent agency could be required to give OIRA and to publicly disclose (1) “an assessment, including the underlying analysis, of benefits . . . [and] costs . . . anticipated . . . with, to the extent feasible, a quantification of those benefits . . . [and] costs,” (2) a similar assessment of all “potentially effective and reasonably feasible alternatives to the rule, identified by the agencies or the public,” and (3) a statement of why the rule is superior to alternatives.147 For this purpose, the bill defines an ESR as a rule with an annual effect on the economy of $100 million or more. Independent agencies could be required to submit any ESR for a ninety-day OIRA review of whether the rule “has complied” with these requirements, with the OIRA review also to be part of the published record for the rule. Independent agencies would also be required to publish a finding that the rule did comply with the bill, with an explanation of that finding, or “if applicable, an explanation why the independent regulatory agency did not comply.”148

The bill states that “compliance” by an independent agency with the bill is not subject to judicial review. However, it also states that in any court challenge to an independent agency’s rule under other laws, such as the APA, all material produced by the independent agency and OIRA under this bill would be “part of the whole record” for the court to review.149 As discussed more in Part IV, mandating an open interagency process formally not subject to judicial review might seem innocuous: how could it impede rulemaking for an independent agency to simply get the input of another agency? But this naïve reading misses the fact that any public interagency process will create a larger record that will be used by litigators to attack particular agency judgments as arbitrary and capricious under the APA: any disagreement between the agencies, for example, will provide grist for the litigation mill. The cases reviewed in this Part show how aggressive some D.C. Circuit panels have been in overturning agency actions on CBA grounds, particularly when an agency’s commissioners have been divided over judgments needed for any regulatory change. Trebling the number of pages or components of a CBA available for judicial second-guessing, and adding the possibility of interagency disagreement to the mix, will almost certainly incite more judicial interventions.150 Before we can assess whether such interventions might be net beneficial, however, we need to consider CBA/FR itself. Could it offer precise, reliable estimates of the costs and benefits of financial regulation?

III. how might cba of financial regulation work?

In this Part, I outline how the kind of quantified CBA/FR envisioned by its proponents might work in practice. The goals of this Part are to illuminate what we might expect of CBA/FR policy, to advance the substantive research project of developing CBA/FR, and to provide a better empirical basis for evaluating CBA/FR law in Part IV.

To accomplish these goals, I outline the CBA/FR that was performed for four specific rules: (1) SEC regulations under Sarbanes-Oxley Act section 404 (SOX 404); (2) the SEC’s 2002 mutual fund governance proposals; (3) Basel III’s enhanced capital requirements for banks; and (4) the Volcker Rule. These analyses are followed by a review of two rules that have been subject to CBA/FR and have been held up as the “gold standard” by CBA advocates: (5) the SEC’s cross-border swap rules and (6) the UK/FSA’s mortgage market rules.

The first, third, and fourth case studies represent the kind of significant rulemakings that CBA/FR proponents agree should be the focus of CBA/FR,151 and because they are clearly “economically significant rules,” they would trigger the highest degree of interagency review under the CBA Executive Orders and OMB Guidance if the independent agencies were brought under those process requirements. The second case study focuses on rules that led to the D.C. Circuit decisions reviewed above and stimulated the SEC’s Chief Economist to publish two extensive CBA/FR-related memos that provide one of the better (if imperfect) examples of what CBA/FR as conducted by a financial agency could look like.

In each case the analysis draws on the best research by economists, finance scholars, and legal scholars, all using the kinds of methods that are closest to the idealized vision of quantified CBA/FR that its proponents are asking financial agencies to pursue.152 This review illustrates that guesstimated CBA/FR of each of the rules reviewed would (or did) require the same kinds of macroeconomic or political models used to set monetary policy, or entailed causal inferences that are unreliable under standard regulatory conditions, or both.

These case studies were also chosen to reflect representative types of major financial regulations. They focus on regulations promulgated by a variety of financial regulators: the SEC, the Federal Reserve, the Office of the Comptroller of the Currency (OCC), the Federal Deposit Insurance Corporation (FDIC), the CFTC, and the UK’s Financial Services Authority (FSA). The regulations employ a range of the kinds of regulatory instruments that are commonly analyzed or proposed for the financial markets: disclosure, governance regulations, capital requirements, activity restrictions, and transactional restrictions and process requirements. The regulations address a variety of market failures: fraud, asymmetric information more broadly, conflicts of interest, externalities (systemic crises arising from the effects of transactions on third parties), and the absence of competition. But they also give rise to a typical array of regulatory costs: compliance costs, constraints on potentially optimal private governance arrangements, smaller or less complete markets, and prohibition of potentially optimal transactions, possibly reducing economic activity and surplus. And, finally, most of the regulations were adopted following at least some conceptual CBA by the relevant agencies, and two were adopted after at least some efforts at quantification of the relevant costs and benefits.

A. Case Study #1: Control Disclosures for Public Companies153

The first case study is of the Sarbanes-Oxley Act (SOX). SOX was Congress’s response to the widespread fraud at Enron, Tyco, Worldcom, and other corporations. The core of SOX consisted of two parts154: (1) the creation of a quasi-public regulatory body to oversee public company audit firms—the Public Company Accounting Oversight Board (PCAOB)—and (2) the requirement of new disclosures by public companies about “control systems.”155 Among other things, the case study illustrates the way that a common goal of financial regulation—the reduction of fraud—implicates important externalities, as well as non-market goods (such as psychological effects of fraud), which cannot be reliably reduced to precise monetary estimates, given current research technologies.

1. The SEC’s CBA of Rules Implementing SOX 404

SOX required the SEC to enact regulations to carry out the goals of SOX 404.156 The SEC did this in August 2003, a year after SOX’s passage.157 In its adopting release, the SEC included a 1400-word CBA, which, as noted above, was not a legal requirement for the SEC.158 The release contained a separate 500-word analysis of the rule’s effects on efficiency, competition, and capital formation,159 and a longer analysis under the PRA and RFA.160 In its CBA, the SEC provided a qualitative listing but no quantification of the rule’s benefits. The benefits identified were: (1) generally to (a) enhance the quality of public company reporting and (b) increase investor confidence, and (2) specifically to (a) improve disclosure about management’s responsibility for financial statements and controls and how management discharges that responsibility, (b) encourage companies to devote adequate resources and attention to controls, (c) help companies detect fraud earlier, and (d) deter fraud or minimize its effects. The bottom-line benefit, then, was to reduce fraud.161

The SEC also provided a qualitative listing of the rule’s direct costs (administrative burdens and fees to attorneys and auditors). The SEC noted that companies were already required to have a control system under the Foreign Corrupt Practices Act (FCPA) and that many issuers were already voluntarily providing the required disclosures, raising conceptual issues (discussed below) for what baseline and set of effects to assume in any CBA/FR of the rule—issues that the SEC did not explicitly address. The SEC provided a partial quantification of the costs of its rules under SOX 404. That estimate focused exclusively on the requirements of subsection (a) of SOX 404, disclosures by management, which the SEC estimated would cost covered companies an average of $91,000 per year.162 The SEC explicitly noted it had no information that would allow it to quantify the costs created by subsection (b) of SOX 404, the auditor attestation requirements, which it acknowledged could be large.163

Of note for assessing CBA/FR’s effects on public understanding, the SEC has been strongly criticized for the CBA/FR in its release—but only for the part of its CBA/FR that provided a quantitative estimate of costs, which one commentator has claimed is “off by a factor of over 48.”164 However, this critique of the SEC’s CBA/FR is demonstrably mistaken. The SEC’s estimate was solely for SOX 404(a), while the FEI/FERF estimate was for both SOX 404(a) and 404(b).165 For several reasons, auditor attestation costs can be expected to exceed internal costs by a multiple (as in fact has been the case).166 The SEC explicitly acknowledged this gap in its cost estimate,167 but the criticisms of the SEC ever since—including by SEC Commissioner Gallagher himself—have mistakenly claimed the estimate was for SOX 404 as an entirety.168 CBA/FR advocates, in other words, have publicly and repeatedly criticized the SEC for underestimating the cost of apples and oranges when the SEC’s estimate was for the cost of apples alone. The spectacle may undermine an observer’s faith in the value of public discourse stimulated by CBA/FR.

A better critique of the SEC’s CBA/FR of SOX 404 is that it failed as conceptual CBA/FR for not identifying indirect costs of the rule. Indirect costs include potential reductions in risk-taking, dilution in strategic focus, and the opportunity costs of devoting excessive management time to compliance and working through the initial control attestation process with outside auditors, internal audit staff, and members of companies’ audit committees (which SOX required to be wholly independent for the first time).169 While quantifying these costs would have been nearly impossible for the SEC at the time (as discussed below), the SEC could have pointed to the possibility of these costs in its rulemaking.

Conversely, the SEC in 2006 did not identify (much less quantify) increased fraud as a possible cost of the deferral of SOX 404 requirements for small and newly public companies, nor did it identify (much less quantify) increased fraud as a possible side effect (cost) of the relaxation of the SOX 404 requirements in 2007.170 While these efforts were deregulatory in nature, they would be just as subject to CBA under Executive Order 12,866 for an executive agency as would the imposition of new regulations.171 The fact that the more prominent CBA/FR proponents172 do not mention these gaps in the SEC’s deregulatory rulemaking process under SOX tends to undermine their general depictions of CBA/FR as a politically neutral procedure for improving regulation generally.

2. An Overview of CBA/FR of SOX 404

Now that ten years have passed since its adoption, how might SOX 404 fare under a CBA/FR? Quantifying the costs and benefits of the rule would require multiple research tasks. These include (1) establishing better estimates of the incidence and direct costs of fraud,173 (2) securing consensus on how to treat “transfers” for purposes of analyzing fraud, (3) generating new models and data on fraud’s externalities, (4) creating better instruments for estimating the rule’s causal effects, (5) developing better models and data on the chilling effects that the rule could have on legitimate activity, and (6) promoting better understanding of how compliance costs vary across firms and over time. Each task will be difficult and likely require a separate stream of research before any plausible quantified estimate of the costs and benefits of a rule under SOX 404 could be developed.

3. Estimating the Incidence of Fraud and Its Direct Costs

The first task is to develop better methods of measuring the incidence of corporate fraud and its direct costs. This task is a prerequisite even to a rough estimate of the effects of regulations aimed at reducing fraud, such as SOX 404. Yet, with few exceptions, research on fraud to date has only attempted to establish relationships between fraud and its correlates, and it does not present evidence of how strong these relationships are, or what the overall incidence of corporate fraud is.174

One difficulty confronting such studies is that all concerned have incentives to hide fraud.175 Partial observability presents challenges to empirical modeling,176 but until recently, few researchers used models adapted to those challenges. Such models study both fraud incidence and detection together, exploiting partial overlap in indicators of fraud incidence and detection to draw better inferences about correlates of fraud overall from detected frauds.177

Building on this work, one study exploits the collapse of Arthur Andersen to estimate an incidence of fraud among public companies at approximately fifteen percent.178 The study also estimates that fraud generates direct losses of between twenty-two percent and forty percent of enterprise value, implying a lower bound on hidden fraud of three percent of enterprise value (0.15 × 0.22 = 0.03), or losses of over $500 billion. This study is a promising start to estimating how much fraud exists and how costly it is. But it is, as yet, unpublished and relatively isolated,179 and it needs more scrutiny before it can provide a reliable rulemaking foundation. Future research could use more comprehensive measures of fraud, including fraud outside the scope of audits that nevertheless might be revealed by a stronger control system, such as insider trading and self-dealing (as at Enron), fraudulently obtained compensation (as at Tyco), frauds involving third parties (as at Worldcom), or technically GAAP-compliant but deceptive accounting choices (as at Lehman).

4. How Should Transfers Be Treated?

An open conceptual issue in estimating the costs of fraud is how to treat transfers accomplished through fraud—in other words, whether to count the utility of a fraudster in estimating welfare effects of fraud.180 Data on crime generally suggests the issue could have a significant effect on a CBA/FR of SOX 404.181 Canonical economic theory would count the loss as zero, as would the OMB Guidance on CBA,182 but it seems implausible as a political, policy, or legal matter for the SEC to ignore for purposes of CBA/FR of SOX 404 the losses of Enron’s defrauded investors on the ground that they were mere transfers to Ken Lay and Andrew Fastow. OMB Guidance suggests including transfers in an analysis of “distributional effects” distinct from quantified costs and benefits,183 but that does not answer the question of how an agency should weigh the transfers in its overall CBA.184

5. Measuring the Externalities and Psychological Costs of Fraud

If more work is needed to model the incidence of and transfers caused by fraud, no researchers have systematically attempted to study and measure the social costs of corporate fraud. Without estimates of such costs, an assessment of rules that reduce fraud, such as SOX 404, would have to remain qualitative. Research is needed both on externalities185and psychological costs. On externalities, consider these categories186: (a) fraud increases the cost of capital for all firms;187 (b) fraud results in the misallocation of resources;188 (c) fraud destroys value through (costly) acquisitions and bankruptcy;189 (d) fraud induces precautionary costs;190 and (e) fraud imposes costs on non-investor third parties.191 Consider the Madoff scandal, which imposed significant direct losses on over 15,000 individual investors, each of whom presumably had an average of two dependents or heirs, and many of whom were co-investors and borrowers with yet others, or makers of charitable donations to non-profits.192 To date, the liquidation of the Madoff entities has generated over $700 million in expenses—all a pure loss to investors, over and above the amounts stolen by Madoff himself.193

As a broader example, consider how fraudulent home loans (whether due to borrower fraud, lender fraud, or both) had ripple effects in the last financial bubble, partly generated through leverage and intermediation, so the one fraudulent loan would affect not only the immediate parties to the loan but also securitization lenders, sponsors, and related parties; collateralized debt obligation investors, sponsors, and related parties; structured investment vehicle investors, sponsors, and related parties; investors in the banks that sponsored those vehicles; borrower-customers of those banks, whose capital constraints and heightened risk-aversion following the crisis caused a withdrawal or increase in the cost of credit; employees and customers of businesses that failed as a result of the capital constraints generated by the banks’ losses; family members of those employees; and so on.

Psychological effects (fear, distrust, stress) can result in tangible consequences, including drug addiction, job loss, reduced income, health effects, and even suicide. In the context of securities fraud, elevated levels of post-traumatic stress disorder and related behavioral effects have been found among Madoff’s victims.194

The take-away from these thought experiments—and they remain just that—is that the external costs of fraud are likely to exceed, perhaps by a large amount, direct transfers from victim to fraudster. As a result, the quantified benefit of SOX 404 is likely to be found not in estimating direct losses prevented, but in increasing those losses by a multiple to reflect its externalities. How do we translate anecdotal examples into more general methods for estimating the full effects of fraud on society as a whole?

In the context of SOX, only one unpublished paper attempts to estimate fraud’s social costs.195 The authors treat widespread revelations of fraud as a “shock” to the equity premium and estimate its social effects with a macroeconomic model. For this purpose—and this is worth stressing in light of the discussion of the Taylor Rule in Part I.D above—they adapt a model used by the Treasury Department and the Federal Reserve to set monetary policy.196 They first guesstimate that 25% to 100% of the market decline from March to July 2002 was caused by the scandals of Enron, Tyco, Worldcom, and other companies.197 They then rely on the U.S./Fed. model to estimate that investment would fall 0.8% per year in response to a 20% decline in the stock market, guesstimating first-year impacts ranging from nineteen to fifty-seven billion dollars.198 These projections underestimate costs if the impact of the frauds lasts longer, and could over- or underestimate costs if the economy’s response to fraud-driven equity shocks differs from responses to other kinds of shocks, or if the assumptions of the U.S./Fed. model are varied.199

Finally, research on fraud’s social costs could draw on research on crime generally, which uses several families of methods200: (1) estimating hedonic models in which variation in prices affected by crime is used to infer social costs;201 (2) surveying willingness-to-pay for a reduction in crime;202 (3) aggregating estimates of each direct and indirect effect;203 and (4) relating responses to surveys of crime victims to respondent wealth or income and inferring a “shadow price” for the effects of crime.204 Each method has limitations205: guesstimates based on willingness-to-pay surveys have been stringently criticized as too subjective and internally inconsistent to be reliable for CBA purposes,206 and to date these methods have not been undertaken in the context of fraud.

6. Estimating Causal Effects of SOX 404

With a better framework for estimating the incidence and costs of fraud in hand, researchers could then better estimate the benefits of regulatory changes such as SOX 404. Where a regulation is an innovation, regulators are not in a position to “study” its causal effects at all, but must forecast those effects. For SOX 404, this type of prediction would have been impossible; indeed, few observers (even hostile commentators, who had incentives to exaggerate) anticipated the full extent of the direct costs that SOX 404 would initially generate.

Ex post or retrospective studies, coupled with regulations that sunset absent re-adoption based on the result of the ex post studies, are more promising, and would be better able to enlist academic research in the service of better financial regulation. To date, however, most retrospective studies of SOX have not used research designs allowing reliable causal inferences about its effects. Instead, most researchers have used before-and-after comparisons that fail to control for contemporaneous changes in the objects of study.207 Better are a handful of difference-in-difference studies, such as those used to study some of the effects of SOX.208 In such studies, researchers match, as best they can, the companies affected by a regulation with unaffected companies and compare the before-and-after effects. But even those studies are commonly misleading in the kinds of rich, interdependent environments that characterize the financial markets. Long-term trends may manifest differently in the treated and nominal control group, and common factors omitted from the matching criteria that affect events in the nominal control sample may differentially affect the nominally “treated” sample, creating a spurious impression that the regulation had effects it did not actually have.209

Better for identifying causal effects ex post are discontinuity designs, which look at the before-and-after effects of a regulation on firms just above a threshold triggering compliance and compare them with changes at firms just below the threshold.210 However, the findings of such studies rarely generalize beyond firms “near” the discontinuity, making them of limited use in CBA/FR.211 This point is illustrated by Figure 1, which depicts how one of the best studies of SOX 404, by Peter Iliev, used such a discontinuity design.212 While that study provided convincing evidence on causality, it provides very limited information about SOX’s overall effects, because of how different the firms near the discontinuity are from the firms most likely to generate significant costs and benefits.

Figure 1.

limits on external validity of single best sox study to date213

Perhaps the best of feasible ex post studies are time-series designs studying multiple events, which were used by a small number of studies to analyze SOX 404. Leuz and coauthors studied differences-in-differences among covered and exempt groups of companies over several events in the phase-in of the rule, including extensions by the SEC of exemptions for small firms, and Arping and Sautner studied the staged phase-in for foreign firms cross-listed in the United States.214 Neither study attempted comprehensive measurement of changes in fraud or direct and indirect costs at covered firms, but in principle these types of studies provide the best path towards a possible retrospective CBA/FR of SOX 404.

7. What Baseline and Set of Counterfactuals Should Be Used?

Even if a research design could produce reliable inferences about the effects of financial regulation, it is unclear how (if at all) to modify the results of such a study to reflect the context in which the rule was adopted. As mentioned above, the SEC’s CBA/FR of SOX 404 noted two facts about the rule’s context: (1) covered companies were already subject to the FCPA, which requires companies to have effective control systems, and (2) many companies already voluntarily made disclosures similar to ones required by the rule. These facts raise several open questions about the baselines and counterfactuals to be used in assessing the rule.

First, what baseline should be used to assess the effects of SOX? One possibility is to assume a baseline of full compliance with prior law. Another is to use a realistic baseline of average actual compliance, in which case both costs and benefits would likely be higher (reflecting the gap between full and average compliance, on the reasonable assumption that effects of new enforcement pressures from SOX would have a diminishing effect as compliance increases). A third possibility is to try to estimate levels of baseline compliance that vary with observable firm characteristics. Nothing in the SEC’s governing statutes or other relevant law resolves which baseline to use, but the answer would likely have a significant effect on any quantified CBA/FR of the rule.215

Second, how should analysts treat indirect behavioral effects of eliciting information for purposes of CBA/FR? Suppose, for example, that disclosure reduces risk-taking (as SOX 404 may have done) not because it distracts management but simply because it prevents managers from hiding behind information asymmetries to deflect blame from losses caused by risks they caused the firm to take. Assume that in a world of symmetric information, those risks would not have been taken, but might have generated expected net gains for a firm (perhaps due to differences in risk aversion between managers and diversified shareholders). Should the lost gains due to this reduction in risk-taking be counted? Asymmetric information is treated as a market failure in conventional economics and in the OMB Guidance on CBA. Does that imply that “costs” (such as reduced risk-taking) causally attributed to elimination of some (but not all) information asymmetries should not be counted in CBA/FR? Such a question arises for all disclosure regulations, which anticipate and rely on private responses to the disclosure.216

8. How Do Compliance Costs Vary Across Firms and over Time?

Better methods are also needed for estimating costs, even direct costs. Affected companies and their agents (who know the most about the likely direct costs of a rule) have incentives to exaggerate costs in public comments.217 These exaggerations are evident from the strong contrasts between the FEI/FERF survey results on SOX 404218 and the findings on direct costs from surveys by the SEC, the GAO, and CAQ, a firm catering to the audit industry.219 Compliance costs also vary across firms.220 The SEC’s own studies of the effects of SOX 404221 contain information on some relevant differences, but future CBA/FR could usefully build such differences into better models of direct compliance costs, rather than relying on rationally biased inputs from private actors.

9. Modeling and Measuring Chilling Effects of Financial Regulation

Although direct costs of SOX 404 were most salient to firms, because these costs were borne directly by firms and paid out of their treasuries, indirect costs of SOX 404 may have been larger. They are likely to remain high and may increase rather than diminish over time, as direct costs typically do. Indirect costs include those flowing from changes in risk-taking and investment, which can plausibly dwarf direct costs in magnitude. SOX 404 is said to have caused changes in the risk of personal liability facing managers and directors and in the risk of reputational harms and opportunity costs created by litigation.222 If true, difficult-to-explain and legitimate business risks may be foregone, and firms may decline to go public or otherwise avoid the burdens of the law, with resulting social costs. However, the challenges of estimating indirect costs are also larger than for direct costs. Causal inference for indirect costs is just as difficult as for a regulation’s benefits, requiring quasi-experimental research designs that will only be imperfect, even after the fact. Powerful empirical proxies for risk-taking, investment, and capital costs remain elusive and contested.

10. Summary and Illustrative Integrated Assessment Model

The previous sections have described the kind of CBA/FR of SOX 404 that could (in theory) be done today, from the distinctly advantaged after-the-fact perspective of ten years after the rule was adopted. The bottom line is that no one could hope to conduct a precise and compelling quantified CBA/FR of such a rule now or in the near future. The one component of CBA/FR that could be quantified—direct costs—has generated estimates that vary by an order of magnitude.223 Other, larger components, including benefits from reduced fraud and indirect costs from effects on risk-taking, investment, and management, all remain unquantifiable.

To produce quantified CBA/FR, the SEC would need an “integrative assessment model” (IAM) similar to those used in estimating the social cost of carbon in climate change analysis.224 An IAM would have to combine a sub-model of fraud incidence, a sub-model of the costs of fraud, including transfers and externalities (possibly consisting of a macroeconomic model), and a predictive empirical sub-model for how SOX 404 would affect the incidence of fraud. Indirect costs would have to be estimated in yet another sub-model.

To illustrate what an IAM might look like, consider the following: beginning with the formula for the present value of a perpetuity,225 an annual per-firm direct cost stream for SOX 404 ranging from $300,000 to $2 million per year226 would range from $10 to $67 million (at a three percent discount rate) or from $4 to $29 million (at a seven percent discount rate).227 As of 2003, there were roughly 4,400 U.S. public companies covered by SOX 404,228 producing a present value of direct costs ranging from $19 to $293 billion.

How would this compare to a possible range of benefits for SOX 404? Suppose fraud incidence was—as estimated by Dyck et al.229—three percent of market capitalization, on average, but could range from 50% to 200% of that estimate. These assumptions produce direct fraud costs ranging from $140 to $700 billion. Suppose SOX 404 permanently reduced annual fraud risk by an amount ranging from 1% to 10%.230 When applied to our direct fraud cost estimates, the range of fraud reduction implies benefits from SOX 404 ranging from $2 to $84 billion. Finally, assume fraud externalities range from one to three times direct costs.231 This implies benefits ranging from $4 to $336 billion.

Table 2.

illustrative quantitative cba/fr of sox 404

Table 2 summarizes. The high end of costs is far higher than the low end of benefits, producing a net cost of $289 billion, but the low end of costs is far lower than the high end of benefits, producing a net benefit of $317 billion. Depending on assumptions, guesstimated CBA suggests that SOX 404 could be a very good idea, a very bad idea, or anything in between. If one arbitrarily chose the range’s midpoint, SOX 404 created a net benefit of $9 billion. But this bottom line is highly sensitive, as reflected in Table 3, with net benefits changing by between 2x and 13x as one moves from low to high values for each of five major inputs into the illustrative IAM.

Table 3.

sensitivity of output of illustrative iam to inputs

This illustrative IAM is crude: it implicitly resolves all of the open issues reviewed above and uses many assumptions. The IAM could be challenged on numerous fronts: (a) indirect costs are omitted; (b) open issues on baselines and counterfactuals are resolved in favor of higher cost estimates, but discounted by an arbitrary thirty percent; (c) the current run-rate for direct costs is assumed to last indefinitely, contrary to the SEC’s survey of SOX 404 costs that suggests that costs can be expected to fall;232 (d) transfers from U.S. to non-U.S. persons are ignored; (e) transfers from fraud victims to other shareholders are counted; (f) the ratio of externalities to transfers is borrowed from research on crime, not fraud; (g) discount rates are from OMB Guidance; (h) the fraud reduction effect is assumed to be a one-time permanent reduction; (i) the rate of fraud reduction is derived from a before-and-after study that may wrongly misattribute changes to SOX; (j) the rate of fraud reduction is derived from studies of earnings, and not the full range of fraud that SOX might reduce; and so on. A change in any of these assumptions would change the bottom line. This list of serious debatable limits could be extended for many pages. Any serious contest between opposed analysts would add to the upper ends of ranges of both costs and benefits.233

B. Case Study #2: Independent Boards for Mutual Funds

The second case study is of the mutual fund governance rules proposed by the SEC in the wake of the market-timing scandals of the early 2000s. Together with the review of the Chamber of Commerce case in Part II.B, this case study illustrates, among other things, how judicial review of CBA can penalize an agency for transparency about the limits of its ability to quantify the costs and benefits of a rule.

The ill-fated mutual fund governance rules had their origins in 2003, when New York Attorney General Eliot Spitzer ended his prepared remarks at a Harvard Law School reunion event with a dramatic J’accuse! Pointing a finger at a fellow panelist—a lawyer from Fidelity Management234—Spitzer announced that his office was about to reveal widespread fraud in the mutual fund industry. Over the next year, twenty-six advisory companies settled cases alleging violations of the securities laws in which select investors were permitted to harm funds and other investors by engaging in late or frequent trading that was either contrary to SEC rules or contrary to disclosed fund policies.235

Scandals at this scale had not hit the fund industry in decades, and while the wrongdoing alleged varied from fund complex to fund complex, the most troubling charges involved conflicts of interest between the fund advisors and the funds they advised.236 Conflict-of-interest transactions had been banned in 1940, but because many conflict-of-interest transactions could benefit funds, the SEC had adopted a series of exemptions, subject to a fund’s meeting set conditions.237 In 2001, the SEC had tightened the conditions, increasing the share of independent directors from forty percent to a majority for funds wanting to use the exemptions (as most funds wanted).

1. The Rules

In response to the scandals highlighted by Attorney General Spitzer, the SEC proposed further tightening of the conditions for exemption, (1) requiring a fund’s board to contain seventy-five percent independent directors and (2) adding a requirement that a fund board chair be independent of the advisor.238 The latter requirement was anathema to Fidelity Management—one of the largest fund complexes, privately held, and dominated by its founder, Ned Johnson, who chaired boards of all 292 funds advised by Fidelity.239 Fidelity paid for a study that found a negative correlation between independent board chairs and fund performance but which acknowledged that the correlation could be due to “other important differences that may have impacted performance results,” such as the prevalence of split chairs in bank-sponsored fund groups.240

The SEC adopted the more stringent conditions by a three-to-two partisan vote in August 2004. In its rule release, the SEC included a 1,680-word CBA/FR and a lengthier discussion of the conditions’ benefits in its general assessment of the conditions.241 The CBA/FR was qualitative, and the rule was justified because, in the SEC’s view, independent directors and chairs were “more likely to be primarily loyal to the fund shareholders rather than the fund adviser”;242 in addition, the independent directors and chairs were more likely to effectively manage conflicts of interest such as those involved in the 2003 scandals.243 The SEC explicitly noted it had not conducted a quantified CBA/FR, as it could not quantify either costs or benefits. The agency also stated that it was “not aware of any conclusive research that demonstrates that the hiring of an independent chairman will improve fund performance or reduce expenses, or the reverse.”244 Within weeks, Fidelity persuaded Senator Judd Gregg (a Republican from New Hampshire) to include a rider to an omnibus bill;245 the rider required the SEC to study the need for tightened conditions, resulting in an SEC study that was released in April 2005. That study contained seventy-seven pages of conceptual CBA/FR, showing that the Fidelity-commissioned study was sensitive to assumptions and could not reliably establish what it purported to show. After the SEC conducted the study, the Chamber of Commerce sued to overturn the rule under the APA and the ICA, a suit that ended in Chamber of Commerce v. SEC as described in Part II.246

2. The Aftermath of Chamber of Commerce II

After the second Chamber of Commerce decision,247 the SEC requested that the SEC’s Chief Economist248 reevaluate the governance rules yet again. That request led to two memos—publicly released with a request for public comment.

In those memos, the Chief Economist concluded that (1) more independent boards were more likely to better protect investors, but (2) little evidence existed to establish that board composition would create higher returns.249 These two conclusions, seemingly in tension, could be reconciled by one or more of three further conclusions: (a) “no sound structural model [exists] . . . to isolate the effect of a . . . board decision on performance”; (b) “inherent limitations to data and statistical tools . . . may render it difficult for research to identify relations that . . . may be economically significant”; or (c) “there may not be a unique relation between governance and performance.”250 In other words, the state of finance research was such that no CBA/FR of the mutual fund governance rules was feasible. At the same time, economic theory (particularly agency cost theory) and the Chief Economist’s judgment, based on the research reviewed in the memos, supported a qualitative judgment that the rules would better protect investors.

3. What Would CBA of the Mutual Fund Governance Rules Require?

Implicit in the Chief Economist’s memos is a sketch of what quantified CBA/FR of the governance rules would look like. As the memos noted, the best board structure (in terms of independence) depends

on the . . . consequences of increasing the influence of outsiders . . . . [O]utsiders may bring expertise and independence [and] improve the quality of management decisions and manage conflicts of interest that insiders have, thereby increasing the value of the firm [but] may lack information about the “inner-workings” of the firm and other firm-specific knowledge [which if] difficult to extract . . . may diminish the quality of management decisions and reduce the value of the firm.251

Because this tradeoff may vary by fund, a fixed minimum share of independent directors may benefit investors in one fund by preventing an advisor from influencing the board to nominate too few independent directors. The same minimum may harm investors in another fund by raising the level of independence beyond the optimum for that fund.

Because optimal boards likely vary, however, and because board structure is only one of many factors that influence firm value, an empirical comparison of value at funds with more independent directors in the pre-rule context would not generate reliable information about the effect of the rules. Governance scholars have known this fact—that cross-sectional observational studies produce only weak information about the merits of endogenously chosen governance structures—for some time.252 A source of governance variation that is exogenous with respect to fund value is a necessary but insufficient condition for identifying the average effect of a proposed rule about the feature. Few exogenous sources of variation for fund governance exist, other than as a result of SEC rule changes—and even for those changes, the effects they have caused are likely to be sufficiently small as to be difficult to discover, even with the best cross-sectional modeling.

This identification challenge is fundamental and greatly undermines the reliability of any guesstimated CBA/FR of rules on fund (or corporate) governance.253 The difficulty is even more severe than it was in the SOX case study because the plausible importance of any detail of governance is lower for SOX, which combined multiple institutional and enforcement changes. Anything that changes fund values—for example, anything that changes the value of a fund’s investments—can confound the ability of researchers to identify the effects of governance changes. Fund investment values undergo changes that are continuous and large (money funds aside) relative to the effect of governance details. A shift from fifty percent to seventy-five percent independent directors will not have an effect on value approaching a small fraction of common market-affecting events.254 In the language of econometrics, the “power” of statistical tests given available data is too weak to detect, much less reliably and precisely quantify, the effects of most governance changes—even if we had examples of changes that were plausibly exogenous. All of these points are made plain in the Chief Economist’s memos, even if they were not explained in the SEC’s releases or subsequent D.C. Circuit opinions.255

To be sure, any change in governance mandates will generate adjustment costs—the focus of both the SEC in the “cost” section of its rule release and of the Chamber of Commerce in its lawsuit—that could be quantified (or at least bounded) based on survey evidence. But if benefits of a rule cannot be quantified, and larger potential costs of the rule due to fund performance cannot be quantified, it remains unclear why the failure to quantify adjustment costs is a significant failing or how (if provided) such information would materially improve public understanding of the effects of the rule. This point is even more compelling when, as here, even the quantifications would vary depending on private responses that could not be forecast with any precision, as the D.C. Circuit acknowledged in Chamber of Commerce. Put differently, litigation challenging the SEC’s rule focused on an immaterial subset of the likely costs and benefits of the rule, and had the SEC done exactly what the D.C. Circuit ultimately said it had to do, the result would likely have had no material effect on any assessment of the rule. Yet one would not know this from reading the D.C. Circuit’s opinions or much of the commentary on the case.256 The litigation is a perfect example of how CBA law—here, judicial review of CBA/FR—can obscure more than illuminate.

4. The Aftermath of the Aftermath

Because it was unclear if the Chief Economist’s memos represented the end or the beginning of another stage in the SEC’s efforts to revise governance rules, Fidelity filed a 141-page comment in response, including a twenty-two-page analysis of the Chief Economist’s memos by me (for which I was paid a fee, giving me a financial interest in this topic).257 In my analysis, I critiqued the memos on the ground that the research used to support the qualitative conclusion that the rules would better protect investors was weak, inconsistent, and at times at odds with the summary in the memos. I also outlined a number of potential costs to the proposed rules that had not been noted in the Chief Economist’s memos.258

Subsequently, the SEC has taken no more action to re-propose its governance reforms. What is unclear, however, is whether its decision was based on a genuine change of policy. Two less optimistic possibilities exist: (1) between Chamber of Commerce II and the SEC’s giving up on the rules, the SEC Chair changed identity (William Donaldson was replaced by Christopher Cox); and (2) the ongoing litigation threat, coupled with the fact that reliable quantified CBA/FR for the rules remains unfeasible, led the SEC not to want to risk another morale-draining, resource-depleting court loss,259 even if it continued to believe that the governance rules would benefit investors at a low cost. In favor of the last possibility is the fact that the Chief Economist’s memos were released after Christopher Cox became Chairman, but they supported re-adoption, and nothing in the public commentary (including my comment) provided any compelling quantitative reason for the SEC to change its mind. While the qualitative reasoning in the public comments may be part of the explanation, a dysfunctional system of judicial review seems likely to be a bigger part of the explanation.

C. Case Study #3: Heightened Capital Requirements for Banks

The third case study is of bank capital rule reforms adopted in the wake of the 2008 to 2009 financial crisis. Among other things, the case study illustrates how difficult it is to quantify one of the core benefits of a great number—perhaps the majority—of financial regulations: reductions in systemic risk. The difficulty is in part due to the relatively small number of comparable crises from which to draw reliable inferences about the effects of crises or the effects of regulations on them.

“[Y]ou only find out who is swimming naked when the tide goes out.”260 Warren Buffett’s perhaps self-congratulatory moral was occasioned by losses facing casualty insurers after 9/11, but it captures a central fact of the 2008 crisis: banks were revealed to be grossly undercapitalized for risks they had been running. Undercapitalization was evident in the failures of both commercial banks—Washington Mutual, Wachovia—and investment banks—Lehman, Bear Stearns. But it was also true of the more than 700 banks bailed out by the United States.261 Even the “best” performing U.S. banks during the crisis lost significant amounts of money, needed to raise capital on terms suggesting pre-crisis undercapitalization,262 and would have failed without massive infusions of liquidity by the Federal Reserve, through near-zero interest rates and three rounds of “quantitative easing,” six years after the crisis began.263

Capital shortfalls were global but not universal. Banks in the United Kingdom, France, Germany, and Belgium failed or needed government support to stay open, while banks in Canada and Australia did not, in part due to tight capital regulation.264 In a cross-section of banks, those with more capital and those based in countries with more stringent capital regulation did better than banks elsewhere, controlling for other factors.265

1. Regulatory Response

It was thus inevitable that regulators around the world would impose new, higher capital requirements. Capital regulation is coordinated for global banks (on a voluntary multilateral basis) by the Bank for International Settlements (BIS) based in Basel. More precisely, BIS hosts the Basel Committee on Banking Supervision (Basel Committee), composed of members from twenty-seven countries, which from time to time reaches consensus on a unified set of capital regulations for banks.266 Bank regulators in the member countries then transpose the consensus to national regulation.

Each U.S. banking agency (the Federal Reserve, the Office of the Comptroller of the Currency (OCC), and the Federal Deposit Insurance Corporation (FDIC))267 participates in the Basel Committee.268 Following the crisis, the Committee hosted talks on new capital guidelines (called Basel III to distinguish them from two prior guidelines). This new round focused on tougher capital guidelines for large banks engaged in cross-border transactions or activities with a lack of substitutes,269 and on liquidity requirements—with the aim of addressing liquidity risks that played a greater role in 2008 than in prior crises.270 The new capital guidelines included leverage ratios so banks will need to hold a minimum ratio of capital to assets, even if those assets nominally have a low level of risk, such as highly rated mortgage-backed securities.271 Other requirements in the guidelines included more common equity; tougher treatment for credit default swaps and counterparty risk; securitizations; and risk management; and a surcharge for the very largest, most complex, and interconnected banks, known as “systemically important financial institutions,” or SIFIs.272 The Committee circulated capital guidelines in December 2010 (revised in June 2011) and liquidity guidelines in January 2013. The U.S. agencies proposed capital requirements for U.S. banks in August 2012, eliciting over 2,500 comments before being finalized in October 2013, and proposed new liquidity requirements in November 2013.273

2. CBA/FR of Basel III

Although U.S. banking agencies briefly discussed costs in reviewing comments on their rules,274 none of the U.S. banking regulators included formal CBA in transposing Basel III to U.S. law. However, the Basel Committee itself, in consultation with the International Monetary Fund, published its own CBA/FR.275 The Committee elicited CBA/FR that focused on costs to its members (central banks and bank regulatory agencies), twenty-three of which obtained data and analyses from 263 large banks worldwide.276 Subsequently, the Fed’s counterpart in the United Kingdom (then the FSA)277 extended the Basel Committee’s CBA/FR in published white papers.278

Collectively, the work of the Basel Committee and the FSA on Basel III’s higher capital rules provides another detailed illustration of what CBA/FR looks like for a financial regulation with large if narrow significance. A review of these publications does not leave a reader with much confidence in using guesstimated CBA/FR to guide regulation. CBA/FR of the new rules required complex social and economic predictions. First, the analysis had to estimate benefits of heightened capital and liquidity requirements; those benefits were identified as less frequent and/or severe financial crises. Two sub-models were needed, one to estimate the cost of a crisis and one to predict the frequency of crises. The CBA/FR then faced the challenge of forecasting the causal effect of the requirements on each modeled relationship (incidence and effects). Finally, the CBA/FR had to estimate the costs of the requirements—posited to be lower lending by the banks subject to the rules. Each of these models is discussed below.

3. Costs of a Financial Crisis

Focus first on the costs of a crisis. One pair of commentators has suggested that this element of CBA/FR should be “easy”: “Agreement on a figure in the range 150 billion to 3 trillion dollars (viz. a crisis cost between 1 percent and 20 percent of US GDP . . .) would seem relatively easy to reach given the widely respected estimates of Reinhart and Rogoff.”279 Unfortunately, this view is too sanguine by more than half. Other estimates of the costs of financial crises range from 90% to 350% of world GDP (Bank of England); 18% to 48% of UK GDP (FSA 42); and 10% to 210% of UK GDP.280

To state the obvious: these ranges do not even overlap. The high end of Posner and Weyl’s range (20% of U.S. GDP) is less than one-fourth of the low end of the Bank of England’s range and is barely above the low end of the FSA’s range. The high end of the Bank of England’s estimate is seventeen times that of Posner and Weyl’s, and Yan et al.’s estimate is eleven-and-a-half times larger than Posner and Weyl’s. In absolute, comparable, present value dollars, these differences are enormous: trillions, not billions.

One may object, fairly, that Posner and Weyl’s estimate is for all future crises, whereas the other ranges are for the recent crisis. But there are two responses. First, with respect to the recent crisis, the ranges still vary substantially. Second, as discussed more below, no consensus approach exists to resolve which historical data one should use in estimating the cost of future crises. Data from 1929, included in Reinhart and Rogoff, on which Posner and Weyl rely,281 are not obviously more or equally relevant to future crises than data from 2008. A longer set of historical data has the advantage of allowing costs to vary with factors that fluctuate or cycle over decades, and dampens the effect of differences of estimated costs of any particular crisis. A shorter set of data from more recent periods has several advantages as well. First, a shorter dataset promotes better modeling of current economic, legal, and political conditions, including the centrality of finance to the economy,282 which has arguably increased over time; it also allows one to account for the presence of laws and institutions that socialize some of the risks of crises, such as FDIC deposit insurance, and that did not exist in 1929. Second, a shorter dataset enables one to reduce the number of disputes that can be expected over which crises to include in the dataset. Given the tradeoffs between a shorter and a longer dataset, neither choice clearly dominates.

An examination of CBA/FR conducted for the Basel Committee reveals methodologies and estimates of the costs of crises more disparate than in the studies just summarized. The Committee reviewed twenty-one studies. Two provided estimates of peak-to-trough losses during the crises studied, while thirteen provided cumulative loss estimates. The present value of the average cost in the latter studies ranged from 16% to 302% of pre-crisis GDP (sometimes measured against domestic GDP, sometimes global). Several include a lower bound of zero(!),283 while the highest upper bound was 1041% of pre-crisis GDP. One study presented results from two methods that varied at the mean by a factor of five and at the high end by a factor of ten.284

The Basel Committee’s qualitative summary is “that results in the literature are surprisingly consistent.”285 But this conclusion is inconsistent with the committee’s statement elsewhere in its report that one can find “a significant range of crisis outcomes across studies and individual episodes.”286 Presumably, the “significant range” of outcomes is “surprisingly consistent” when measured against prior expectations that the results would lack coherence altogether.

The table summarizing the committee’s findings,287 converted into Figure 2 here, shows the sensitivity of the results to assumptions and methodological choices. The primary drivers of the sensitivity of results are: (1) selection of historical data points; (2) assumptions about whether economic losses will be permanent or temporary, and if temporary, how long crises will last; and (3) what policy response will be triggered by the crisis. For each driver, a number of choices must be made, and each choice has large effects on the bottom line of the CBA/FR.

Figure 2.

range of estimates of costs of financial crisis288

For the simplest driver—choice of data—at least three contestable choices are required. First, a “financial crisis” must be defined: crises can be subjectively and judgmentally chosen (“I know it when I see it” approach)289 or objectively chosen, and either way can be based on a variety of data, including data regarding market volatility,290 bank runs,291 bank closures or nationalizations,292 bank bailouts,293 stock market declines,294 and ratios of non-performing loans to bank assets.295 Some distinguish banking from market crises; others include banking crises as a subset of financial crises.296 Second, time periods must be chosen—both for the overall dataset (how far back to go in history?) and for each crisis (because the duration of a crisis affects the count and size of effects).297 Third, one must decide what geographic scope to consider: should one consider only crises in the United States, in developed countries (and if so, how to define “developed”?), or all countries?

These choices have large effects on outputs. One study of the costs of financial crises presents two historical samples, with its bottom line estimate doubling depending on which sample is used.298 Even over the same historical period, one study counts 160 banking crises, including many that caused relatively small losses, reducing the average loss caused by the crises counted, while another study counts twenty-three, which caused large average losses.299 The differences are attributable to (a) basic definitional choices; (b) whether to count poor, developing nations or nations with poorly developed financial markets; and (c) how (and whether) to count countries that experienced multiple crises close in time: if all crises are counted separately, the average cost falls, because some of the crises are brief episodes paving the way to a larger crisis.300

Further illustrating the fragility of cost-of-crisis models is the recent kerfuffle involving Reinhart and Rogoff (R&R), on whose “widely respected estimates” Posner and Weyl rely. R&R’s publications on the effects of crises turned out to be indisputably301 flawed because of a spreadsheet error that went undetected for over three years302 (in spite of the fact that the study was cited prominently in policy debates).303 While the spreadsheet error caused R&R’s analysis to drop data for five countries they intended to include, the error had no effect on their estimates of the direct costs of financial crises—that is, the fiscal costs incurred by governments attempting to resolve crises. However, the error did affect estimates of the indirect costs of financial crises—that is, the depressive effects on growth caused by higher levels of debt incurred as part of a policy response. As discussed below, whether and how to count indirect effects of policy responses are further sources of sensitivity in modeling the cost of crises. The same researchers who discovered the spreadsheet error also challenged separate choices by R&R in their analyses—what the critics termed a “selective exclusion of . . . data” (for Australia, New Zealand, and Canada) and an “unconventional weighting of summary statistics” that amplified the effects of exclusion of New Zealand.304 While R&R disagree on these points, they do so in part on the ground that their work is historical, consisting of “archival research, involving constant judgments at every step.”305

Even if observers agreed on historical crises to estimate the cost of future crises, two additional output-sensitive inputs—temporary versus permanent effects and policy responses—intensify the unreliability of CBA/FR of Basel III. Some studies assume the effects of a crisis on the economy are transient—that is, a crisis causes a temporary drop in activity, followed eventually by higher-than-normal “catch-up” growth, bringing long-term output trends back to where they would have been without the crisis. Other studies assume that the effects are permanent—that is, economic activity never catches up to where it would have been without the crisis. If one takes the median of the average of estimated losses across studies, as the authors of the BCBS 173 did, the difference caused by this one assumption triples the losses.306 If harms are large (for example, 158% of pre-crisis GDP in BCBS 173), then differences between permanent-harm and temporary-harm models are even larger—up to a hundred times larger.307 A related force increasing the sensitivity of results in permanent-harm models—which by definition extend into the indefinite future—is the choice of discount rate.308

A third source of sensitivity of social costs to modeling assumptions is perhaps the most troubling for anyone hoping CBA/FR can produce reliable information: the political and policy response to the crisis. As the last crisis reminded us, a major financial crisis can provoke a range of policy responses. Politicians may bail out banks; tighten, loosen, or repeal regulations; increase liquidity through conventional monetary policy (cutting interest rates) and less conventional instruments (“quantitative easing”); stimulate activity directly with government spending or tax cuts; other responses; or some combination. Each response can have benefits and costs, ranging from lending constraints, moral hazard, and the future frequency of crises; inflation; deficits; debt; and reduced medium- to long-term growth. These policy responses can vary in intensity as well. Depending on the policy response, the effect of a crisis can vary significantly, and the models reviewed in the Basel Committee CBA/FR make assumptions about the policy responses and their effects.

To predict policy responses, CBA/FR must include what amounts to political speculation. For if economic inputs to CBA/FR models are uncertain, political inputs are even more so.309 To see this, simply note the varying policy responses across developed economies to the recent crisis. The United States created a very large (relative to the economy or the tax base) fiscal stimulus through deficit spending, while the United Kingdom “committed itself to early fiscal retrenchment.”310 The United States implemented the most aggressive monetary program in history, through the novel technique of buying massive amounts of mortgage-backed and other fixed income securities, while the European Central Bank remained more focused on preventing inflation, and the Bank of Japan’s balance sheet increased only slightly over the crisis period.311 Policy responses also change in response to learning (or claims to learning) from past crises—compare recent U.S. monetary and fiscal policy to responses to the Great Depression312 and to that of Japan during the 1990s313—but that implies that predicting future policy requires predicting the future path of economic theory and the results of retrospective analyses of past policy interventions. This is not to mention financial rescue programs, such as TARP.

These are not second-order considerations. Informed observers have attributed much of the difference in the duration of the current U.S. recession, on the one hand, and the contemporaneous U.K. recession and the historical U.S. Great Depression, on the other hand, to policy responses.314 Should the current legitimacy of otherwise desirable regulation turn, to any significant degree, on debates or assumptions about predictions of future politics? That is what CBA/FR advocates effectively, if tacitly, presume.315

4. Frequency of Financial Crises

Even if the costs of financial crises could be estimated with precision and reliability, these costs would have to be paired with estimates of the frequency of crises to arrive at an estimate of the benefit from regulations that reduce crises’ frequency. This modeling faces similar challenges as estimating effects: subjectivity in selection among relatively small numbers of historical data points and sensitivity of results to choice of data points. The Basel Committee simply took average frequencies from two studies316 over an arbitrarily chosen period and set of countries (1985 to 2009 for G10 and BCBS countries, except Russia and China, which were included from 1992 on) and made the heroic assumption that this average was a good estimate of the probability of a crisis for any given year and country.317

The FSA, by contrast, used a longer time period (1970 to 2007), a narrower set of countries (OECD countries), and relied on a multivariable logit approach relating the likelihood of a crisis in a given year “to a vector of explanatory variables,” with observed crises in the past coded one and non-crisis years coded zero.318 This approach relies on the logistic cumulative distribution to predict future crises and is an improvement over BCBS 173 if interdependencies among time-varying observables affect crisis frequency, as seems likely. For example, housing prices have varied over time, and crises often coincide with (partly causing, partly being caused by) bubbles in housing prices, so crisis odds would not be uniform over time but would vary in cycles and across countries. However, the small number of crises that can be modeled this way (FSA 38’s data included fourteen) limits the value of this approach, in statistical degrees of freedom and in robustness, and the functional form imposes assumptions on the shape of the distribution of crisis probabilities that is nowhere defended in the FSA’s publications.

Because of differences in approach, the FSA’s results differ markedly from the Committee’s results. BCBS 173 reports an estimated baseline probability of a crisis per year for all countries of 4.5%.319 FSA 38 reports a baseline probability ranging from 0.7% (for Germany) to 21.7% (for the United Kingdom)—that is, from one-sixth to five times the estimate used by BCBS 173.320 Again, the sensitivity of outputs to assumptions illustrates how fragile CBA/FR of capital regulation remains.321

5. Effects of Higher Capital Requirements on Financial Crises

A third task necessary to estimate the benefits of higher capital requirements is to estimate how higher capital will affect the frequency and effects of future crises. The challenges are similar to those outlined in the case studies of SOX and mutual fund governance above, if slightly less difficult. The challenges are less difficult because capital levels have a more mechanical relationship to bank failure than disclosure and governance regulations have to fraud and fund performance, respectively. If a bank’s capital falls below zero, it is by definition insolvent and will be either closed, nationalized, or bailed out (and/or suffer a bank run)—all of which (at least by most definitions) feed directly into the occurrence of a financial crisis.

Nevertheless, the modeling exercise remains difficult here, too, and includes a long list of challenges. Three are reviewed here: (1) baselines; (2) packages; and (3) international externalities.322 The first question in any CBA is what baseline to use. Similar to the effect of fraud revelation on disclosure practices in the SOX case study, financial crises stimulate banks to raise their capital levels even without regulatory reform, as private actors increase the price of lending or investing in now apparently riskier banks. So how should one measure the effect of a regulatory mandate for new capital—against the baseline of pre-crisis capital levels, or against levels that could be expected in the wake of the crisis without the regulation? The argument for the former—advanced in FSA 42—is that “banks will tend to relax their post-crisis holdings of capital as the economic cycle strengthens.”323 This seems sensible as a rough prediction, but it is not anchored in an equilibrium model of bank behavior. After all, banks observe the same indicia of the probability of a crisis as used in the FSA’s CBA/FR of Basel III. Bank investors can observe those indicia and bank capital levels, so why should we assume that bank capital levels only subside, rather than rise and fall as the risk of a systemic crisis rises and falls? It may be that private actors lack sufficient incentives to demand that an optimal level of capital be retained by banks, but for CBA/FR of capital requirements, the baseline itself—the capital that private actors would demand—is likely to change over time in unpredictable ways.

Part of the reason that private actors may lack incentives to demand that banks retain optimal capital is that they face moral hazard due to the likelihood of bailouts and other policy interventions. But that fact calls into question the validity of using pre-crisis capital levels as appropriate baselines altogether. Has moral hazard increased, decreased, or remained the same after the bailouts of 2008? Lehman failed, and Bear Stearns and Merrill Lynch were forced to sell at fire-sale prices—so perhaps investors are now less certain about future bailouts. But, of course, more than 700 U.S. banks were bailed out,324 not to mention the indirect bailouts through the various liquidity facilities established by the Federal Reserve Board—so perhaps investors face even more moral hazard than before. FSA 42 asserts that the pre-crisis period was one in which “banks’ decisions . . . were not distorted by the immediate influence of the crisis or regulators’ response to the crisis.”325 But it presents no evidence to support that assertion. Any rational actor who anticipates a crisis should, given policy responses to past crises, also anticipate that a bailout may occur with some probability, and the capital levels it will demand will be affected by that anticipation. The better point, then, is that a model of the effect of future capital regulation should start with a baseline that explicitly takes into account moral hazard as a permanent condition of financial markets without adequate regulation. However, establishing such a baseline would require estimating the subsidy provided by the moral hazard to bank investors—a task not yet convincingly tackled by researchers.

Another challenge is that Basel III consists of a package of reforms, not one reform. As FSA 42 notes, if the probability of a crisis is non-linear in the level of bank capital, as assumed in a logit model (and as seems likely), then the effect on that probability of each piece of the reform package will depend on the sequence in which the pieces are adopted.326 As with SOX, the best one may be able to do in estimating the causal impact of a package of reforms is to evaluate the package as a whole. For the CBA/FR of any given package of reforms, this is not a critical problem, but it does undermine the value of CBA/FR because it allows regulators to determine (to an extent) what is being evaluated—and may allow a package to include some reforms that are net positive (if evaluated on their own) with other reforms that are net negative (if evaluated on their own), as long as the former outweigh the latter.

A third challenge to estimating the causal impact of Basel III, also noted in FSA 42,327 is that it is a voluntary multilateral initiative, which means that it will be implemented in different ways at different times in different countries. Implementation in one country will affect how banks in other countries act, independent of the effect of implementation by their own regulators. If, for example, U.K. banks are required to increase capital, they may not only reduce lending but focus continued lending on geographies or sectors where interest margins are highest, which in turn may affect currency and trade flows. An increase in U.S. capital regulation under Basel III, being evaluated in a CBA/FR by a U.S. regulator, should take into account the simultaneous shift in lending activity by U.K. banks, as well as the direct effect on U.S. banks. In a global financial market, the externalities of regulation create modeling difficulties of their own—adding yet more necessary assumptions regarding how the regulations will actually affect the probability or impact of future crises.

6. Costs of Higher Capital Requirements: Less Lending?

Finally, the costs of higher capital requirements must be estimated. The standard framework, employed by the Basel Committee and the FSA,328 is to assume that a bank required to hold an increased amount of capital will raise corporate borrowing costs and so cut lending. The reasoning is simple: banks must pay their investors a minimum expected rate of return on their invested capital; if more capital is required, the bank will have to generate greater return; to generate a higher return, a bank must charge more to its borrowers; at a higher cost of borrowing, less lending will occur. The model further assumes that with lower lending by banks, economic output will fall.

As with the models of the benefits of capital requirements, however, models of the effects on the amount of lending (and its knock-on effects on output) require numerous contestable assumptions, and their outputs are sensitive to those assumptions. Among the assumptions are: (a) the cost of bank equity and whether it will fall in response to the change in capital levels required by the rule; (b) the ability of borrowers to substitute among different sources of financing (and at what cost); and (c) how non-bank sources will be affected by an increase in bank capital requirements and the reduction in risks and effects of financial crises.329 Each has major impacts on the output of the cost model alone.

The uncertainties associated with these assumptions are underscored by the fact that one prominent set of economists believes the social costs of higher capital requirements “would be, if there were any at all, very small.”330 The authors point out that higher taxes, if paid by banks as a result of shifting from debt to equity finance in response to capital requirements, are not a social cost, because the shift reduces the distortive effects of a socially harmful tax code.331 The authors argue that moral hazard induces banks to remain larger than is socially efficient, so that even if higher capital induced large banks to shrink, the overall impact on lending would be offset by increases in lending by other banks or financial institutions.332 By contrast, the Basel Committee, based on its modeling and inputs from self-interested banks, concluded that the proposed requirements in Basel III would reduce steady-state output (gross domestic product) by between 0.25 and 0.92 percentage points,333 which translates into $1.4 trillion in present value terms at the mid-point of this range for the United States alone. As with estimates of benefits, respectable CBA/FR opinions vary in their assessments of the present value of Basel III’s costs by more than $1 trillion.

D. Case Study #4: The Volcker Rule

The fourth case study also focuses on a rule emerging from the financial crisis: section 619 of the Dodd-Frank Act, colloquially known as the “Volcker Rule.” That rule bans U.S. banks from speculating for their own account (that is, from engaging in “proprietary trading” or holding “ownership interests” in hedge or private equity funds, subject to a number of exceptions).334 This case study reinforces the points made in the prior case study, and also illustrates how difficult it is to assess many important kinds of financial regulations in advance, given the lack of any past data on how new markets will operate.

Specific regulations implementing the Volcker Rule were approved (after many delays) in December 2013 and went into effect on April 1, 2014.335 The formal releases published by the financial agencies in the Federal Register contain no general CBA/FR, presumably because (1) as discussed in Part II.A, no general CBA/FR mandate exists for those agencies; (2) the statutory requirement for and authorization of the rules is part of the Bank Holding Company Act of 1956,336 which does not contain any equivalent to the requirement in the securities laws that the SEC consider “efficiency” or in the commodities laws that the CFTC consider costs and benefits;337 and (3) nothing in the language of section 619 requires CBA.338 The formal rulemaking contained limited cost-related information in its analyses under the RFA and the PRA339 but no information about benefits or non-compliance costs.

The OCC, however, did release separately a CBA/FR of the Volcker Rule.340 It identified a number of “non-monetized” (qualitative) benefits: improved supervision by bank regulators (due to metrics reporting required by the rule); better management of risk by bank managers (for the same reason); reduced conflicts of interest; protecting “core banking services” and improved bank safety and soundness (reduced risk of bank failures); reduced “tail risk” from trading activities and reduced risks of financial crises; improved corporate governance of banks resulting from reduced stock market liquidity; and reduced harms caused by excess liquidity.341 As the OCC noted, “benefits of the regulation can be difficult to quantify including the value of enhanced economic stability.”342

The OCC also identified a number of costs. For a subset, the OCC provides quantified estimates: compliance costs ($405 to $541 million); additional capital costs for permissible investments in covered funds ($0 to $165 million); the OCC’s own costs of supervising compliance with the new rule ($10 million); and a one-time hit to the value of assets owned by banks but restricted by the rule, resulting from reductions in demand for those assets due to the rule. For the last type of cost, the OCC drew on academic research estimating a similar haircut in corporate bond values when bonds are downgraded by credit rating agencies and insurance companies (subject to regulations limiting their ownership of junk bonds) are forced to sell such bonds, deriving a range of costs from $0 to $3.6 billion.

However, the types of costs that are likely to be the largest ongoing costs were not quantified. Foremost among these non-quantified costs is the reduced liquidity in markets where banks were significant trading participants, particularly arising from inter-dealer trading, which is not treated as a permissible source of “customer” demand under the rule.343 Banks, as a result, will not be able to hold certain assets as “inventory,” which will reduce liquidity in the markets for those assets and make it harder for banks to share risk with other banks when permissible customer-driven trading results in banks’ taking on large blocks of equities. As a result, banks may incur higher costs to hedge or shed those risks, or face more difficulties in managing risks. Further, the reduction in liquidity caused by the ban on inter-dealer trading will likely reduce the depth of those markets and the ability of issuers to raise capital in those markets.344 Another potential cost of the rule is similar to one noted above for the Basel III rules: migration of trading activity to non- or less-regulated “shadow” banks, which could pose systemic risks, offsetting (and possibly exceeding) the benefits of risk reduction within the banking system.

In sum, as with the foregoing case studies, the OCC’s CBA/FR did not include a quantification of the benefits and only quantified a subset—and likely a small portion—of the costs of the Volcker Rule. The result was that the OCC confidently categorized the rule as “major” for purposes of the CRA,345 because that categorization only requires bounding the rule’s costs, but did not reach any conclusion about the rule’s net costs and benefits.

Could the agencies go beyond conceptual CBA and conduct a reliable, precise, quantified CBA/FR? The short answer is no. There is simply no historical data on which anyone could base a reliable estimate of the benefits of preventing banks from engaging in proprietary trading or investing in hedge and private equity funds. Any effort to quantify those benefits will run straight up against the difficulties described in the case studies above. While Basel III capital rules address the “liability” side of a bank’s balance sheet, and the Volcker Rule addresses its “asset” side, both rules have as a core intended benefit the reduction in the frequency and magnitude of systemic financial crises. Thus, as with Basel III, any complete quantified CBA/FR of the Volcker Rule would require the same components discussed above for Basel III to estimate the costs and frequency of financial crises (macroeconomic modeling, subjective data selection, prediction of policy responses).

The difficulties with the Volcker Rule are compounded beyond Basel III, however, for two reasons. First, the rule has additional, separate benefits, such as the mitigation and reduction of conflicts of interest, which (as with the mutual fund governance rules) can only be quantified by relying on causal inferences with low-powered tools about complex institutional arrangements. Second, and perhaps more important, it remains unclear how, if at all, the Volcker Rule will in fact reduce the risk or cost of financial crises. The Rule’s proponents (including Volcker himself) strongly believe that it will, by decreasing the role of speculation within banks and perhaps by limiting the ability of banks to attract and retain individuals with a risk-taking temperament.346 But those judgments do not rest on historical data, nor is there any mechanical relationship between an activity (proprietary trading) and failure, as with capital levels. Ironically, then, the primary category of benefits (reduced systemic crisis risk from less speculation by banks) is inherently speculative, as with any novel structural rule or activity ban of a similar kind.

Quantifying the aggregate costs of the rule would be equally difficult. While the OCC quantified a subset of costs, it did not quantify the costs that are likely to be largest—especially the costs of lower liquidity. As the OCC noted, it is possible to quantify those costs: there are research papers estimating the cost of reduced liquidity for specific categories of assets.347 But, as the OCC also noted, any estimates produced by relating predicted reductions in liquidity to this sparse research literature would be “difficult.”348 Among other things, a full set of cost estimates would require predicting the impact of the rule on liquidity across a range of financial markets (including anticipating entry by institutions not subject to the rule—institutions that could be expected to take advantage of any competitive opportunities opened up by the exit of banks subject to the rule). Those estimates would then have to be linked to estimates of the impact on the cost of capital from any expected reduction in the liquidity of one channel for capital raising, again taking into account possible substitution effects from other channels. Then, finally, the effects on output of any estimated capital cost increase would have to be quantified, using a macroeconomic model. As with Basel III, the result would be complex, difficult, constrained by limited data, highly contestable, and sensitive to modeling assumptions.

E. “Gold Standard” Examples of CBA/FR

Perhaps other significant regulations—beyond those explored in the case studies presented above—are more susceptible to quantified CBA/FR. Taking a cue from the adversarial legal system, in which neutral judges rely on advocates to advance the best evidence in favor of a cause, this section reviews two regulations that CBA/FR proponents hold up as examples of “gold standard” quantified CBA/FR349—the SEC’s cross-border swaps rules and the FSA’s mortgage market reforms—on the theory that they should provide the best evidence that quantified CBA/FR is capable of being done in a reliable, precise way. These rules are also high-profile and indisputably significant, and are of interest for evaluating CBA/FR law because the agencies did conduct and publish CBA/FR in response to CBA/FR law: the SEC was responding to the D.C. Circuit decisions reviewed above, and the FSA was complying with a U.K. statute requiring CBA/FR, precisely the kind of mandate that CBA/FR advocates hope to bring to the U.S.

Does either of those rules demonstrate that quantified CBA/FR is feasible and desirable? Far from it—they instead show how easily CBA/FR can camouflage the effects of rulemaking, rather than discipline it. Both case studies show that even motivated and relatively expert members of the public—specifically, the Center for Capital Market Regulation, composed of leading financial industry participants and staffed by technically trained lawyers and economists—can apparently misread the contents and achievements of a lengthy and technical cost-benefit analysis. The case study of the FSA’s mortgage reforms also illustrates that even the most creative and sustained effort to quantify the costs and benefits of a fairly narrow but important financial regulation remained fragile, imprecise, and incapable of significantly constraining regulatory judgment, by the admission of the staff carrying out the analysis.

1. The SEC’s Cross-Border Rules on Swaps

One of the few examples of CBA/FR of U.S. financial regulatory rules praised by CBA/FR proponents was conducted by the SEC, in its proposed rules on cross-border swaps under the Dodd-Frank Act (the Cross-Border Swap Release).350 Those rules are designed to fill a regulatory gap351 relating to over-the-counter (OTC) derivatives markets, which exploded over the past two decades and exacerbated the 2008 financial crisis, causing the insolvency of one of the world’s largest insurance companies (AIG) and triggering a bailout through an unprecedented series of actions by the U.S. Treasury and the Federal Reserve Board.352

The Dodd-Frank Act authorizes the SEC and the CFTC353 to register and regulate entities active in the OTC swap markets,354 and to establish rules for clearing and trade execution, recordkeeping, real-time reporting, and disclosure. Pursuant to this authority, the SEC (in conjunction with the CFTC) has issued two releases defining terms355 and proposed or adopted ten sets of rules on domestic swap activities.356 The Dodd-Frank Act was clear that swap regulation should also cover cross-border activity that could affect the U.S. markets.357 To that end, the SEC proposed a rule in May 2013 to address cross-border swaps comprehensively, issuing one large release collecting, discussing, and analyzing all of the swap-related rules as they would apply to cross-border activities. That release contained roughly 200 pages labeled “economic analysis,” a third of the total release—including both conceptual and limited elements of quantified CBA/FR—and cross-referenced lengthy CBA/FR in previously issued releases.358 By comparison to CBA/FR in most prior SEC releases, the length of the CBA/FR is indeed impressive, which is part of why CBA/FR advocates praised it.359 The SEC’s CBA/FR was also praised because it focused on full, quantified CBA/FR, “estimating the quantitative impact of each key aspect of the proposed rule, rather than simply assess[ing] firm-specific compliance costs.”360

However, a careful (if exhausting) review of the CBA/FR in the Cross-Border Swap Release shows that it is comprehensive only in its qualitative economic analysis of the proposed rules and contains little quantified information, other than for a subset of compliance costs. As noted in passing towards the beginning of the CBA/FR, “Many of the resulting costs and benefits are difficult to quantify with any degree of certainty, especially as the practices of market participants are expected to evolve and adapt to changes in technology and market developments.”361 The SEC divides its CBA/FR into “assessment” costs—the costs of determining if a given entity is subject to swap regulation, a subset of compliance costs—and “programmatic” costs and benefits due to subjecting swaps to regulation.362 The primary programmatic benefits the SEC identified were promoting competition by increasing market access and transparency, reducing search costs, and increasing price efficiency.363 The primary programmatic costs the SEC identified were reduced liquidity and depth in the swap markets due to market participants’ withdrawing because transparency requirements will reveal valuable information, and a potentially increased incentive to “race to the bottom” as participants relocate cross-border operations to jurisdictions with less regulation.364

Almost no information relating to “programmatic” costs and benefits is quantified. No models of competition, liquidity, or prices under the rules are presented. Instead, the SEC repeatedly said that it lacks data and/or an inferential basis for quantifying those costs and benefits. Exceptions include, for example, a quantification of the costs of building a compliant swap execution facility from scratch and maintaining it thereafter or modifying an existing trading platform into compliance and maintaining it.365 But these exceptions prove only the general absence of quantification, as they relate to a subset of the costs of a subset of the rules proposed in the release—a subset of a subset of a subset of what a full quantified CBA/FR would include.

This description is not meant to criticize the absence of quantification. The SEC’s decision not to quantify is fully justified, given the state of available information and research methods. The Dodd-Frank Act effectively required the creation of entirely new OTC swaps markets. Private actors will be reacting to these novel regulations in ways that cannot be reliably predicted. The realization of the rules’ major potential benefit—increased competition—depends upon latent demand for products (transparently cleared swaps). Private actors had only limited incentives to provide these products under prior rules and the value of the products will be altered by other new aspects of the rules, such as segregation and capital requirements. The realization of the rules’ major potential cost—reduced liquidity and depth relative to prior markets—will also be a function of latent demand. The size of the cost will also turn on the importance of proprietary information that may be revealed in more transparent markets. Another major potential cost—an increased incentive for participants to relocate to other jurisdictions—depends on political and policy outcomes in other countries, as well as the ability of international regulatory coordination to cope with or blunt those incentives.

Although justified in this respect, the SEC’s CBA/FR nevertheless must be fairly viewed as conceptual, not quantified. Rather than showing quantification is possible and desirable, as a matter of policy or law, the Cross-Border Swap Release shows just the opposite. Yet CBA/FR advocates have singled out the Cross-Border Swap Release for accomplishing something it did not accomplish.366 How could that be? Perhaps the praise was false, a mere rhetorical pretense in service of the political goal of promoting CBA/FR.

But a more charitable possibility exists: perhaps CBA/FR advocates did not see through the camouflage of the SEC’s release. As noted, the CBA/FR is 200 pages long and incorporates lengthier CBA/FR sections in other related releases. It is turgid, vague, and full of jargon. Discussions of less important assessment costs are longer than discussions of more important programmatic costs and benefits. Specific quantified amounts appear regularly,367 so someone skimming the analysis might surmise that it was filled with quantitative analysis, while in fact the vast majority of the amounts relate to assessment costs or a small subset of programmatic costs, not to programmatic benefits or the most important programmatic costs.368 The release contains lengthy discussions of qualitative costs and benefits of a de minimis exemption from coverage by the rules, while nowhere setting forth a detailed conceptual outline of how one might (in theory) measure the costs and benefits of being covered by the rules. Important points relevant to the limited quantification in the release are buried in footnotes,369 while whole pages are taken up with text such as this:

Segregation requirements would limit the potential losses for security-based swap customers if a registered security-based swap dealer fails. The extent to which assets are in fact protected by proposed Rule 18a-4(a)-(d) would depend on how effective they are in practice in allowing assets to be readily returned to customers. In the cross-border context, the effectiveness of the segregation requirement with respect to foreign security-based swap dealers in practice may depend on many factors, including the type and objective of the insolvency or liquidation proceeding and how the U.S. Bankruptcy Code, SIPA, banking regulations, and applicable foreign insolvency laws are interpreted by the U.S. bankruptcy court, SIPC, Federal Deposit Insurance Corporation, and relevant foreign authorities. In the Capital, Margin, and Segregation Proposing Release, we stated that it would be difficult to measure the benefits of the segregation requirements proposed by the Commission under Section 3E of the Exchange Act; however, we believe that Rule 15c3-3, the existing segregation rule for broker-dealers, would provide a reasonable template for crafting the segregation requirements for security-based swap dealers. The ensuing increased confidence of market participants when transacting in security-based swaps, as compared to the OTC derivatives market as it exists today, should increase the desire to trade security-based swaps and generally benefit market participants.370

Perhaps someone finds this and similar paragraphs illuminating. I do not. Did including it in a 200-page section labeled “economic analysis” in a 650-page release inform the public about the costs and benefits of requiring dealers in cross-border swaps to segregate customer assets? In what way is it “economic” analysis, as distinct from the more general form of analysis that has long been included in adopting releases? The paragraph would look out of place in an economics journal. Even if these 210 words were boiled down to a more succinct, social-scientific style,371 would a law requiring such a statement discipline the SEC, improve the public’s ability to comment on the proposals, or correct the SEC’s potential cognitive biases? I cannot see how.

Again, I do not intend to criticize the authors of the Cross-Border Swap Release; to the contrary, I commend them. They accomplished an important goal—eliciting praise from a group of critics of the SEC’s CBA practices—and likely helped set up the SEC to defend itself against any court challenges to its rules. The staff accomplished here what any rational actor at a regulatory agency would want to accomplish given the court decisions reviewed in Part II above—decisions that have created a strong incentive for regulators to generate precisely the kind of qualitative, lengthy, and largely opaque “gold standard” CBA/FR included in the Cross-Border Swap Release.

2. The FSA’s Mortgage Market Reforms

A second example held up as model CBA/FR is the set of mortgage market rules passed by the FSA in 2011. The FSA was abolished in 2010 (effective in 2013) for its failures to foresee, prevent, and mitigate the 2008 crisis.372 Among its pre-crisis failures was allowing significant amounts of mortgage loans to be made to borrowers who could not repay the loans other than by refinancing or reselling their homes into what optimists hoped would be an ever-rising market.373 Reforms adopted in 2011 require lenders to assess affordability of homes before lending to buyers, to include the possibility of interest rate increases in making those affordability assessments, and to evaluate interest-only mortgages without assuming (as opposed to demonstrating) the possibility of a refinancing.374

a. The FSA’s CBA/FR

Since 2000, UK law has required the FSA to publish a CBA/FR of its regulations and guidance,375 such as the mortgage reforms. That 131-page CBA/FR was attached as an annex to the reform proposal (a “consultation paper” in European legal jargon).376 In it, the FSA summarized the benefit of the main reform (mandatory affordability analysis) as protecting some borrowers “from mortgage impairment,” and its cost as “prevent[ing] [other borrowers] from taking out the mortgage they want.”377

In an effort to quantify and compare those primary benefits and costs, the FSA used a multistep process. First, it applied a multivariate logistic model to a large (n=730,000) sample of loans from 2005 to 2010 to estimate the probability of loan “impairment.”378 It then used ordinary-least-squares regression of the probability of impairment on factors it selected as contributing to impairment to quantify the contribution each factor made to impairment risk.379 It used “judgment” to choose factors relevant to loan underwriting to identify a cut-off where impairment risk increased “markedly,”380 on the theory that this was where the new affordability requirement would have affected sample loans.381

With those models, the FSA concluded that the rules would have prevented roughly 200,000 loans from entering default (“unaffordable” loans), and constrained approximately 530,000 borrowers to take out smaller or delayed loans than they could have taken out and repaid without the rules.382 The FSA then assumed the rules would prevent similar future defaults, which the FSA assumed would create solely social costs and so counted solely as benefits of the rules.383 The FSA further assumed the rules would generate social costs but no benefits if they prevented or delayed borrowers who could have afforded larger or earlier loans from obtaining consumption benefits.384

To quantify a comparison between these direct costs and benefits of the new rules on borrowers, the FSA needed a common metric. Because the FSA had no data on actual demand for loans in a hypothetical world without information asymmetries (a market failure addressed by the rules),385 it estimated effects not on welfare but on psychological “well-being,” for which it had proxy data, derived from a U.K. government household panel survey with data from 1991 to 2008.386 By regressing self-reported well-being scores on “housing-related events” in a fixed-effects regression with other controls from the survey, the FSA generated parameters387 for changes in well-being for events that were (by assumption) related to unaffordable loans (for example, payment problems) or affordable loans (for example, becoming a home owner rather than a renter, moving into a larger home). The FSA found that effects on well-being were “much greater” for payment problems and defaults than for foregone improvements in housing,388 such that the net effects on all affected borrowers were positive overall, despite being expected to stop more affordable loans than unaffordable loans.389

While this procedure allowed for a comparison of direct effects of the rule, by design it did not monetize the effects for use in a full, quantified CBA. To do that, self-reported well-being figures needed to be converted to pounds, to compare to other costs and benefits. Nevertheless, the FSA exploited the happenstance that the effects on well-being of loans’ falling into arrears were similar in size to the effects of a person’s becoming unemployed, a condition more easily monetized by reference to income data.390 The bottom line was an average benefit of £350 per borrower over the period 2005 to 2010.391 Added to this was an additional benefit of ten pounds per borrower in the form of fees and repossession costs that the rules would have prevented.392

Finally, the FSA estimated compliance costs for the new rules at between 47 and 170 million pounds per year, for an average of £109 million per year, based on a combination of its own survey of lenders, input from a consulting group (Oxera) that conducted its own surveys, and internal FSA data.393 Using the FSA’s discount rate of 3.5%,394 one can derive a present value of compliance costs of between £1.3 and £4.9 billion. The FSA did not explain how it was able to relate the per-borrower benefits it estimated from its main analyses to the per-year compliance costs it estimated. However, it did present a per-borrower compliance cost (£120 per borrower), which can be related to its aggregate average compliance cost estimate (£109 million per year), to derive a per-year benefit from the earlier analyses of £300 million per year. Using the FSA’s 3.5% discount rate, that annual amount has a present value of nine billion pounds. The bottom line implicit in the FSA’s analysis, then, is a total benefit (net of compliance costs) of six billion pounds.

Separately, the FSA used a macroeconomic (“NiGEM”)395 model to estimate effects of the rules on output. With many assumptions,396 the model predicted six categories of sequential monetary impacts.397 The long-run effects in the sixth category—increased output from increased business investment—more than outweighed categories (such as reduced home lending, home prices, and household consumption) that would reduce output in the short run. The net effect was estimated at over £300 million more per year of output.398 Using the FSA’s discount rate of 3.5%, the present value of this increase would be nine billion pounds,399 as much as the total direct benefits. Yet elsewhere, without explanation or detail, the CBA/FR stated it had not included output in its bottom-line summary of costs and benefits because “the margin of error inherent in the estimation of the macroeconomic impacts means that in reality this impact could either be positive or negative.”400

b. Assessing the FSA’s CBA/FR

Any assessment of the FSA’s CBA/FR should begin by acknowledging it is better as an academic exercise—more complex, detailed, and creative—than anything yet produced by any U.S. financial regulatory agency. It relies on academic working papers, several different datasets, and multiple modeling techniques, and tackles a host of difficult estimation problems. It actually attempts to quantify the benefits of a financial regulation—something that the rest of Part III shows is rarely done. If CBA/FR has a role to play in the United States, the FSA’s CBA/FR is a useful example of a path forward, just as CBA/FR advocates suggest by calling it the “gold standard.”

However, it should also be recognized that the FSA’s job here was by many measures easier than that faced in other regulatory contexts. The mortgage reforms were important and will have complex effects, but their importance and complexity pale beside those of more general regulations such as Basel III or the Volcker Rule. The mortgage reforms impose relatively light mandates on the process and terms of one class of consumer financial product—an important class, to be sure—but one that is considerably simpler than, for example, swaps or even common stock issued by a variety of public companies with a variety of governance arrangements and disclosure practices. A home mortgage is a loan, with clear and definite terms, and a limited set of straightforward purposes. Other important transactions have similar characteristics—consumer loans, credit card loans, student loans—and regulations of those markets are also likely to be more tractable for CBA/FR than the more complex regulations reviewed here.401

Despite being in a simpler regulatory context, a review of the FSA’s CBA/FR of its mortgage reforms nevertheless shows how fragile and unreliable the analysis remains, and how susceptible such CBA/FR is to being used as camouflage, rather than as discipline—particularly as it gets more complex and ambitious (as it will have to do to approach the goals that its advocates have for it). Below is a short list of weaknesses in the FSA’s CBA/FR that illustrate both its shortcomings and how it could just as easily mislead as inform the public.

First, the FSA is clear in its exposition that it used judgment in a number of crucial places. Examples include: (1) it created its own loan impairment model, where its staff effectively chose their own underwriting criteria, rather than relying on industry models, due to data limitations; (2) it chose where the new rules would begin to bind on lending decisions, using visual inspection of a figure rather than more quantitative methods; and (3) it chose how to “weight” the well-being results given the multiple comparisons it had with its data. Another important judgment the FSA made was to ignore the output of its macroeconomic modeling, as noted above, despite the fact that the net benefits on output of the rules were comparable to the direct benefits to borrowers. Each of these decisions, while defensible, required judgment.

Second, the FSA’s entire well-being analysis, which is its core method for estimating the effects of the rules, was usable only because of the happenstance that its output could be related to unemployment data. If the net effect on well-being had been significantly larger or smaller, this method would have been unavailable, and the FSA would have had to use another method to monetize the well-being effects, something that is—as the FSA noted—“notoriously problematic.”402 This difficulty calls into question the viability of this “gold standard” CBA as a model for the future.

Third, the FSA made a number of assumptions that affected its CBA: (a) it assumed that loans would not be made if they were reduced by thirty percent in size (an arbitrary figure) due to the new rules, but would be made otherwise;403 (b) it assumed that delayed loans would never be made;404 (c) it assumed that repossession had no effects on well-being distinct from default, because it had too few observations in its well-being dataset to estimate a different effect;405 (d) by using a fixed effects model to generate causal inferences about loan rules and well-being, it assumed that unobserved variation in individual respondents does not co-vary with home-related events;406 (e) it assumed that data from 2006 to 2011—a period of concededly low and falling interest rates—predicts future home market conditions;407 (f) it implicitly assumed that its modeling of the effects of Basel III were correct, but as discussed in Part III.C above, that is a fragile assumption; (g) it estimated compliance costs from a small survey (n=15, response rate 60%) of firms that would be subject to the new rules, resulting in potentially biased data;408 and (h) it assumed that the social cost of transfers represented by repossessions and resales of repossessed homes (as opposed to the transaction costs of those events, which it did estimate) was zero.409 Each of these assumptions is defensible as a matter of regulatory discretion, as each simplified the analysis or coped with data limits. Together, however, they demonstrate the lack of reliability or precision in the overall analysis.

Two other strong assumptions are nowhere discussed or explained in detail: that all “unaffordable” loans would produce only social losses, and that all “affordable” loans would produce only social gains. Both assumptions seem dubious. Some loans that turn out to be unaffordable represent gambles by borrowers that turn out badly, but which, ex ante, even on a fully informed basis, the borrowers would take again. The new rules will likely prevent those gambles, and while one can make good arguments in favor of preventing such gambling, at least some normative approaches to welfare analysis would treat preventing informed consumers from making knowing gambles as a welfare harm. Some loans on which borrowers never default are nevertheless the product of avoidable misunderstandings by borrowers, and others are the product of deception and fraud by lenders: the fact that a borrower chooses not to default on such a loan does not imply that the borrower would take it out again, were the borrower adequately evaluated and warned about the loan’s potential risks. Indeed, the FSA’s own data showed that many non-defaulting borrowers experienced high levels of stress and difficulty in making payments, suggesting that they may regret their loans. The new rules will likely reduce some of those loans, but none of the associated increase in well-being was counted in the FSA’s analysis. Nowhere does the FSA identify these possibilities in a clear manner, and the technical language in which it presents its well-being analysis may prevent many readers from even understanding the assumptions that have been made, much less appreciate what effect they have on the bottom line.

Finally, despite the relative merits of the substance, the FSA’s presentation is not a model of clarity or candor in other respects. The assumptions listed in the two paragraphs above are not collected in one place in the FSA’s paper, but are mentioned in scattered locations, or are not explicitly noted at all. The sensitivity of the bottom-line results of the CBA/FR to important assumptions is not made clear.410 For example, the FSA does show that parts of its analysis are sensitive to assumptions about future levels of lending activity. It does so by breaking its historical data into two sub-periods: the “boom” period of the 2000s, when the new rules would have affected between 1.7% and 10.5% of borrowers, and the “subdued” period after the collapse, when they would have affected no more than 0.4% of borrowers.411 Similarly, the FSA shows that in the subdued period, seven percent of borrowers who would have been affected by the reforms faced actual impairment in its historical data, while thirty percent would have faced impairments in the boom period.412 The FSA does not, however, translate this sensitivity into bottom-line effects on benefits (gross or net). It does not present sensitivities to most of the assumptions discussed in the prior two paragraphs, and because it does not translate per-borrower benefits from its well-being analysis into present values, it does not allow readers to compare those benefits with the possible range of macroeconomic effects of the rules.

In sum, it is not clear that this “gold standard” CBA/FR, while distinctly more ambitious and interesting than other examples of CBA/FR, was a net benefit to an assessment of the mortgage rules. The FSA’s analysis is thought-provoking and may represent a step on a path toward regulatory capacity to use CBA/FR to generate outputs that can help the public assess the value of regulations such as the mortgage rules. Nevertheless, the bottom line of the FSA’s CBA/FR depends on assumptions and limited data to such an extent that, with equally plausible assumptions or different data, it could have come out with a different sign or order of magnitude attached to it. The FSA does include a number of disclaimers precisely to this effect—writing that “certain data, for example on relevant households’ expenditure, are not available . . . [such that] this CBA has been unusually difficult to prepare [and led to a] wide margin of uncertainty around its results.”413 Elsewhere, the FSA notes that the analysis “is inherently highly uncertain,” with the result that “[t]o a significant extent . . . the decision on whether to proceed with the proposed rules has to be based on social and political judgments.”414 And further: “It is extremely difficult to identify exactly how the responsible lending requirements will change borrowing in the market or the likely scale of this. It requires some judgemental assumptions on the basis of imperfect evidence.”415 None of this would be apparent to anyone reading U.S. white papers advocating CBA/FR legal reform.416 Therefore, the FSA’s CBA of its mortgage rules came with some unquantifiable cost in increasing misunderstandings of what CBA/FR is capable of, while failing to improve the public’s ability to evaluate the merits of the rules or achieving any other obvious benefit. It is an example of why quantified CBA/FR should not be mandated, rather than an example of why it should be.

F. Summary of Case Studies

The substantive rules reviewed in the foregoing case studies are summarized in Table 4.

Table 4.

summary of case studies

As reflected in Table 4, the case studies range across representative regulatory instruments417: disclosure (SOX 404), governance(mutual fund rules), capital regulation (Basel III), and activity limits (Volcker Rule). The cross-border swaps rules cover a large number of regulatory instruments, including disclosure and capital requirements, but also rules requiring segregation, risk management, margin limits, and fair dealing. The mortgage reforms represent a final, important category of financial regulation—consumer protection, in the form of required process and constraints on contract terms. The rules’ benefits range across public goods pursued by financial regulation: more competition, fewer systemic crises and harmful conflicts of interest, and reduced levels of asymmetric information.418

Table 5.

challenges for possible or actual efforts at guesstimated cba/fr in case studies

Table 5 summarizes the conclusions of the case studies on the feasibility of quantitative CBA/FR. As can be seen, it shows that any substantial financial regulatory rules will face one or more of five serious challenges: (1) data limitations, (2) causal inference challenges, (3) the need to incorporate judgmental macroeconomic models, (4) the need to incorporate even more judgmental policy/political models, and (5) the need to make contestable, judgmental assumptions or modeling choices that have large effects on the outputs of the analysis. Not every challenge is as acute for every kind of rule—political/policy modeling is probably not a first-order component of an analysis of an anti-fraud or governance rule, for example. But all rules face data challenges and are highly sensitive to assumptions; all face causal inference challenges more severe in kind than the ones faced in many non-financial contexts (as discussed more in Part IV); and most require the analyst to embed (explicitly or not) a macroeconomic model of the same judgmental nature as that used in setting monetary policy.

The central conclusion of the case studies is that quantitative CBA/FR is not currently feasible with any degree of precision and reliability for representative types of financial regulation. Anything presented as quantified CBA/FR is in fact judgmental in nature, not an actual alternative to judgment but rather its equivalent in numerical form—“judgment in disguise.” Such quantitative CBA/FR as has been done is better understood as “guesstimated,” and has been presented without clear disclaimers and sensitivity analyses. As a result, it is more likely to mislead and camouflage than inform or discipline. The only kind of CBA that is currently feasible for representative types of financial regulation is conceptual CBA, augmented by limited elements of quantified evidence that will be more illustrative than disciplinary.

IV. what are the implications of these case studies?

The case studies in Part III suggest that the capacity of anyone—including financial regulatory agencies, OIRA, academic researchers, CBA/FR proponents, litigators, and courts—to conduct quantified CBA/FR with any real precision or confidence does not exist for important, representative types of financial regulation. This Part discusses the reasons for and implications of this conclusion.

A. Why Is Quantified CBA/FR So Unreliable?

A straightforward implication of the case studies is that efforts by the financial agencies at quantified CBA/FR will for the foreseeable future produce only guesstimation. Back-of-the-envelope guesses at ranges of magnitudes are currently feasible, but precise and reliable estimates are not. Too many variables are in play for any given rule, and too many contestable assumptions are required, for anyone producing or consuming guesstimated CBA/FR to have any confidence in any specific estimate of costs or benefits, even if expressed in ranges or bounds.419 While guesstimated CBA/FR can draw on social scientific disciplines, such as financial economics, and while the agencies themselves may reasonably attempt quantified CBA/FR on occasion as a way of helping analysts better understand the implications of a given regulation, quantified CBA/FR will not be replicable, reliable, or predictive.

CBA/FR should be understood not as science but as number-laden guesswork, and should be treated as such by the public, regulators, and courts. While guesstimation can be a legitimate part of decision making, as one input into a judgmental choice, it should not “guide” policy except in the loosest sense. Basing policy on specific quantitative outputs would simply be a poor exercise of judgment.

This conclusion—that quantitative CBA is not a good basis for setting policy—may contrast with practice in other regulatory domains, where quantitative CBA appears to be used in setting policy.420 Possibly the conclusions generated by these case studies might be generalizable to some non-financial domains. But it is worth considering whether there are features of CBA/FR that make it more difficult to perform effectively than CBA in other domains, at least when considering “typical” financial and non-financial regulations. While this topic warrants considerably more analysis than is provided in this Article, here are three tentative explanations for why CBA/FR is so hard, with the recognition that some of what follows may also characterize some non-financial domains, at least in part.421

1. Finance Is Central to the Economy

Part of the explanation for how far we are from reliable and precise quantified CBA/FR estimates is that finance is at the heart of the economy. Any change in regulation with a material impact on finance will have a material impact on the economy, and large and complex effects on welfare. Recall from Part III.E.2 that the FSA’s mortgage reforms—relatively simple consumer protection regulations on the surface—were conceptually identified as having multiple, complex effects on the macroeconomy.422 They would cut home lending, lower home prices, reduce consumer spending, increase consumer saving, reduce consumer borrowing, and increase business lending and investment. The FSA used one of a large family of materially different but respectable macroeconomic models to derive a positive net effect of $9 billion, which it then claimed it was ignoring as too unreliable. Many rules would have more complex effects.

Macroeconomic models that include finance are still highly contested. They are the stuff of newspaper op-eds and blogs as much as consensus models in academic journals.423 The ripple effects of financial regulation are too large and complex, relative to its direct effects, to allow for reliable predictions of net effects. As noted in Part I.D, this reason explains why even CBA proponents concede that monetary policy should remain unregulated by CBA laws. What advocates have not grasped, but Part III shows, is that important financial regulation is always likely to interact with the economy—perhaps not to the same extent as quantitative easing, but with enough impact to generate large (and uncertain) effects on economic growth.

By contrast, consider the Department of Transportation’s proposed rule to increase rear visibility in motor vehicles.424 While there were uncertainties associated with estimating the rule’s benefits—owing to the question of whether to value children differently than adults—and the costs—owing to the possibility that compliance costs might fall over time and to the appropriate discount rate to use in estimating future costs—estimating neither costs nor benefits required a macroeconomic model.425 Indeed, it is hard to imagine a financial regulation important enough to warrant significant CBA/FR costs that would be as simple to model as this rule. Yet this rule is typical of many non-financial regulations, which generate direct compliance costs and result in straightforward improvements in safety, with few knock-on systemic effects.

2. Finance Is Social and Political

A second reason why quantitative CBA/FR is hard is that the main units of variation and change in finance are not things, or even individuals, but groups of people—groups with not only economic but also social and political relations. Finance is about firms, corporations—groups of people coming together to form and fund a business—and financial markets—groups of people routinely trading intangibles. These features of finance can be contrasted with some non-financial domains, where objects of regulation are inanimate (for example, chemicals, rear-facing car cameras) and regulations are designed to achieve relatively simple ends (for example, changing the frequency and intensity of the use of identified chemicals, or requiring installation of cameras). While a chemical can interact with the environment in ways that are challenging to model and predict, those interactions are generally simpler than interactions of groups of humans. Every human possesses agency and interacts with others in non-linear, unpredictable ways. As stated by one theoretical physicist, “Computational approaches [to modeling] have been very useful in physics because the knowledge of microscopic laws constrains theoretical modeling in extremely controlled ways. This is almost never possible for socioeconomic systems.”426

Chemicals can also be subjected easily to randomly controlled experiments, but experiments are more difficult for humans and are frequently not feasible for groups. Because finance affects the economy, modeling policy also becomes necessary to quantify effects of financial regulation; finance is more routinely and powerfully political than chemistry. Part of evaluating the costs of a crisis, as Part III showed, requires predicting how governments will respond. No similar efforts are required for most typical non-financial regulations.427

3. Finance Is Non-Stationary

A third reason that may help explain why quantified CBA/FR is hard is that underlying regularities that enable quantification are commonly “non-stationary” in finance—more likely to change over time than in other domains. The proverbial “rocket science,” for example, uses relatively simple models of inert objects moving through space, with key inputs—such as the gravitational constant and gravitational acceleration428—that do not change.429 By contrast, most relationships in finance change through time, often rapidly. Consider the striking decline from 1978 to 1999 in the dividend payout ratio or the steady fall since 1930 in the ratio of directly to institutionally invested stocks in U.S. retail portfolios, both changes with large implications for the costs and benefits of many financial regulations.430

One reason for the greater degree of non-stationarity in finance is that finance is non-physical, such that technology shocks have larger and more unpredictable effects on optimal financial choices. This point is reflected in the case studies in Part III: new technologies of derivatives and securitization were significant causes of the last crisis,431 which gave rise to several of the rules reviewed. While technological progress affects all regulatory domains, physics, chemistry, and biology are more central to non-financial regulation than to financial regulation, and regularities uncovered in those disciplines have proven more durable than those found in finance. As summarized by the same physicist quoted above:

Nature has been there since ever, but it has taken centuries to develop a reasonable understanding of little parts of it. Many of the things which are traded nowadays in financial markets did not exist few decades ago, not to speak of internet communities. In addition, we face a situation in which the density and range of interactions are steadily increasing, thus making theoretical concepts based on effective non-interacting theories inadequate.432

No doubt there are other explanations for why quantitative CBA/FR is so unreliable; some have to do with historical unwillingness of the financial agencies to invest sufficiently in the task. No doubt, too, there are areas of non-financial regulation in which science is weak, and CBA there, too, cannot be reliably used as a strong guide for regulation. But the problems in financial regulation are real and likely to persist for the foreseeable future.

B. New CBA/FR Mandates Should Be Passed Only If CBA/FR Satisfies CBA

A second implication of the case studies in Part III is that new legal mandates for CBA/FR such as those reviewed in Part II are a bad idea, at least until CBA/FR can be shown to pass its own test—that is, to be likely to result in benefits that outweigh its costs. It is hard to understand how any CBA advocate could argue to the contrary. Instead, CBA/FR should be conducted only to the extent and in the manner the expert agencies choose, since they are in the best position to decide whether CBA/FR will be, in a given instance, likely to pass its own test. This conclusion is particularly true when it comes to quantified CBA, because of how unreliable quantified CBA/FR remains. CBA/FR law’s purpose—to discipline agencies and reduce agency costs—will not be furthered by forcing analyses that amount to no more than guesstimation and camouflage—again, “judgment in disguise.”

Conceptually, what would the benefits of CBA/FR be, given the conclusion of Part III? If CBA/FR were precise and reliable, it might generate the benefits of disciplining agencies, informing the public through increased transparency, and counteracting cognitive biases faced by the agencies. But CBA/FR’s benefits have been low, and are likely to remain low, for the reasons sketched in Part IV.A above: CBA/FR is by definition about finance—and finance is at the heart of the economy; is social and political; and is composed of non-stationary relationships that exhibit secular change. These features undermine the ability of science to precisely and reliably estimate the effects of financial regulations, even retrospectively. Whenever agencies face such sensitive and speculative forecasting abilities, quantified CBA is not capable of disciplining regulatory analysis, and it will generate low benefits.

The analysis is even worse for CBA/FR mandates when one focuses not only on the abstract benefits they might create but also on the marginalbenefits they might create—over the baseline of the status quo. Other constraints—the general goals of the agencies, the screening and socialization of the agency staff, and the political oversight of the agencies by Congress, through confirmations, budgets, hearings, and public criticism of the sort reviewed in Part II—will prevent new regulation or deregulation that is so extreme in generating costs without offsetting benefits that it could not be justified by the current art of guesstimated CBA. Within the range of plausible regulatory action set by those other constraints, the financial agencies retain too much discretion to select inputs and make assumptions in CBA/FR, meaning that numbers that emerge in any effort at quantification are unlikely to demonstrate whether a proposed change is net beneficial. Worse, the goal of disciplining agencies may be undermined if the result is to encourage agencies to use CBA/FR as camouflage—to hide discretionary judgments under impressive numbers.

As discussed more in Part IV.D below, CBA/FR remains a useful conceptual framework, quantified CBA/FR is a worthy long-term research goal, and attempts to quantify may advance the research needed to achieve reliable, precise estimates, making it a worthwhile project for agencies to pursue, in parallel with their other activities.433 But the current benefits of CBA/FR remain low, because its real effects remain far off in time; like any regulatory benefit, the benefits of these real effects should be discounted to present value. Moreover, CBA/FR will produce costs—resources consumed, regulatory delay, diffusion of regulatory focus, and potential decreases in regulatory transparency—particularly if regulatory agencies and any courts involved in reviewing agency action do not have good incentives to be honest about the limits and uncertainties of the results.

Empowering courts to review even conceptual CBA/FR policy analysis is likely to be a bad idea. Judicial review is not likely to generate any significant improvement in CBA/FR itself, as agencies will likely respond to the threat of such review by hiding, not exposing, the weaknesses in their analyses. Nothing produced by the back-and-forth between the SEC and the D.C. Circuit over the mutual fund rules reviewed in Parts II and III meaningfully advanced public understanding of the qualitative costs and benefits of requiring more independent fund boards; the compliance costs on which the Chamber of Commerce court focused were minor even by the lights of the Chamber of Commerce itself.434 The SEC’s cross-border swap CBA, reviewed in Part III.E, provides a clear picture of how little the threat of such review will accomplish, relative to what conceptual CBA voluntarily presented by an agency might do.

Mandating an open interagency process for CBA—such as requiring a financial agency to publish not only its CBA but also the views of OIRA on its CBA—will also worsen outcomes.435 The result will be a bigger record that will continue to be largely ignored by the public but used by litigators to pick at particular agency judgments as arbitrary and capricious under the APA. The benefits such a mandate might achieve can already be achieved if the financial agency sees the process as valuable, as evidenced by the voluntary cooperation between the CFTC and OIRA during the Dodd-Frank Act rollout.436 The cases reviewed in Part II show how aggressive some D.C. Circuit panels have been in using such review to overturn agency actions, particularly when an agency’s commissioners have been divided in making any regulatory change. Trebling the number of pages or components of CBA available for judicial second-guessing, and adding the possibility of interagency disagreement to the mix, will incite more interventions, with no clear benefit to anyone other than litigators.437

More extensive judicial review will have other pernicious consequences. Not only will agencies rationally use CBA/FR as camouflage, but they can also be expected to go to Congress to lobby for the establishment of rules through detailed congressional mandates, which will likely receive greater deference from courts than rules adopted pursuant to congressional delegations of discretion to achieve general goals.438 Both the litigation and the shift towards congressional mandates will produce a general slowdown, not just of regulation, but also of deregulation and regulatory reform, and will likely increase partisan polarization in and deterioration of public opinion of the very courts charged with that review.

The CBA of CBA just sketched is preliminary and incomplete. Completing a CBA of CBA would require evidence: quantitative studies of the degree to which CBA results in better regulations and more transparency in the regulatory process, as well as quantified estimates of the costs—delay, confusion, camouflage, partisanship—that CBA can introduce. Until evidence is developed to illuminate when CBA/FR passes its own test, we must rely on judgment, just as agencies must when they regulate. As currently informed by the poor results of judicially reviewed CBA/FR (discussed in Part II), it is hard to see how laws that give courts a greater role in second-guessing the choice of when to conduct CBA/FR, or the details of CBA/FR when it is used, could be judged a good idea.439

C. Existing CBA/FR Laws Are Little Better in Practice

A final implication of Part III is that existing interpretations of the APA and the financial agencies’ governing statutes should be restored to their state prior to Chamber of Commerce, to reduce the influence of concentrated interests through litigation and of politically partisan but unaccountable judges on regulatory outcomes. As shown in Part II, the D.C. Circuit’s new interpretations of the APA have permitted (some) panels to overturn regulatory changes on the ground that a court would conduct its guesstimated CBA differently than an agency’s guesstimated CBA/FR. As shown in Part III, the state of CBA/FR is such that one can reasonably argue that all guesstimated CBA/FR of major financial regulations inevitably will contain multiple arbitrary assumptions and judgments simply in order to allow for rough guesstimates to be made. A legal system that simultaneously requires arbitrary judgments by agencies, and then allows them to be overturned by a court for being arbitrary, depending on which panel of the D.C. Circuit is randomly (that is, arbitrarily) chosen, is self-evidently indefensible.440

Even if one agrees with a given court that a given rule represents bad policy (as I do with respect to the fund governance rules reviewed in Part III.B), better means exist for those affected by such rules to protect their interests, such as through the legislative process or by developing regulatory proposals to await a new set of regulators—who, after all, are more frequently replaced by politically accountable Presidents than are the judges on the D.C. Circuit. In sum, the current, erratically applied law of CBA/FR raises agency costs as between citizens and their political agents, rather than lowering them as CBA/FR is supposed to accomplish.

Often, the current state of the law on CBA/FR of financial regulation is perceived in simple partisan terms—Republican judges will strike down regulations adopted by regulators appointed by a Democratic President—and this is viewed as good by Republicans (and financial institutions) and bad by Democrats (and individual investors and bank customers). But in a few years the same unfortunate dynamic may reverse, with Democratic judges striking down deregulatory changes adopted by regulators appointed by a Republican President. Regardless of the current state of partisan power sharing, or of one’s political inclinations, it should require more theory and evidence than CBA/FR proponents have developed to leave financial regulation wrapped in the unlovely arms of litigators and the partisan lottery that is the D.C. Circuit.441

To remedy the situation, two recommendations made by Kraus and Raso for the SEC442 should be extended to all financial agencies. First, an exemption from the “sunshine” laws443 should be added to permit closed-door, pre-decisional discussions of CBA/FR among financial agency commissioners, between commissioners and the economic staffs of the agencies, among the agencies, and between the staffs of the agencies and the staffs of OIRA and the OFR. Until CBA/FR is considerably more developed, such deliberations are best conducted in a setting that encourages candor and creativity, rather than defensive camouflage and obfuscation in anticipation of litigation or requests under the Freedom of Information Act.444 Such a reform would likely increase the willingness of agencies to comply with existing requirements under the CBA Executive Orders445 that they submit CBA of their annual regulatory agendas to OIRA, requirements that have long been given short shrift by the financial agencies.446

Second, a “safe harbor” for CBA/FR should be added to the APA and the financial agencies’ governing statutes. The safe harbor can be modeled on the CRA,447 which courts have interpreted as barring judicial review of agency compliance with the statute, including agency determinations of whether a rule is “major.” As Kraus and Raso put it, “private litigants must not be allowed to throw [CBA/FR] back at the agency as ‘party admissions against interest,’ undermining the validity of the very rules that the analysis informed.”448 Anyone genuinely interested in fostering CBA/FR should recognize that, with the current, politicized D.C. Circuit only likely to become more polarized after the elimination of the filibuster, the absence of such a safe harbor may well lead agencies to be overly cautious, long-winded, and opaque in their CBA/FR—lawyerly virtues, not economic ones.

D. CBA/FR Remains a Potentially Valuable Component of Policy Analysis

A naïve response to the case studies in Part III would be to jettison CBA/FR altogether. If CBA cannot generate reliable, precise estimates of costs and benefits, one might conclude that it has no value, even as a discretionary component for policymaking. If CBA/FR cannot produce reliable quantification, then it has only costs and no benefits. This response would be a mistake for four reasons. First, it is possible that some financial regulations are susceptible of quantified CBA/FR. There may be some relatively simple financial regulations in which the costs and benefits will be more straightforward to estimate reliably, particularly if the regulations are implemented in a careful way and combined with a retrospective CBA/FR. The case studies in Part III are only a sampling of rules.

Second, conceptual CBA/FR remains the best available overarching framework for organizing and communicating the pros and cons of a proposed regulation. Conceptual CBA/FR is a commonsense way to begin the analysis necessary to evaluate a proposed rule by comparing it to the status quo and plausible alternatives. Indeed, it is hard to imagine conducting any sort of policy analysis without at least engaging in tacit conceptual CBA/FR. Organizing analysis in a conceptual CBA framework will provide some benefit for public understanding, even if the benefit is modest, and even if the negative effects of guesstimated camouflage can easily overwhelm that benefit.

Third, CBA may have effects other than the conventional set outlined in Part I.C (discipline, transparency, and camouflage). CBA guidelines, such as those in the OMB Guidance,449 also serve a brainstorming function, as a checklist to prompt analysts to be more creative in regulatory design and evaluation. Precisely because conceptual CBA is not an entrenched and exclusive piece of any one agency’s historical lore, evaluating regulatory proposals within a CBA framework can open up new channels of thought and nudge regulators beyond a baseless enthusiasm for tried but perhaps less helpful models of regulation. Conceptual CBA involves a common language and mode of thought that could facilitate interagency dialogue by floating above any one statutory mandate or set of agency-specific regulatory goals. Such dialogue can improve thinking about CBA-related problems (for example, how to phase in or randomize regulation so as to generate useful information while meeting legitimate expectations about equal treatment under the law).450 Thinking through conceptual CBA for a rule can lead to novel insights about how the rule is (or is not) similar to rules issued by other agencies, or how it might generate unintended consequences.

Fourth and most broadly, and with the greatest potential value, conceptual CBA/FR can facilitate improvements in quantified CBA/FR. Quantified CBA/FR, after all, would be highly valuable if it could generate precise and reliable estimates of the social costs and benefits of a regulatory change. Anything that promotes the long-term research agenda needed for reliable, precise quantitative CBA/FR has high potential value. To pursue that agenda, it would be useful for financial agencies to frame the questions that they face in evaluating regulations in terms of conceptual CBA, so as to stimulate and guide research. Research in economics, sociology, psychology, and other relevant fields proceeds along paths that are not random, but shaped by incentives, social cues, and psychological rewards. If agencies ask pointed research questions in their rulemaking proposals, they will encourage private researchers to answer those questions. Private actors with an interest in the answers may fund such research; tenure can be granted in part on the ground that an academic has answered a socially valuable question; and grant proposals are more likely to be funded if they relate to research topics that have direct potential value to regulatory agencies.

For conceptual CBA to be useful in this way, however, careful attention must be paid to institutional details, where the devil always lurks. Conceptual CBA/FR will not be useful in stimulating thought or guiding research if it consists of a simple, abstract list of the benefits and costs of a category of regulations. For example, it is correct in most instances for the SEC to include in the category of qualitative benefits “investor protection” and “investor confidence,” but it would be useless to leave things at that. How, precisely, does a rule improve confidence—through which channels? How does improved confidence constitute a social benefit—how does it affect the cost of capital? Nor will conceptual CBA/FR be useful if it consists of lengthy and opaque boilerplate circumlocutions designed to deflect or confuse judicial review rather than actually communicate to researchers or those who fund, evaluate, or publicize research.

A review of CBA conducted by the financial regulatory agencies demonstrates that fleshing out the benefits of financial regulation is a largely incomplete conceptual task, one that I hope the case studies in Part III will help advance. Similarly, indirect or systemic costs of regulation remain undeveloped. CBA/FR proponents have a strong point when they mock past CBA/FR efforts as exercises in “paperclip counting.”451 Those who are unhappy with the financial agencies are striving to promote quantified CBA through law in part because they rightly worry that regulatory practices that focus only on easily quantified subsets of costs in isolation will achieve little good.

The question, then, is how to encourage financial regulators to engage in meaningful, detailed conceptual CBA for its own sake—which should enhance public understanding and may also assist regulators themselves—but also because more and better conceptual CBA should stimulate research on quantitative CBA by making more apparent the key quantities to be estimated, and so by stimulating academics to think harder about research designs that would permit that quantification. How can lawmakers or law affirmatively encourage the use of conceptual CBA to stimulate thought and innovation? While a detailed set of proposals is beyond the scope of this Article,452 suffice it to say here that the challenge is primarily managerial, not methodological, a challenge not susceptible to simple legal commands or conventional judicial review. The challenge is not going to be met by specifying in meta-regulations methods to be used to conduct CBA/FR, but by using law and the lawmaking process to encourage expert agencies to better manage their resources and rulemaking processes in the short run—with the long-run goal of facilitating reliable, precise, quantified CBA/FR.

Conclusion

This Article has attempted to fill a significant gap in writing about CBA. It has shown how CBA/FR analysis would be conducted if—as advocated by some members of Congress, the D.C. Circuit, and legal academia—the law extended the current requirements that executive agencies engage in CBA to financial agencies, and required those agencies to produce as part of their rulemaking quantified CBA that could be subject to review under the requirements of the agencies’ authorizing statutes and the APA. Detailed case studies of six rules reveal that precise, reliable, quantified CBA remains unfeasible. Quantified CBA of such rules can be no more than “guesstimated,” and is not a true alternative to expert judgment—it is simply judgment in (numerical) disguise. As a result, for the near future, at least, judicial review of quantified CBA of financial regulation is not likely to generate benefits that exceed its costs. Until CBA/FR passes CBA’s own test, no new legal mandates should be adopted to require such review and more serious attention should be given to how to improve the capacities of the agencies to improve the reliability and precision of CBA in practice.