New Evaluation Guides Available from DOE and Action Plan

DOE and the EPA recently commissioned two important evaluation guides that were presented by their respective authors during a general session of the January program meeting.

Steve Schiller, an engineer with thirty years of energy industry experience, is the author of the Model Energy Efficiency Program Impact Evaluation Guide from the National Action Plan for Energy Efficiency, or Action Plan. The Action Plan was created by more than sixty organizations with a goal of creating a sustainable, aggressive national commitment to energy efficiency programs through gas and electric utilities, utility regulators, and partner organizations.

The Model Energy Efficiency Program Impact Evaluation Guide was the result of a year long national committee process that identified a need to foster best practices and promote consistent evaluation of programs. The guide shows how to calculate energy and demand savings and avoided emissions for resource acquisition-focused energy efficiency programs.

John H. Reed, PhD, of Innovologie, LLC, spoke about the Impact Evaluation Framework for Technology Deployment Programs, which he co-authored with Gretchen Jordan from Sandia National Laboratories and Ed Vine from Lawrence Berkeley National Laboratory. DOE’s Office of Energy Efficiency and Renewable Energy commissioned the framework document for energy efficiency programs to make it fast and easy to organize, design, and implement evaluations in accordance with the expectations for evaluations of federal programs.

The presentations were followed by a meeting of the Evaluation Committee that focused on comparing the Guide and Framework and exploring their relationship. The participants concluded that the Guide and Framework are complementary in their approaches and scope, and together provide a fairly comprehensive overview of impact evaluation for all types of program approaches—and that this conclusion needed to be communicated in some brief and pithy way to potential users of the two documents. Following the meeting, CEE Evaluation and Research Manager Monica Nevius worked with the Guide and Framework authors to develop the CEE Guide to the Evaluation Guides, which devotes three pages to pinpointing the relationship between the manuals and clarifying language.

The “Guide to the Guides” explains that the Impact Evaluation Framework for Technology Deployment Programs is based on the premise that “identifying the linkages between outputs and outcomes is one of the most critical and most difficult problems in program evaluation.” To help managers and evaluators address this problem, it provides specific tools to use in identifying the linkages between program activities or outputs and the resulting impacts or outcomes.

Identifying these linkages helps to clarify and prioritize what should be measured in the evaluation, enabling evaluators to apply with greater effectiveness the more technically oriented measurement and analysis tools presented in guides such as the Model Energy Efficiency Program Impact Evaluation Guide. The identification of outputs, outcomes, and the linkages among them also helps to separate program-induced impacts from the same effects that may be generated by other factors. This ability will become increasingly important as more players enter the field where ratepayer funded energy efficiency programs used to play alone, offering messages, programs, or services designed to reduce energy use for a variety of different reasons.

The Model Energy Efficiency Program Impact Evaluation Guide provides technical guidance for calculating energy and demand savings and avoided emissions from energy efficiency programs via a set of practical processes and methodologies. It focuses on evaluation for program approaches that rely primarily on direct energy savings. The Guide lays out clearly the steps in selecting the appropriate measurement and analysis approach for the program and evaluation goals. This includes, but may not be limited to, the use of billing analysis, deemed savings, and project or facility-level data collection, monitoring and analysis (M&V). It also provides important context and background for implementing the International Performance Measurement and Verification Protocol (IPMVP) as part of evaluation. Basic approaches to limited market effects measurement in impact evaluation for the calculation of net savings are provided. For more extensive treatment of market effects evaluation and for the evaluation of programs relying mostly or exclusively on indirect effects, such as market transformation programs and education or training programs, the Guide refers users to the Impact Evaluation Framework for Technology Deployment Programs.

All programs are carried out in a larger structural context—that is, in the real world of society, the economy, and markets. Off the drawing board and in the real world, many factors unrelated to the program and beyond the control of program administrators can affect program outcomes. The technical guidance offered in the Model Energy Efficiency Program Impact Evaluation Guide is likely to result in a more robust evaluation when implemented in the context of a clear understanding of the linkages between program activities and outcomes as well as the other factors that could affect energy savings, demand savings, and other potential program outcomes. Whenever possible, program managers and evaluators should strive to develop and communicate a clear understanding of these factors and linkages as part of program and evaluation planning.

The “Guide to the Guides” is available on the CEE web site.

[ back to top ]