 |
Evaluation/Research Guide to Evaluation Resources |
|
 |
|
|
Energy
Efficiency Program Evaluation: A Guide to the Guides Download
this document as a printable Adobe Acrobat pdf file. Table of Contents There are
several documents available that explain energy efficiency program evaluation
and provide guidance on how to conduct impact evaluations of efficiency
programs in the United States. The
goal of this "Guide to the Guides" is to clarify the relationships among the
documents. It focuses on documents
with a national scope, particularly the DOE/EERE's Impact
Evaluation Framework for Technology Deployment Programs and the National Action Plan for
Energy Efficiency's Model
Energy Efficiency Program Impact Evaluation Guide, two important national-level
documents about energy efficiency program impact evaluation that were released
in 2007. This "Guide to the Guides" also identifies other evaluation guides
written for particular regions or states that may have broader implications,
and other useful documents and resources related to evaluation. The Impact
Evaluation Framework for Technology Deployment Programs and the Model Energy Efficiency
Program Impact Evaluation Guide are complementary in their approaches and scope, and
together provide a fairly comprehensive overview of impact evaluation for all
types of program approaches. The Impact
Evaluation Framework for Technology Deployment Programs uses the term "technology
deployment programs" instead of "energy efficiency programs." This can be confusing to anyone
focusing on the administration or evaluation of ratepayer-funded efficiency
programs. This term is used
because the document is meant to provide guidance on evaluating clean energy programs
and renewables programs as well as efficiency programs. For efficiency program evaluation
purposes, however, this language can simply be interpreted as "energy
efficiency programs." The Impact
Evaluation Framework for Technology Deployment Programs is based on the premise that
"identifying the linkages between outputs and outcomes"—that is, how what
the program does (its activities or "outputs") are translated by partners and
target audiences into actions that produce a variety of impacts or "outcomes,"
including but not limited to energy and demand impacts—"is one of the
most critical and most difficult problems in program evaluation."[1] To help managers and evaluators address
this problem, it provides specific tools to use in identifying the linkages
between program activities or outputs and the resulting impacts or outcomes. Identifying these linkages helps to clarify and prioritize what should
be measured in the evaluation, thus enabling evaluators to apply with greater
effectiveness the more technically oriented measurement and analysis tools
presented in guides such as the Model Energy Efficiency
Program Impact Evaluation Guide. The
identification of outputs, outcomes, and the linkages among them also helps to
separate program-induced impacts from the same effects that may be generated by
other factors. The ability to
separate program-induced impacts from other factors will become increasingly
important as more players enter the field in which ratepayer funded energy
efficiency programs used to play alone, offering messages, programs, or
services designed to reduce energy use for a variety of different reasons. The Model Energy Efficiency
Program Impact Evaluation Guide provides technical guidance for calculating energy and
demand savings and avoided emissions from energy efficiency programs via a set
of practical processes and methodologies.
It focuses on evaluation specifically for program approaches relying
primarily on direct energy savings.
It lays out clearly the steps involved in selecting the appropriate
measurement and analysis approach for the program and evaluation goals. This includes, but may not be limited
to, the use of billing analysis, deemed savings, and project- or facility-level
data collection, monitoring and analysis (M&V). It also provides important context and background for
implementing the International Performance Measurement and Verification
Protocol (IPMVP) as part of evaluation.
It provides some basic approaches to including limited market effects
measurement in impact evaluation for the calculation of net savings. It refers users to the Impact
Evaluation Framework for Technology Deployment Programs for more extensive treatment of
market effects evaluation, and for the evaluation of programs relying mostly or
exclusively on indirect effects, such as market transformation programs and
education or training programs. All
programs are carried out in a larger structural context—that is, in the
real world of society, the economy, and markets. Off the drawing board and in
the real world, many factors unrelated to the program and beyond the control of
program administrators can affect program outcomes. The technical guidance offered in Model Energy Efficiency
Program Impact Evaluation Guide is likely to result in a more robust evaluation when
implemented in the context of a clear understanding of the linkages between
program activities and outcomes as well as the other factors that could affect
energy savings, demand savings, and other potential program outcomes. Whenever possible, program managers and
evaluators should strive to develop and communicate a clear understanding of
these factors and linkages as part of program and evaluation planning. International Performance Measurement and Verification
Protocol (IPMVP) The IPMVP,
a product of the Efficiency Valuation
Organization, is a set of framework documents used to develop strategies
and plans for quantifying energy and water savings at the project level—that is, in individual
facilities or groups of facilities—for retrofits and new
construction. This document is
referred to Model Energy-Efficiency Program Impact Evaluation Guide described above, which explains the
relevance of the IPMVP to efficiency programs and when to use each of its
parts. The IPMVP describes appropriate approaches to selecting facilities for
measurement and verification (M&V); measuring and verifying equipment
installation and usage; monitoring indoor environmental quality under different
circumstances; and addressing how to quantify avoided emissions from facilities
and projects. National Action Plan for Energy Efficiency The National
Action Plan for Energy Efficiency is a joint effort of the U.S.
Environmental Protection Agency, U.S. Department of Energy, and more than 80
energy, environmental, and other organizations. Its goal to create a sustainable, aggressive national commitment
to energy efficiency through gas and electric utilities, utility regulators,
and partner organizations. Action Plan participants are identifying key
barriers limiting greater U.S. investment in energy efficiency, and developing
and documenting sound business practices for removing these barriers. The
Action Plan's initial
report identifies key barriers to limiting greater investment in energy
efficiency and reviews sound business practices for removing these barriers,
detailing findings and outlining recommendations and options for overcoming
them. In addition to the Model Energy Efficiency
Program Impact Evaluation Guide discussed above, the Action Plan is producing a series of
documents to help guide regulators and program administrators in planning for
and evaluating energy efficiency as a resource. These include two documents useful for planning
energy-efficiency portfolios and programs, the Guide for
Conducting Energy-Efficiency Potential Studies and Guide to
Resource Planning with Energy Efficiency. For a
comprehensive list of Action Plan guides and papers, go to http://www.epa.gov/cleanenergy/energy-programs/napee/resources/guides.html. U.S. DOE Evaluation Publications In addition to the Impact
Evaluation Framework for Technology Deployment Programs described above, there is a twelve
page overview and an application example designed to help in learning how to
implement the approach described in this document. Also on the DOE website are: *
Overview
of Evaluation Methods for R&D Programs. This booklet introduces managers to
a variety of methods for evaluating R&D programs. Related to this are A Toolkit for
Evaluating Public R&D Investment Models, Methods, and Findings from ATP's
First Decade and a patent-searching database. *
EERE
Guide for Managing General Program Evaluation Studies. This Guide focuses on the management
and use of general program evaluation studies performed by outside experts and
contractors. In addition to outcome and impact evaluation, the guide addresses
process, cost-benefit evaluation, and market assessment evaluation. *
EERE Peer
Review Guide. The
peer review guide describes steps to plan, design, and implement external peer
reviews. California
Evaluation Framework (2004) The 2004
California Evaluation Framework provides a comprehensive set of guidelines for conducting
evaluations of California's energy efficiency programs. The framework includes
recommendations for conducting impact evaluations as well as process, market
effects, information/education/training program and non-energy benefits
evaluations. It describes evaluation methodologies and presents guidelines for
evaluation sample design and statistical analysis and for assessing and
reducing the level of uncertainty of evaluation results. The framework includes
a set of decision protocols for deciding what to evaluate and when to conduct
evaluations. California Evaluation Protocols
(2006) The Evaluation
Protocols are the official document that evaluation professionals are to
rely upon when planning and conducting their evaluation efforts for California
programs. Ontario Power Authority
Evaluation Framework & Protocols The Ontario
Power Authority (OPA) is in the process of finalizing an evaluation
framework, protocols, and other related guidance documents for conducting
energy efficiency program evaluation of OPA programs. The evaluation framework
is to provide the background for a consistent and systematic methodology for
tracking, reporting and evaluating the impacts and effectiveness of
Conservation programs funded by the OPA. New England Forward Capacity
Market M&V Manual As part of the newly formed wholesale capacity market in New
England, or Forward Capacity Market (FCM), demand resources, including energy
efficiency, load management, DG, real-time demand response, and real-time
emergency generation, can compete alongside supply resources in the annual
auctions. The New England
Independent System Operator (ISO-NE) has developed an M&V
Manual for demand resources which sets forth the standards that demand
resource providers must meet in order to qualify their resources and receive
capacity payments during the commitment period. California Demand Response
Evaluation The California Public Utilities Commission has posted a
document that is expected to form the basis of the state's protocols on load
impact estimation for demand response.
This document is helpful in outlining issues regarding the evaluation of
demand response programs, and also discusses forecasting the effects of demand
response for program planning. It
should be available for downloading in September. Check the proceedings section of
the California Public Utilities Commission's website for this
document. (Refer to Rulemaking
07-01-041.) CEE's Evaluation Web Pages CEE's evaluation
web pages include a wealth of materials to support the evaluation of
publicly funded energy efficiency programs and help new program administrators
and evaluators to understand and keep up with the evolving field of energy
efficiency program evaluation.
After February 27, 2008, these will include materials supporting CEE's
evaluation webinar, such as descriptions of categories and types of evaluation,
examples of program portfolios and program logic models, and a downloadable
recording of CEE's evaluation webinar.
Other materials currently available include links to a calendar of evaluation training
opportunities, links to organizations
that offer evaluation training on-line or on a regular basis, results of
seven years of CEE's annual survey of household
awareness of the ENERGY STAR label, and an annual state-by-state summary
of energy-efficiency budgets and a regional summary of savings impacts from
CEE members. This "Guide to the Guides" is also available on CEE's evaluation
web pages. California Evaluation Framework
(2001) The predecessor to the current
(2004) California Evaluation Framework, California's 2001 Framework for Planning and
Assessing Publicly-Funded Energy Efficiency, provides a thorough overview of
energy-efficiency program evaluation theory and approaches. It includes a
glossary of energy-efficiency evaluation terms and useful explanations of
concepts and measurement challenges that are rarely treated elsewhere. This and
the 2004 Framework are excellent background and reference documents for both
novice and experienced evaluators.
EPA's Clean
Energy-Environment Guide to Action The Clean Energy-Environment Guide to
Action identifies
and describes 16 clean energy policies and strategies that states have used to
meet their clean energy objectives. It describes how states are successfully
expanding the role of clean energy in the U.S. energy system and shares the
experience and lessons learned from successful state clean energy policies. Best Practices Benchmarking for Energy Efficiency
Programs The Energy
Efficiency Best Practices Project seeks to identify and communicate
effective energy-efficiency practices nationwide to enhance the design,
implementation, and evaluation of energy-efficiency programs. The Web site
contains a database of best practices reports and individual program summary
reports. Emerging Technologies Coordinating Council The Emerging Technologies
Coordinating Council is an organization run by California utilities to
assess promising new technologies for California's energy customers. This
website contains a database of current and recent new technologies projects
undertaken by the council. Energy Information Administration (EIA) The Energy Information
Administration (EIA) generates an annual survey of electric utilities. The
EIA Web site contains the collected data
from form EIA-861, which includes the following information: peak load,
generation, electric purchases, sales, revenues, customer counts and
demand-side management programs. Annual results are available for download. CEE Evaluation Clearinghouse CEE's Market
Assessment & Program Evaluation (MAPE) Clearinghouse is a database of energy-efficiency
program evaluation reports, potential studies, and related documents. The
documents are all publicly available and most can be downloaded directly from
the CEE Web site as PDF files. Most are supplied by CEE members. It also offers
full-text searching of IEPEC conference proceedings. While
the Clearinghouse was designed for the use of CEE's membership, in the interest
of improving program evaluation across the entire energy-efficiency industry,
CEE and its members make the Clearinghouse available to the public. California Measurement Advisory
Council (CALMAC) CALMAC provides a forum for the development, implementation,
presentation, discussion, and review of regional and statewide market
assessment and evaluation (MA&E) studies for California energy efficiency
programs conducted using Public Goods Charge funds. CALMAC hosts a large
searchable database of downloadable evaluation reports from the state's
three IOUs and other energy efficiency program administrators. International Energy Program
Evaluation Conference (IEPEC) IEPEC is a biennial professional conference for energy
program implementers, evaluators of those programs, federal and state agency
representatives, and academic researchers. The purpose of the conference is to provide a forum for the
presentation, critique and discussion of objective evaluations of energy
programs. IEPEC is next scheduled
for August 2009. Abstracts of
IEPEC documents can be searched on-line at
IEPEC's website or via CEE's
Clearinghouse. Proceedings
are available for purchase on the IEPEC website. American Council for an
Energy-Efficient Economy (ACEEE) ACEEE is a nonprofit organization dedicated to advancing
energy efficiency as a means of promoting both economic prosperity and
environmental protection. They
host two bi-annual conferences that include evaluation-related papers, and
conduct and publish their own research.
A list of
publications—many available for free downloading—and conference
proceedings, and information about their conferences, are available
on-line. American Journal of Evaluation The American Journal of
Evaluation is an interdisciplinary social science magazine which covers
diverse topics within the field of evaluation. This website offers both current
and archived articles for subscribers. The New York State Energy Research and Development
Authority (NYSERDA) The NYSERDA Web
site contains information
about the organization's evaluation approach and reporting. Users can
access the full text of annual and quarterly evaluation reports filed with the
New York State Public Service Commission over the past five years, as well as
other special reports such as Cost-Effectiveness and Low-Income Program
evaluation results. Also available is a listing of 100-plus detailed reports
completed by NYSERDA's evaluation contractors in recent years that can be
obtained by request. Northeast Energy Efficiency
Partnerships NEEP is a nonprofit
organization that conducts policy and program research as well as evaluation
related to energy-efficiency activities from Maine to New Jersey. The Web site
includes links to a clearinghouse of energy-efficiency data for the Northeast,
updates on developments in energy-efficiency policy, results of regional
protocols research, and information on regional initiatives. The Northwest Energy Efficiency
Alliance (NEEA) NEEA is a nonprofit
corporation supported by Bonneville Power Administration, electric utilities,
public benefits administrators, state governments, public interest groups and
energy-efficiency industry representatives. Its mission is to catalyze the
Northwest marketplace to embrace energy-efficient products and services.
Publications, including evaluation reports and market research reports pertaining
to the Pacific Northwest, are available on the Research+Reports
page of the NEEA Web site. Energy Trust of Oregon The Energy Trust of
Oregon is a public-purpose organization dedicated to energy efficiency and
renewable energy generation. They
host an on-line library
as well as a list of
evaluation reports. Wisconsin Department of
Administration The following items are available on the Wisconsin
Department of Administration Web site: FAQs, facts about Wisconsin, fact
sheets, document library (forms, publications and other working documents) and
listing of other on-line resources related to Wisconsin program evaluations.
Also available are evaluation
reports of Focus on Energy, the statewide residential efficiency program. The Energy Center of Wisconsin, a private, non-profit
organization dedicated to improving energy sustainability including support of
energy efficiency, renewable energy, and environmental protection, also hosts
an extensive assortment of evaluation reports in their library. Database for Energy Efficient Resources (DEER) The Database for
Energy Efficient Resources (DEER) is sponsored by the California Energy
Commission and California Public Utilities Commission (CPUC). It is designed to
provide well-documented estimates of energy and peak demand savings values,
measure costs, and effective useful life (EUL) all with one data source. DEER
has been has been designated by the CPUC as its source for deemed and impact
costs for program planning. California Residential Appliance Saturation Survey (RASS) The California
Residential Appliance Saturation Survey (RASS) is a research product
containing statewide and utility specific results, including data on all
appliances, equipment and general usage habits. All of the survey data is
available on the Web site. U.S. DOE's Online Search Tool for
Patents For use with R&D evaluation. Search patents
from 1940-present. This document was developed with input from the Steve
Schiller of Schiller Consulting, author of the Model Energy Efficiency
Program Impact Evaluation Guide, and John Reed of Innovologie and Ed Vine of Lawrence
Berkeley National Laboratory, two co-authors of the Impact Evaluation
Framework for Technology Deployment Programs. [1] Reed, J., Jordan, G. and E. Vine. 2007.
Impact Eval Framework for Technology Deployment Programs. U.S. Department of Energy, Office of Energy
Efficiency and Renewable Energy. Page iv.
|
© 2000-2008, Consortium for Energy Efficiency, Inc.
http://www.cee1.org
98 North Washington St., Suite 101, Boston MA 02114-1918
617-589-3949
fax 617-589-3948
All rights reserved. Terms and conditions for use of this Web site.
Send comments about this site to our Webmaster