81 FR 87949 - HUD Program Evaluation Policy-Policy Statement

DEPARTMENT OF HOUSING AND URBAN DEVELOPMENT

Federal Register Volume 81, Issue 234 (December 6, 2016)

Page Range		87949-87951
FR Document		2016-29215

This policy statement of HUD's Office of Policy Development and Research articulates the core principles and practices of the office's evaluation and research activities. This policy reconfirms the Department's commitment to conducting rigorous, relevant evaluations and to using evidence from evaluations to inform policy and practice.

Federal Register, Volume 81 Issue 234 (Tuesday, December 6, 2016)

[Federal Register Volume 81, Number 234 (Tuesday, December 6, 2016)]
[Notices]
[Pages 87949-87951]
From the Federal Register Online  [www.thefederalregister.org]
[FR Doc No: 2016-29215]


-----------------------------------------------------------------------

DEPARTMENT OF HOUSING AND URBAN DEVELOPMENT

[Docket No. FR-5985-N-01]


HUD Program Evaluation Policy--Policy Statement

AGENCY: Office of the Assistant Secretary for Policy Development and 
Research, HUD.

ACTION: Notice.

-----------------------------------------------------------------------

SUMMARY: This policy statement of HUD's Office of Policy Development 
and Research articulates the core principles and practices of the 
office's evaluation and research activities. This policy reconfirms the 
Department's commitment to conducting rigorous, relevant evaluations 
and to using evidence from evaluations to inform policy and practice.

DATES: December 6, 2016.

FOR FURTHER INFORMATION CONTACT: Mark D. Shroder, Associate Deputy 
Assistant Secretary, Office of Research, Evaluation, and Monitoring, 
Office of Policy Development and Research, U.S. Department of Housing 
and Urban Development, 451 7th Street SW., Washington, DC 20410, 
telephone (202) 402-5922. The listed telephone number is not a toll-
free number. Persons with hearing- or speech-impairments may access 
this number through TTY by calling Federal Relay Service at 1-800-877-
8339 (this is a toll-free number).

SUPPLEMENTARY INFORMATION:

I. Background

    The mission of HUD's Office of Policy Development and Research 
(PD&R) is to inform HUD policy development and implementation to 
improve life in American communities through conducting, supporting, 
and sharing research, surveys, demonstrations,

[[Page 87950]]

program evaluations, and best practices. Within HUD, PD&R is 
responsible for nearly all program evaluations. The office provides 
reliable and objective data and analysis to help inform policy 
decisions. Program evaluation has been a core activity of PD&R since 
its formation in 1974.
    In July 2016, the Government Accountability Office (GAO) issued a 
report entitled ``Department of Housing and Urban Development: Actions 
Needed to Incorporate Key Practices into Management Functions and 
Program Oversight,'' (GAO 16-497) in which GAO presented a broad 
assessment of HUD's management of its operations and programs.\1\ In 
the report, GAO examined HUD efforts to: (1) Meet Federal requirements 
and implement key practices for management functions, including 
performance planning and reporting, human capital, financial, 
acquisition, and information technology (IT) management; and (2) 
oversee and evaluate programs.
---------------------------------------------------------------------------

    \1\ See http://www.gao.gov/assets/680/678551.pdf.
---------------------------------------------------------------------------

    PD&R is the primary office within HUD responsible for data 
analysis, research, program evaluations, and policy studies that inform 
the development and implementation of programs and policies across HUD 
offices. PD&R undertakes program evaluations, often by using a process 
that includes convening expert panels. However, GAO found that PD&R had 
not developed agency-wide, written policies for its program 
evaluations, nor documented the criteria used to select the expert 
panels and review the quality of program evaluations.
    This policy statement responds to the GAO report by setting out the 
core principles and practices of PD&R's evaluation and research 
activities. This statement incorporates some language from a policy 
statement by the Office of Policy, Research, and Evaluation of the 
Administration for Children and Families of the U.S. Department of 
Health and Human Services.

II. HUD Program Evaluation Policy

    PD&R has identified the following core principles and practices as 
fundamental to ensuring high-quality and consistent evaluation results: 
rigor, relevance, transparency, independence, ethics, and technical 
innovation. This policy applies to all PD&R-sponsored evaluations and 
economic analyses of regulations; they apply as well to the selection 
of projects, contractors, and PD&R staff that is involved in 
evaluations.

Rigor

    PD&R is committed to using the most rigorous methods that are 
appropriate to the evaluation questions and feasible within budget and 
other constraints. Rigor is not restricted to impact evaluations, but 
is also necessary in implementation or process evaluations, descriptive 
studies, outcome evaluations, and formative evaluations; and in both 
qualitative and quantitative approaches. Rigor requires ensuring that 
inferences about cause and effect are well founded (internal validity); 
requires clarity about the populations, settings, or circumstances to 
which results can be generalized (external validity); and requires the 
use of measures that accurately capture the intended information 
(measurement reliability and validity).
    In assessing the effects of programs or services, PD&R evaluations 
use methods that isolate to the greatest extent possible the impacts of 
the programs or services from other influences such as trends over 
time, geographic variation, or pre-existing differences between 
participants and non-participants. For such causal questions, 
experimental approaches are preferred. When experimental approaches are 
not feasible, PD&R uses the most rigorous approach that is feasible. 
PD&R ensures that contractors and grantees conducting evaluations have 
appropriate expertise through emphasizing the capacity for rigor in 
requests for proposal and funding opportunity announcements.
    PD&R also employs a strategic human capital development plan to 
hire, train, and retain a workforce that ensures the staff has the 
tools and resources to accomplish the mission.

Relevance

    The PD&R evaluation agenda reflects the legislative requirements 
and policy issues related to HUD's mission. PD&R solicits input from 
stakeholders, both internal and external, on the selection of programs 
to be evaluated, initiatives, demonstrations, and research questions. 
For new initiatives and demonstrations in particular, evaluations will 
be more feasible and useful when planned in advance, in concert with 
the development of the initiative or demonstration, rather than as an 
afterthought.
    PD&R disseminates findings in ways that are accessible and useful 
to policy-makers and practitioners. PD&R partners with other HUD 
program offices to inform internal and external stakeholders through 
disseminating evidence from PD&R-sponsored evaluations.

Transparency

    PD&R will release methodologically valid evaluations without regard 
to the findings. Evaluation reports must describe the methods used, 
including strengths and weaknesses, and discuss the generalizability of 
the findings. Evaluation reports must present comprehensive results, 
including favorable, unfavorable, and null findings.
    PD&R publishes a 5-year Research Roadmap that outlines the research 
and evaluation that we believe would be of greatest value to public 
policy. PD&R lists all ongoing evaluation projects at the HUDUSER.gov 
Web site, and updates it monthly. PD&R will release evaluation results 
timely, usually within 4 months of receiving the final report.
    PD&R will, where possible, archive evaluation data for secondary 
use by interested researchers. PD&R typically builds requirements into 
contracts to prepare data sets for secondary use.

Independence

    Independence and objectivity are core principles of evaluation. 
Agency and program leadership, program staff, service providers, and 
others participate actively in setting evaluation priorities, 
identifying evaluation questions, and assessing the implications of 
findings. However, it is important to insulate evaluation functions 
from undue influence and from both the appearance and the reality of 
bias. To promote objectivity, PD&R protects independence in the design, 
conduct, and analysis of evaluations. To this end:
     PD&R conducts evaluations through the competitive award of 
grants and contracts to external experts who are free from conflicts of 
interest.
     PD&R also conducts evaluations in-house and supports 
unsolicited external evaluation proposals with funding, data, or both.
     The Assistant Secretary for PD&R has authority to approve 
the design of evaluation projects and analysis plans; and has authority 
to approve, release, and disseminate evaluation reports. The Assistant 
Secretary does so, in consultation with career staff.

Ethics

    PD&R-sponsored evaluations must be conducted in an ethical manner 
and safeguard the dignity, rights, safety, and privacy of participants. 
PD&R-sponsored evaluations must comply with both the spirit and the 
letter of relevant requirements such as regulations governing research 
involving human subjects. In particular, PD&R protects the privacy of 
HUD-assisted households

[[Page 87951]]

and HUD-insured borrowers through the Rule of Eleven; that is, PD&R 
allows no disclosure of information about the characteristics of any 
group of individuals or households numbering less than eleven by PD&R 
staff, contractors, grantees, or licensees.

Technical Innovation

    PD&R supports and employs new methods of data collection and 
analysis that more reliably and efficiently answer research questions 
than old methods do.

Application of These Principles to Economic Analysis of Regulations

    Economic analysis of regulations, properly conducted, is a critical 
tool in improving public policy. In any PD&R Regulatory Impact 
Analysis:
     PD&R analyzes whether the issues addressed by the 
regulation stem from a market failure, government failure, or other 
systemic problem, and whether the regulation addresses the root causes 
of those problems.
     PD&R uses and as necessary produces the best objective 
estimates of the benefits, costs, and transfers resulting from the 
regulation, taking into account gaps and uncertainties in the available 
data.
     Where clear alternatives to the regulatory actions exist, 
PD&R objectively estimates the benefits, costs, and transfers of those 
alternatives as well.

    Dated: November 30, 2016.
Katherine O'Regan,
Assistant Secretary for Policy Development and Research.
[FR Doc. 2016-29215 Filed 12-5-16; 8:45 am]
 BILLING CODE 4210-67-P

87950 Federal Register / Vol. 81, No. 234 / Tuesday, December 6, 2016 / Notices

program evaluations, and best practices. Rigor Transparency
Within HUD, PD&R is responsible for PD&R is committed to using the most PD&R will release methodologically
nearly all program evaluations. The rigorous methods that are appropriate to valid evaluations without regard to the
office provides reliable and objective the evaluation questions and feasible findings. Evaluation reports must
data and analysis to help inform policy within budget and other constraints. describe the methods used, including
decisions. Program evaluation has been Rigor is not restricted to impact strengths and weaknesses, and discuss
a core activity of PD&R since its evaluations, but is also necessary in the generalizability of the findings.
formation in 1974. implementation or process evaluations, Evaluation reports must present
In July 2016, the Government descriptive studies, outcome comprehensive results, including
Accountability Office (GAO) issued a evaluations, and formative evaluations; favorable, unfavorable, and null
report entitled ‘‘Department of Housing and in both qualitative and quantitative findings.
and Urban Development: Actions approaches. Rigor requires ensuring that PD&R publishes a 5-year Research
Needed to Incorporate Key Practices inferences about cause and effect are Roadmap that outlines the research and
into Management Functions and well founded (internal validity); evaluation that we believe would be of
Program Oversight,’’ (GAO 16–497) in requires clarity about the populations, greatest value to public policy. PD&R
which GAO presented a broad settings, or circumstances to which lists all ongoing evaluation projects at
assessment of HUD’s management of its results can be generalized (external the HUDUSER.gov Web site, and
operations and programs.1 In the report, validity); and requires the use of updates it monthly. PD&R will release
GAO examined HUD efforts to: (1) Meet measures that accurately capture the evaluation results timely, usually within
Federal requirements and implement intended information (measurement 4 months of receiving the final report.
key practices for management functions, reliability and validity). PD&R will, where possible, archive
including performance planning and In assessing the effects of programs or evaluation data for secondary use by
reporting, human capital, financial, services, PD&R evaluations use methods interested researchers. PD&R typically
acquisition, and information technology that isolate to the greatest extent builds requirements into contracts to
(IT) management; and (2) oversee and possible the impacts of the programs or prepare data sets for secondary use.
evaluate programs. services from other influences such as
Independence
PD&R is the primary office within trends over time, geographic variation,
HUD responsible for data analysis, or pre-existing differences between Independence and objectivity are core
research, program evaluations, and participants and non-participants. For principles of evaluation. Agency and
policy studies that inform the such causal questions, experimental program leadership, program staff,
development and implementation of approaches are preferred. When service providers, and others participate
programs and policies across HUD experimental approaches are not actively in setting evaluation priorities,
offices. PD&R undertakes program feasible, PD&R uses the most rigorous identifying evaluation questions, and
evaluations, often by using a process approach that is feasible. PD&R ensures assessing the implications of findings.
that includes convening expert panels. that contractors and grantees conducting However, it is important to insulate
However, GAO found that PD&R had evaluations have appropriate expertise evaluation functions from undue
not developed agency-wide, written through emphasizing the capacity for influence and from both the appearance
policies for its program evaluations, nor rigor in requests for proposal and and the reality of bias. To promote
documented the criteria used to select funding opportunity announcements. objectivity, PD&R protects
the expert panels and review the quality PD&R also employs a strategic human independence in the design, conduct,
of program evaluations. capital development plan to hire, train, and analysis of evaluations. To this end:
and retain a workforce that ensures the • PD&R conducts evaluations through
This policy statement responds to the staff has the tools and resources to the competitive award of grants and
GAO report by setting out the core accomplish the mission. contracts to external experts who are
principles and practices of PD&R’s free from conflicts of interest.
evaluation and research activities. This Relevance
• PD&R also conducts evaluations in-
statement incorporates some language The PD&R evaluation agenda reflects house and supports unsolicited external
from a policy statement by the Office of the legislative requirements and policy evaluation proposals with funding, data,
Policy, Research, and Evaluation of the issues related to HUD’s mission. PD&R or both.
Administration for Children and solicits input from stakeholders, both • The Assistant Secretary for PD&R
Families of the U.S. Department of internal and external, on the selection of has authority to approve the design of
Health and Human Services. programs to be evaluated, initiatives, evaluation projects and analysis plans;
II. HUD Program Evaluation Policy demonstrations, and research questions. and has authority to approve, release,
For new initiatives and demonstrations and disseminate evaluation reports. The
PD&R has identified the following in particular, evaluations will be more Assistant Secretary does so, in
core principles and practices as feasible and useful when planned in consultation with career staff.
fundamental to ensuring high-quality advance, in concert with the
and consistent evaluation results: rigor, development of the initiative or Ethics
relevance, transparency, independence, demonstration, rather than as an PD&R-sponsored evaluations must be
ethics, and technical innovation. This afterthought. conducted in an ethical manner and
policy applies to all PD&R-sponsored PD&R disseminates findings in ways safeguard the dignity, rights, safety, and
mstockstill on DSK3G9T082PROD with NOTICES

evaluations and economic analyses of that are accessible and useful to policy- privacy of participants. PD&R-sponsored
regulations; they apply as well to the makers and practitioners. PD&R partners evaluations must comply with both the
selection of projects, contractors, and with other HUD program offices to spirit and the letter of relevant
PD&R staff that is involved in inform internal and external requirements such as regulations
evaluations. stakeholders through disseminating governing research involving human
evidence from PD&R-sponsored subjects. In particular, PD&R protects
1 See http://www.gao.gov/assets/680/678551.pdf. evaluations. the privacy of HUD-assisted households

VerDate Sep<11>2014 17:39 Dec 05, 2016 Jkt 241001 PO 00000 Frm 00049 Fmt 4703 Sfmt 4703 E:\FR\FM\06DEN1.SGM 06DEN1

Federal Register / Vol. 81, No. 234 / Tuesday, December 6, 2016 / Notices 87951

and HUD-insured borrowers through the State of California. We now make the jurisdiction. A SAR must be based on
Rule of Eleven; that is, PD&R allows no draft SAR available for public review the best scientific information available;
disclosure of information about the and comment. therefore, we prepare it in consultation
characteristics of any group of DATES: We will consider comments that with established regional scientific
individuals or households numbering are received or postmarked on or before review groups. Each SAR must include:
less than eleven by PD&R staff, March 6, 2017. 1. A description of the stock and its
contractors, grantees, or licensees. ADDRESSES: If you wish to review the geographic range;
Technical Innovation draft revised SAR for southern sea otter, 2. A minimum population estimate,
you may obtain a copy from our Web current and maximum net productivity
PD&R supports and employs new site at http://www.fws.gov/ventura.
methods of data collection and analysis rate, and current population trend;
Alternatively, you may contact the
that more reliably and efficiently answer 3. An estimate of the annual human-
Ventura Fish and Wildlife Office, 2493
research questions than old methods do. caused mortality and serious injury by
Portola Road, Suite B, Ventura, CA
source and, for a strategic stock, other
Application of These Principles to 93003 (telephone: 805–644–1766). If
factors that may be causing a decline or
Economic Analysis of Regulations you wish to comment on the SAR, you
impeding recovery;
may submit your comments in writing
Economic analysis of regulations, 4. A description of commercial fishery
by any one of the following methods:
properly conducted, is a critical tool in • U.S. mail: Field Supervisor, at the interactions;
improving public policy. In any PD&R above address;
Regulatory Impact Analysis: 5. A categorization of the status of the
• Hand delivery: Ventura Fish and stock; and
• PD&R analyzes whether the issues Wildlife Office at the above address;
addressed by the regulation stem from a 6. An estimate of the potential
• Fax: 805–644–3958; or
market failure, government failure, or • Email: fw8ssostock@fws.gov. biological removal (PBR) level.
other systemic problem, and whether The MMPA defines the PBR as ‘‘the
FOR FURTHER INFORMATION CONTACT:
the regulation addresses the root causes maximum number of animals, not
Lilian Carswell, at the above street
of those problems. including natural mortalities, that may
address, by telephone (805–612–2793),
• PD&R uses and as necessary be removed from a marine mammal
or by email (Lilian_Carswell@fws.gov).
produces the best objective estimates of stock while allowing that stock to reach
SUPPLEMENTARY INFORMATION: We
the benefits, costs, and transfers or maintain its OSP’’ (16 U.S.C.
resulting from the regulation, taking into announce the availability for review and
comment of a draft revised marine 1362(20)). The PBR is the product of the
account gaps and uncertainties in the minimum population estimate of the
available data. mammal stock assessment report (SAR)
for the southern sea otter (Enhydra lutris stock (Nmin); one-half the maximum
• Where clear alternatives to the theoretical or estimated net productivity
regulatory actions exist, PD&R nereis) stock in the State of California.
rate of the stock at a small population
objectively estimates the benefits, costs, Background size (Rmax); and a recovery factor (Fr) of
and transfers of those alternatives as between 0.1 and 1.0, which is intended
Under the Marine Mammal Protection
well. to compensate for uncertainty and
Act of 1972, as amended (MMPA; 16
Dated: November 30, 2016. U.S.C. 1361 et seq.), and its unknown estimation errors. This can be
Katherine O’Regan, implementing regulations in the Code of written as:
Assistant Secretary for Policy Development Federal Regulations (CFR) at 50 CFR PBR = (Nmin)(1⁄2 of the Rmax)(Fr)
and Research. part 18, we regulate the taking; import;
[FR Doc. 2016–29215 Filed 12–5–16; 8:45 am] and, under certain conditions, Section 117 of the MMPA also
possession; transportation; purchasing; requires the Service and NMFS to
BILLING CODE 4210–67–P
selling; and offering for sale, purchase, review the SARs (a) at least annually for
or export, of marine mammals. One of stocks that are specified as strategic
the MMPA’s goals is to ensure that stocks, (b) at least annually for stocks for
DEPARTMENT OF THE INTERIOR which significant new information is
stocks of marine mammals occurring in
waters under U.S. jurisdiction do not available, and (c) at least once every 3
Fish and Wildlife Service
experience a level of human-caused years for all other stocks. If our review
[FWS–R8–ES–2016–N127; FF08EVEN00– mortality and serious injury that is of the status of a stock indicates that it
FXFR1337088SSO0] likely to cause the stock to be reduced has changed or may be more accurately
below its optimum sustainable determined, then the SAR must be
Marine Mammal Protection Act; Stock revised accordingly.
Assessment Report for the Southern population level (OSP). OSP is defined
under the MMPA as ‘‘the number of A strategic stock is defined in the
Sea Otter in California
animals which will result in the MMPA as a marine mammal stock ‘‘(a)
AGENCY: Fish and Wildlife Service, maximum productivity of the for which the level of direct human-
Interior. population or the species, keeping in caused mortality exceeds the PBR level;
ACTION: Notice of availability; request mind the carrying capacity of the habitat (b) which, based on the best available
for comments. and the health of the ecosystem of scientific information, is declining and
which they form a constituent element’’ is likely to be listed as a threatened
SUMMARY: In accordance with the (16 U.S.C. 1362(9)). species under the Endangered Species
mstockstill on DSK3G9T082PROD with NOTICES

Marine Mammal Protection Act of 1972, To help accomplish the goal of Act of 1973, as amended (16 U.S.C. 1531
as amended, and its implementing maintaining marine mammal stocks at et seq.) [the ‘‘ESA’’], within the
regulations, we, the U.S. Fish and their OSPs, section 117 of the MMPA foreseeable future; or (c) which is listed
Wildlife Service (Service), have requires the Service and the National as a threatened or endangered species
developed a draft revised marine Marine Fisheries Service (NMFS) to under the ESA, or is designated as
mammal stock assessment report (SAR) prepare a SAR for each marine mammal depleted under [the MMPA].’’ 16 U.S.C.
for the southern sea otter stock in the stock that occurs in waters under U.S. 1362(19).

VerDate Sep<11>2014 17:39 Dec 05, 2016 Jkt 241001 PO 00000 Frm 00050 Fmt 4703 Sfmt 4703 E:\FR\FM\06DEN1.SGM 06DEN1

Document Created: 2016-12-06 02:18:31
Document Modified: 2016-12-06 02:18:31

Category		Regulatory Information
Collection		Federal Register
sudoc Class		AE 2.7: GS 4.107: AE 2.106:
Publisher		Office of the Federal Register, National Archives and Records Administration
Section		Notices
Action		Notice.
Dates		December 6, 2016.
Contact		Mark D. Shroder, Associate Deputy Assistant Secretary, Office of Research, Evaluation, and Monitoring, Office of Policy Development and Research, U.S. Department of Housing and Urban Development, 451 7th Street SW., Washington, DC 20410, telephone (202) 402-5922. The listed telephone number is not a toll- free number. Persons with hearing- or speech-impairments may access this number through TTY by calling Federal Relay Service at 1-800-877- 8339 (this is a toll-free number).
FR Citation		81 FR 87949