80 FR 38716 - National Registry of Evidence-Based Programs and Practices

DEPARTMENT OF HEALTH AND HUMAN SERVICES
Substance Abuse and Mental Health Services Administration

Federal Register Volume 80, Issue 129 (July 7, 2015)

The mission of SAMHSA is to reduce the impact of substance abuse and mental illness on America's communities. Established in 1992, the agency was directed by Congress to target effective substance abuse and mental health services to the people most in need, and to translate research in these areas more effectively and more rapidly into the general health care system. NREPP is a key public resource SAMHSA has developed to help meet this directive. This notice announces the redesign of NREPP to better align the registry with the standards and processes of other evidence-based repositories. A re-launch of the Web site with revised content is anticipated in late fall/early winter 2015. The notice explains the changes in how programs and practices will be identified for NREPP, how submissions will be screened and reviewed, and provides guidance on accessing updated information on the NREPP site. Potential applicants should be aware that this notice includes updated information relating to the eligibility of interventions for inclusion in NREPP and changes in the program and practice review process that supersedes guidance provided in earlier Federal Register notices.

[Federal Register Volume 80, Number 129 (Tuesday, July 7, 2015)]
[Notices]
[Pages 38716-38718]
From the Federal Register Online  [www.thefederalregister.org]
[FR Doc No: 2015-16573]


-----------------------------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES

Substance Abuse and Mental Health Services Administration


National Registry of Evidence-Based Programs and Practices

AGENCY: Substance Abuse and Mental Health Services Administration 
(SAMHSA), HHS.

ACTION: Notice regarding SAMHSA's NREPP: Redesign of NREPP.

-----------------------------------------------------------------------

SUMMARY: The mission of SAMHSA is to reduce the impact of substance 
abuse and mental illness on America's communities. Established in 1992, 
the agency was directed by Congress to target effective substance abuse 
and mental health services to the people most in need, and to translate 
research in these areas more effectively and more rapidly into the 
general health care system. NREPP is a key public resource SAMHSA has 
developed to help meet this directive. This notice announces the 
redesign of NREPP to better align the registry with the standards and 
processes of other evidence-based repositories. A re-launch of the Web 
site with revised content is anticipated in late fall/early winter 
2015.
    The notice explains the changes in how programs and practices will 
be identified for NREPP, how submissions will be screened and reviewed, 
and provides guidance on accessing updated information on the NREPP 
site. Potential applicants should be aware that this notice includes 
updated information relating to the eligibility of interventions for 
inclusion in NREPP and changes in the program and practice review 
process that supersedes guidance provided in earlier Federal Register 
notices.

FOR FURTHER INFORMATION CONTACT: Carter Roeber, Ph.D., Social Science 
Analyst, Center for Behavioral Health Statistics and Quality, SAMHSA, 1 
Choke Cherry Road, Room 2-1050, Rockville, MD 20857, telephone 240-276-
1488.

SUPPLEMENTARY INFORMATION: 

Advancing Evidence-Based Programs and Practices Through Improved 
Decision Support Tools: Reconceptualizing NREPP

Introduction

    SAMHSA's NREPP is an evidence-based repository and review system 
designed to provide the public with reliable information about 
behavioral health interventions in the areas of mental health and 
substance abuse. Programs and practices that are accepted for inclusion 
in NREPP undergo a review process that provides information on the 
quality of research and the magnitude and direction of program or 
practice impact on individual outcomes. Materials for dissemination are 
reviewed to determine the type and extent of information available to 
support implementation. The results of these reviews are published on 
the NREPP Web site (http://nrepp.samhsa.gov).
    It should be noted that inclusion in NREPP indicates that some, but 
not necessarily all, of the evidence for a program or practice has been 
reviewed. In some cases, the quality of the research supporting the 
program or practice may have been determined to be poor or insufficient 
to earn a rating. Inclusion in NREPP does not constitute endorsement of 
an intervention as effective by SAMHSA. Moreover, since NREPP has not 
reviewed all behavioral health interventions, the use of NREPP as an 
exclusive or exhaustive list of interventions is not appropriate. 
Policymakers and funders in particular are discouraged from limiting 
providers and/or potential grantees to selecting exclusively from among 
NREPP interventions and from funding NREPP interventions regardless of 
the ratings the interventions receive.
    This notice announces changes to (1) the process for identifying 
new programs and practices for NREPP review, (2) the process for 
announcing open submission periods, (3) the minimum requirements to be 
considered for NREPP review, and (4) the review process. This notice 
also announces the intent to re-review currently posted NREPP programs 
and practices to comport with new review criteria and ratings. The re-
review of programs and practices currently posted will take place over 
the course of the next several years, depending on available resources. 
A re-launch of the NREPP Web site will take place in phases and the 
first phase is planned for late fall/early winter of 2015.

Identifying New Programs and Practices for NREPP

    Open submissions periods, during which applicants may submit 
materials for review, will continue to be used to identify new programs 
and practices for review (see below). Programs and practices addressing 
specific SAMHSA priorities may also be identified by SAMHSA or through 
environmental scans (including literature reviews, focus groups, public 
input, and interviews), as time and resources permit. Programs and 
practices related to priority areas may be reviewed before programs and 
practices identified through the open submission period. SAMHSA will be 
consulting with subject matter experts and leadership in underserved 
groups and populations, including American Indian/Alaska Native Tribes, 
regarding ways to incorporate traditional and culturally-specific 
interventions into NREPP, in order to better meet the needs of groups 
whose efforts to promote behavioral health may not have not been 
routinely evaluated. Innovative, but perhaps less rigorously tested, 
programs and

[[Page 38717]]

practices will be considered if specific to SAMHSA priority areas, as a 
parallel contribution to the standard review procedures.
    Stand-alone pharmacologic treatments are not eligible for review 
and should not be submitted to NREPP. The evidence base for 
pharmacologic treatments is reviewed and approved through the U.S. Food 
and Drug Administration (FDA). FDA approved pharmacotherapy 
interventions (on-label use) are considered for NREPP review only when 
combined with one or more behavioral or psychosocial treatments.
    SAMHSA reserves the right to reject for review programs and 
practices whose goals or activities are determined to be inconsistent 
with SAMHSA's mission, which is ``to reduce the impact of substance 
abuse and mental illness on American communities.''

Open Submission Periods

    SAMHSA accepts new applications for review during open submission 
periods. SAMHSA generally holds one open submission period a year but, 
depending on the number of reviews in progress and resources available 
may hold more or fewer within a calendar year. All future open 
submission periods will be announced on the NREPP Web site. Emails will 
also be sent announcing the open submission period to those on the 
NREPP listserv. Anyone wishing to be notified of future open submission 
periods can join the NREPP listserv by sending a request to 
[email protected].
    Applications can be submitted at any point during an open period. 
Program and practice developers, researchers, and others interested in 
submitting an intervention should read below for information about the 
new minimum requirements to be considered for an NREPP review. 
Additional future changes to the review process and criteria will be 
posted on the NREPP Web site as they are implemented. Therefore, before 
submitting a program or practice for NREPP review, applicants should 
examine the most recent information posted on the NREPP Web site about 
the review process and criteria and the most recent guidance for 
preparing an intervention for submission (see http://www.nrepp.samhsa.gov/ReviewSubmission.aspx). This guidance will be 
periodically updated to reflect the redesign of NREPP during the re-
launch.
    The selection of interventions will take place after the closing of 
the open submission period, and applicants will be notified whether 
they have been accepted after an initial screening to ensure that the 
application meets minimum requirements. The number of reviews conducted 
will depend on the availability of funds, with the timing of reviews to 
be determined by SAMHSA. In submitting an intervention, applicants 
should understand that if interventions are selected for review, the 
results of NREPP reviews are considered public information and will be 
posted on the NREPP Web site. Once a review is completed, the applicant 
will be provided with a summary document (``the program or practice 
profile'') that presents results of the review, ratings of program 
effectiveness, and descriptive information about the intervention. The 
applicant will have the opportunity to comment on the profile before it 
is posted but they will not have the option to refuse posting.

Minimum Requirements

    To be considered for review, interventions must meet three minimum 
requirements:
    1. Research or evaluation of the intervention has assessed mental 
health or substance use outcomes among individuals, communities, or 
populations OR other behavioral health-related outcomes on individuals, 
communities, or populations with or at risk of mental health issues or 
substance use problems.
    2. Evidence of these outcomes has been demonstrated in at least one 
study using an experimental or quasi-experimental design. Experimental 
designs require random assignment, a control or comparison group, and 
pre- and post-intervention outcome assessments. Quasi-experimental 
designs do not require random assignment, but do require a comparison 
or control group and pre- and post-intervention outcome assessments. 
Comparison/control groups must be a no-treatment control group, a wait-
list control group, a treatment-as-usual comparison group, or an 
intervention that is presumed to be ineffective or substantially less 
effective than the intervention (e.g., a ``placebo'' control or, in 
cases in which providing no treatment might be considered unethical, 
less effective treatments, even if not treatment-as-usual, such as 
``supportive therapy''). Studies with single-group, pretest-posttest 
designs or single-group, longitudinal/multiple time series do not meet 
this requirement, but will be considered to identify emerging programs 
and practices for consideration in the Learning Center. Comparative 
effectiveness trials, in which two interventions, both presumed to be 
equally effective, are compared, and studies in which the effects of 
the same intervention on various subpopulations are compared or in 
which various doses or components of the same intervention are compared 
will be reviewed, but only for purposes of providing readers in the 
Learning Center the opportunity to determine whether a particular 
program or practice may have use or show promise in different settings 
or circumstances.
    3. The results of these studies have been published in a peer-
reviewed journal or other professional publication, or documented in a 
comprehensive evaluation report, published within the previous 25 
years. Comprehensive evaluation reports must include a review of the 
literature, theoretical framework, purpose, methodology, findings/
results with statistical analysis and p values for significant 
outcomes, discussion, and conclusions.
    Applicants are required to provide full-text documents at the time 
of submission that demonstrate the intervention meets these minimum 
requirements. Other research articles, published or unpublished 
evaluation reports, grant final reports, and replication studies may be 
submitted as additional supporting documentation. Note: Abstracts or 
links to partial articles are regarded as incomplete and will not be 
considered.
    NREPP will no longer require programs and practices to have 
developed implementation materials, training and support resources, and 
quality assurance procedures. However, programs and practices with such 
dissemination and/or implementation resources will be considered for 
prioritized review if within the priorities established by the SAMHSA 
review process, and the materials, along with the location of their 
availability, will be listed in the program or practice profile.
    Applicants submitting dissemination and implementation resources 
should include a brief narrative description of the materials that are 
being submitted. These materials may include, but are not limited to, 
treatment manuals, information for administrators, information for 
direct service staff, tested training curricula, mechanisms for ongoing 
supervision and consultation, protocols for gathering process and 
outcome data, ongoing monitoring of intervention fidelity, and 
processes for gathering feedback. Applicants should also provide the 
location of where the materials can be obtained.

[[Page 38718]]

Selection of Interventions for Review

    SAMHSA will select interventions for review from among submissions 
meeting the minimum requirements. In selecting interventions for 
review, SAMHSA may give special consideration to interventions that 
meet one or more of the following conditions:
     More than one research study or evaluation has been 
conducted on the same or a similar target population that meets the 
minimum requirements.
     The intervention targets underserved populations (e.g., 
minority populations, tribal communities or American Indian/Alaska 
Native populations, elderly individuals, young adults, individuals who 
are incarcerated, etc.).
     Dissemination and implementation materials (e.g., program 
or practice manuals, training guides, measurement instruments, 
implementation fidelity tools) are available. Lower costs and no-cost 
materials may be prioritized.
     The intervention contributes to a content area in which 
few evidence-based interventions have been previously identified.
    Interventions that are not selected for review may be resubmitted 
by the applicant in a future open submission period.

The Review Process

    The review process has been revised to improve the quality of the 
reviews and utility of information that NREPP can provide its users. In 
addition to articles and reports submitted by NREPP applicants, 
additional studies, articles, and evaluation reports regarding the 
interventions will be identified through literature searches. Studies 
and outcomes to be reviewed will be determined through the systematic 
application of standardized screening criteria, and the number of 
studies and outcomes to be reviewed will be expanded to more 
comprehensively represent the evidence base for the program or 
practice. Inclusion of studies and outcomes will no longer be limited 
to positive significant outcomes; all studies and outcomes that meet 
the standardized screening criteria will be reviewed, including those 
with negative and non-significant effects. Programs and practices will 
be assessed on the basis of evaluation studies of program or practice 
impact, information related to conceptual framework (that is, program 
or practice goals, theory of change, and program or practice 
components), and information about implementation fidelity (that is, 
whether a study employs quality assurance measures to declare that the 
program or practice is delivered as intended to the program's or 
practice's target population).
    The methodological rigor (that is, internal validity, statistical 
validity, and measurement validity) of the research for each program or 
practice will be reviewed, as it pertains to each outcome examined, 
along with the magnitude and direction of the program's or practice's 
effect on each outcome. Based on this information, the program's or 
practice's effectiveness for each outcome will be rated, along with the 
rigor of the research examining the program or practice, and the 
ratings will be displayed on the NREPP Web site.
    In general, each NREPP evidence review will be conducted by two 
trained and certified reviewers. However, based on funding and 
available resources, SAMHSA use one reviewer for programs and practices 
being re-reviewed. When necessary, NREPP may conduct author queries to 
confirm or gather additional information needed for the review. Program 
and practice profiles will be developed on the basis of the information 
gathered. Applicants will have the opportunity to review the program or 
practice profile before it is posted on the NREPP site, but they will 
not have the option to refuse posting.
    Dissemination and implementation materials will no longer be rated 
as they were historically. Instead, descriptions of available materials 
for each program or practice, highlighting information that may be of 
most interest to NREPP users, will be included in the program or 
practice profile, along with information documenting the extent to 
which materials are available.
    Programs and practices currently posted on NREPP will be re-
reviewed as time and resources permit but the re-reviews of currently 
posted programs and practices will take place over the next few years.
    Detailed information about the revised review process will be 
available at http://www.nrepp.samhsa.gov after the re-launch of the new 
NREPP Web site.

Enhancing the Learning Center

    NREPP's Learning Center is a developing and underutilized component 
of the NREPP Web site. With the evolution and enhancement of the 
registry, SAMHSA seeks to bring greater recognition to both rigorously 
evaluated behavioral health interventions and those interventions that 
have been implemented, demonstrate promise, but have not necessarily 
been evaluated in a rigorous manner. To that end, the Learning Center 
is being significantly revamped to support stakeholder engagement and 
to become a shared learning environment for all stakeholders. SAMHSA 
recognizes that the successful promotion and dissemination of evidence-
based programs and practices requires an environment that promotes 
community assessment, program and practice planning and evaluation, as 
well as guidance on the selection and implementation of programs and 
practices listed on NREPP. There are useful types of evaluation 
research, often conducted among underserved populations, which provide 
valuable insights for practitioners, but do not meet the minimum 
criteria required for experimental or quasi-experimental design. SAMHSA 
intends the Learning Center to be a forum for presenting research on 
emerging programs and practices, and exploring ways that pre-
experimental and qualitative research can complement and enrich 
findings from experimental and quasi-experimental research designs. An 
inventory of such programs and practices will be compiled and 
maintained within the Learning Center and will operate in parallel to 
the listing of reviewed programs and practices with experimental and 
quasi-experimental designs. In this way, SAMHSA intends to support 
programs and practices researched with the most rigorous approaches 
while also supporting the development of practice-based evidence, 
especially for certain populations and emerging practices that are 
critical to learning and improving behavioral health outcomes for 
persons with or at risk of developing behavioral health issues.

Summer King,
Statistician.
[FR Doc. 2015-16573 Filed 7-6-15; 8:45 am]
BILLING CODE 4162-20-P