81 FR 2876 - Announcement of Requirements and Registration for “Pill Image Recognition Challenge”

DEPARTMENT OF HEALTH AND HUMAN SERVICES
National Institutes of Health

Federal Register Volume 81, Issue 11 (January 19, 2016)

The Pill Image Recognition Challenge is a National Institutes of Health (NIH) Challenge under the America COMPETES (Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science) Reauthorization Act of 2010 (Pub. L. 111-358). Through this Challenge, the National Library of Medicine (NLM), part of NIH, seeks algorithms and software to match images of prescription oral solid-dose pharmaceutical medications (pills, including capsules and tablets). The objective of the Challenge is the development and discovery of high-quality algorithms and software that rank how well consumer images of prescription pills match reference images of pills in the authoritative NLM RxIMAGE database. NLM may use all or part of any Challenge entry (i.e., algorithm and software) to create a future software system and a future API (Application Programming Interface) for pill image recognition; the system will be freely usable and the API will be freely accessible.

[Federal Register Volume 81, Number 11 (Tuesday, January 19, 2016)]
[Notices]
[Pages 2876-2879]
From the Federal Register Online  [www.thefederalregister.org]
[FR Doc No: 2016-00777]


-----------------------------------------------------------------------

DEPARTMENT OF HEALTH AND HUMAN SERVICES

National Institutes of Health


Announcement of Requirements and Registration for ``Pill Image 
Recognition Challenge''

    Authority: 15 U.S.C. 3719

SUMMARY: The Pill Image Recognition Challenge is a National Institutes 
of Health (NIH) Challenge under the America COMPETES (Creating 
Opportunities to Meaningfully Promote Excellence in Technology, 
Education, and Science) Reauthorization Act of 2010 (Pub. L. 111-358). 
Through this Challenge, the National Library of Medicine (NLM), part of 
NIH, seeks algorithms and software to match images of prescription oral 
solid-dose pharmaceutical medications (pills, including capsules and 
tablets). The objective of the Challenge is the development and 
discovery of high-quality algorithms and software that rank how well 
consumer images of prescription pills match reference images of pills 
in the authoritative NLM RxIMAGE database. NLM may use all or part of 
any Challenge entry (i.e., algorithm and software) to create a future 
software system and a future API (Application Programming Interface) 
for pill image recognition; the system will be freely usable and the 
API will be freely accessible.

DATES: NLM will make a set of consumer-quality images and a companion 
set of reference images publicly available on January 15, 2016.
    The Challenge begins January 19, 2016.
    Submission period: April 4, 2016 to May 31, 2016.
    Judging period: June 6, 2016 to July 15, 2016.
    Winners announced: August 1, 2016.
    Submissions received by NLM after the submission period ends will 
not be considered. A submission is considered to meet the submission 
deadline if it is received by May 31, 2016, 5:00 p.m. EDT. While NLM 
plans to acknowledge receipt of each Challenge submission, the 
Government is under no obligation to acknowledge receipt of the 
information received or provide feedback to respondents with respect to 
any information submitted. NLM will amend this Federal Register notice 
if the timeline or the rules for the Challenge are modified. In 
addition, NLM will notify registered Challenge participants by email of 
any amendments and will include the modified Challenge showing the 
changes.

ADDRESSES: Notifications of any amendment to this Federal Register 
notice and answers to frequently asked questions about it will be 
posted at http://pir.nlm.nih.gov/challenge/notifications-and-FAQs. 
Submissions must be mailed to: Pill Image Recognition Challenge, 
Computational Photography Project for Pill Identification (C3PI), 
National Library of Medicine, Building 38A, Room B1-N30, 8600 Rockville 
Pike, Bethesda, MD 20894.

FOR FURTHER INFORMATION CONTACT: Michael J. Ackerman, Ph.D. at (301) 
402-4100 or [email protected].

SUPPLEMENTARY INFORMATION: 

The IC's Statutory Authority To Conduct the Challenge

    What has become today's National Library of Medicine began in 1836 
as a small collection of medical books and journals in the office of 
the U.S. Army Surgeon General. A 1956 act of Congress (Pub. L. 84-941) 
transferred the library to the Public Health Service and gave it its 
current name. That law authorizes NLM to ``assist the advancement of 
medical and related sciences and to aid the dissemination and exchange 
of scientific and other information important to the progress of 
medicine and to the public health'' and to ``promote the use of 
computers and telecommunications by health professionals (including 
health professionals in rural areas) for the purpose of improving 
access to biomedical information for health care delivery and medical 
research.'' In addition to its subject-matter authority, NLM is 
conducting this competition under the America COMPETES Reauthorization 
Act of 2010 (Pub. L. 111-358).

Subject of Challenge

    Unidentified and misidentified prescription pills present 
challenges for patients and professionals. Unidentified pills can be 
found by family members, health professionals, educators, and law 
enforcement. The nine out of 10 U.S. citizens over age 65 who take more 
than one prescription pill can be prone to misidentifying those pills. 
Taking such pills can result in adverse drug events that affect health 
or cause death. To reduce such errors, any person should easily be able 
to confirm that a prescription pill or a refill is correct. For 
example, a person should be able to easily verify--or not--that a 
refill that has a different color, shape, or text imprinted on the pill 
is a different generic version of equivalent drugs he or she was 
already taking.
    To help address these problems, the NLM Computational Photography 
Project for Pill Identification (C3PI) is developing infrastructure and 
tools for identifying prescription pills. The infrastructure includes 
photographs of such pills taken under laboratory lighting conditions, 
from a camera directly above the front and the back faces of the pill, 
and at high resolution. Specialized digital macro-photography 
techniques were then used to capture JPEG pill images. The NLM RxIMAGE 
database contains these high-quality images and associated pill data 
such as appearance (color, shape, size, text imprinted on the pill, 
etc.), ingredients, and identifiers such as its National Drug Code 
(NDC) [http://www.fda.gov/Drugs/InformationOnDrugs/ucm142438.htm]. 
RxIMAGE images and data are freely available. The freely accessible 
RxIMAGE API provides text-based search and retrieval of images and data 
from the RxIMAGE database. By contributing their algorithm and 
software, Challenge participants will take part in a broader NLM effort 
to develop a freely usable software system and a freely accessible API 
for image-based search and retrieval from a mobile device.
    In a typical scenario for a future NLM mobile app, a person will 
download the app and use it to photograph a prescription pill, possibly 
under poor lighting conditions, from an angle, or at low resolution. 
The future app will communicate with the future pill image recognition 
software system, which may use all or part of any Challenge entry, to 
compare that photo to reference images in the RxIMAGE database, and 
will return one or more reference images that most likely match the 
photographed pill along with their associated pill data.

[[Page 2877]]

    NLM provides two directories of images to Challenge participants 
for use in preparing their submissions:
     Directory DR contains 2000 JPEG reference images of 1000 
pills. For each pill there are two reference images, one of the front 
of the pill and one of the back of the pill. These pill images are the 
same as in the RxIMAGE database.
     Directory DC contains 5000 JPEG consumer-quality images of 
the same 1000 pills that were photographed for DR. However, they were 
taken with a variety of digital cameras, under various lighting 
conditions, and at camera angles not necessarily perpendicular to the 
faces of the pills. They are akin to photos of prescription pills that 
the general public might take.
    For a pill for which there is at least one consumer-quality image 
in DC, DR has two reference images of that pill, one of the front of 
the pill and one of the back of the pill. Conversely, for a pill for 
which there are two reference images in DR (one of the front of the 
pill and one of the back of the pill), DC has two or more consumer-
quality images of that pill, taken under different conditions.
    DR and DC come with a ``ground truth table'' that is a two-column 
table with column headers ref_images and cons_images. Each row of the 
table gives in the first column the name of a reference image and in 
the second column the name of a consumer-quality image corresponding to 
that reference image. There is a separate row in the table for each 
(reference image, consumer-quality image) pair, even when multiple 
reference and consumer-quality images are all photos of the same pill. 
Respondents can use the images in DR and DC, the ground truth table, 
and the RxIMAGE database in developing, training, and validating their 
algorithms and software. They can also supplement these data.

Rules for Participating in the Challenge

    Teams of one or more members can participate in this Challenge. 
There is no maximum team size. Each team must have a captain. 
Individual team members and team captains must register in accordance 
with the Registration Process for Participants below. The role of the 
team captain is to serve as the corresponding participant with NLM 
about the Challenge and to submit the team's Challenge entry. While NLM 
will notify all registered Challenge participants by email of any 
amendments to the Challenge, the team captain is expected to keep the 
team members informed about matters germane to the Challenge.
    (1) To be eligible to win the Challenge prize, a team--
    a. Shall have registered to participate in the Challenge under the 
rules promulgated by the NIH as published in this Notice;
    b. Shall have complied with all the requirements set forth in this 
Notice;
    c. In the case of a private entity, shall be incorporated in and 
maintain a primary place of business in the United States, and in the 
case of an individual, whether participating singly or in a group, 
shall be a citizen or permanent resident of the United States. However, 
non-U.S. citizens and non-permanent residents can participate as a 
member of a team that otherwise satisfies the eligibility criteria. 
Non-U.S. citizens and non-permanent residents are not eligible to win a 
monetary prize (in whole or in part). Their participation as part of a 
winning team, if applicable, may be recognized when the results are 
announced.
    d. May not be a Federal entity;
    e. May not be a Federal employee acting within the scope of the 
employee's employment and further, in the case of HHS employees, may 
not work on their submission(s) during assigned duty hours. Note: 
Federal ethical conduct rules may restrict or prohibit Federal 
employees from engaging in certain outside activities, so any Federal 
employee seeking to participate in this Challenge outside the scope of 
employment should consult his/her agency's ethics official prior to 
developing an submission;
    f. May not be an employee of the NIH, a judge of the challenge, or 
any other party involved with the design, production, execution, or 
distribution of the Challenge or the immediate family of such a party 
(i.e., spouse, parent, step-parent, child, or step-child).
    g. All team members must be at least 18 years old at the time of 
submission.
    (2) Federal grantees may not use Federal funds to develop their 
Challenge submissions unless use of such funds is consistent with the 
purpose of their grant award and specifically requested to do so due to 
the Challenge design, and as announced in the Federal Register.
    (3) Federal contractors may not use Federal funds from a contract 
to develop their Challenge submissions or to fund efforts in support of 
their Challenge submission.
    (4) By participating in this Challenge, each individual (whether 
competing singly or in a group) and entity agrees to assume any and all 
risks and waive claims against the Federal government and its related 
entities (as defined in the COMPETES Act), except in the case of 
willful misconduct, for any injury, death, damage, or loss of property, 
revenue, or profits, whether direct, indirect, or consequential, 
arising from participation in this Challenge, whether the injury, 
death, damage, or loss arises through negligence or otherwise.
    (5) Based on the subject matter of the Challenge, the type of work 
that it will possibly require, as well as an analysis of the likelihood 
of any claims for death, bodily injury, property damage, or loss 
potentially resulting from Challenge participation, no individual 
(whether competing singly or in a group) or entity participating in the 
Challenge is required to obtain liability insurance or demonstrate 
financial responsibility in order to participate in this Challenge.
    (6) By participating in this Challenge, each individual (whether 
competing singly or in a group) and entity agrees to indemnify the 
Federal government against third party claims for damages arising from 
or related to Challenge activities.
    (7) An individual or entity shall not be deemed ineligible because 
the individual or entity used Federal facilities or consulted with 
Federal employees during the Challenge if the facilities and employees 
are made available to all individuals and entities participating in the 
Challenge on an equitable basis.
    (8) By participating in this Challenge, each individual (whether 
participating singly or in a group) and entity grants to the NIH, in 
any existing or inchoate copyright or patent rights owned by the 
individual or entity, an irrevocable, paid-up, royalty-free, 
nonexclusive worldwide license to use, reproduce, post, link to, share, 
and display publicly on the Web the submission, except for source code. 
This license includes without limitation posting or linking to the 
submission, except for source code, on the NLM Pill Image Recognition 
Web site [http://pir.nlm.nih.gov/challenge]. In developing its future 
software system and future API, NLM may include algorithms and software 
from Challenge entries and may consult with individuals or teams that 
submitted entries. Thus, the license also permits NLM to develop the 
future software system and the future API, independently or with 
others, using any algorithms or software from Challenge entries, 
including those obtained from other Challenges or solicitations, and 
NLM may freely use, reproduce, modify and distribute the resulting 
future software system and API without restriction. NLM may work with 
individuals or teams that submitted entries to write articles about 
pill image recognition and submit them to peer-

[[Page 2878]]

reviewed journals. Each participant will retain all other intellectual 
property rights in their submissions, as applicable.
    (9) NIH reserves the right, in its sole discretion, to (a) cancel, 
suspend, or modify the Challenge through amendment to this Federal 
Register notice, and/or (b) not award any prizes if no entries are 
deemed worthy. In addition, NLM reserves the right to disqualify any 
Challenge participants or entries in instances where cheating or other 
misconduct is identified.
    (10) Each individual (whether participating singly or in a group) 
or entity agrees to follow all applicable federal, state, and local 
laws, regulations, and policies.
    (11) Each individual (whether participating singly or in a group) 
and entity participating in this Challenge must comply with all terms 
and conditions of these rules, and participation in this Challenge 
constitutes each such participant's full and unconditional agreement to 
abide by these rules. Winning is contingent upon fulfilling all 
requirements herein.
    (12) Each individual (whether participating singly or in a group) 
and entity grants to NLM and NLM contractors assisting NLM with C3PI 
the right to review the submission, study the algorithms and the code, 
and run the software on other sets of images.
    (13) Submissions must not infringe upon any copyright, patent, 
trade secrets, or any other rights of any third party. Each individual 
(whether participating singly or in a group) or entity warrants that 
he/she or the team is the sole author and owner of any copyrightable 
work that the submission comprises, that the submission is wholly 
original with the participant or is an improved version of an existing 
work that the participant has sufficient rights to use and improve. In 
addition, the submission must not trigger any reporting or royalty 
obligation to any third party. A submission must not include 
proprietary, classified, confidential, or sensitive information.
    (14) The submission does not contain malicious code such as 
viruses, timebombs, cancelbots, worms, trojan horses, or other 
potentially harmful programs or other material or information.
    (15) Notwithstanding the above and consistent with the principal 
objective of the Challenge to make results widely available to the 
public. If the submitter distributes their executable code or source 
code NLM encourages every individual and team to distribute their 
submission's executable code and preferably also its source code to the 
public under an Apache 2.0 License that permits the public to benefit 
from and improve upon the submission. In this case and by mutual 
agreement, NLM may also post or link to the code from the Pill Image 
Recognition Web site. The entry must include a description of how and 
under what license terms it intends to make any code that is part of 
the entry available to the public.
    (16) Challenge participants are free to discuss their submission 
and the ideas and technologies that it contains with other parties, 
except as stated in #7 above and are free to contract with any third 
parties so long as they do not sign any agreement or undertake any 
obligation that conflicts with any agreement that they have entered 
into, such as with any team members, or do enter into regarding their 
submission for the Challenge. For the purpose of clarity, Challenge 
participants acknowledge that the intent of the Challenge is to 
encourage people to collaborate and share ideas and innovations.

Registration Process for Participants

    To participate in this Challenge, team captains must register their 
teams, including providing the names and email addresses of all team 
members, at http://pir.nlm.nih.gov/challenge/register. Early 
registration is encouraged in order to be able to receive email 
notifications if this Federal Register notice is amended to change the 
timeline or the rules of this Challenge.

Submission Requirements

    Participants must provide a complete submission as defined below to 
be considered for the prize. The submission must be saved to a USB 
storage device containing a virtual machine and must be received (not 
simply post-marked) by NLM by May 31, 2016, 5 p.m. EDT. Submissions 
must be mailed to: Pill Image Recognition Challenge, Computational 
Photography Project for Pill Identification (C3PI), National Library of 
Medicine, Building 38A, Room B1-N30, 8600 Rockville Pike, Bethesda, MD 
20894.
    The Submission is defined to include:
    1. Executable software for ranking how well consumer images of 
pills taken by digital cameras match reference images. The software 
shall be a batch-mode program or a script whose input consists of a 
directory of consumer images and a directory of reference images. The 
output shall be a comma-separated-value (csv) M-by-N matrix MR of ranks 
that for i = 1,...,M compares consumer image i with reference images j 
= 1,...,N. For each consumer image, no rank can appear more than once. 
The software does not need to identify pills by name.
    2. Source code for the executable that is both human- and machine-
readable. The source code can be written in any programming 
language(s).
    3. A .csv file containing the matrix MRC (C for Challenge) of ranks 
that is the output from executing the executable using DC and DR as 
input. In this case MRC is a matrix that has at least 5000 rows and has 
2000 columns. For each row i, MRC(i,j) will rank how well reference 
image j matches consumer-quality image i, for j = 1,...,2000. If 
reference image J best matches consumer-quality image I then MRC(I,J) = 
1, and if reference image K is the worst match to consume-quality image 
I then MRC(I,K) = 2000.
    4. A text file written in English and containing the algorithm in 
pseudo-code that the source code implements, and a description of how 
it works and any tools or packages that it uses. The pseudo-code is to 
have the complete pipeline from the input directories to a matrix of 
ranks, and also include any code that implements features or does 
offline training.
    5. A one-page text file written in English that contains the 
following:
a. Title of entry
b. Names and email addresses of the team captain and all team members
c. A five or more character identifier for the entry that is used as a 
prefix in the names of all of the team's submitted files
d. A brief description of the submission

Amount of the Prize; Award Approving Official

    Up to five monetary prizes may be awarded: $25,000 for 1st Place, 
$15,000 for 2nd Place, $5,000 for 3rd Place, and two $2,500 prizes for 
Honorable Mention for a total prize award pool of up to $50,000. The 
names of the winners and the titles of their entries will be posted on 
NLM Web sites. The Award Approving Official is the Director of the 
National Library of Medicine.

Payment of the Prize

    Prizes awarded under this Challenge will be paid by electronic 
funds transfer and may be subject to Federal income taxes. HHS/NIH will 
comply with IRS (U.S. Internal Revenue Service) withholding and 
reporting requirements, where applicable.

Basis Upon Which Winners Will Be Selected

    NLM will first review submissions to determine their suitability 
for judging and eligibility to win the prize. An

[[Page 2879]]

eligible submission (a) complies with the rules in this Federal 
Register notice, (b) follows the detailed submission instructions at 
http://pir.nlm.nih.gov/challenge/, (c) is complete (e.g., meets the 
Submission Requirements above), and (d) is confirmed by NLM that the 
executable is an implementation of the submitted source code, can be 
run on the submitted virtual machine using directories DC and DR as 
input, and the output matrix is the same as the submission's matrix 
MRC.
    Eligible submissions will proceed to the judging process, and will 
be evaluated based on how well the submissions can match pill images 
across a large number of queries. NLM has developed evaluation software 
and created directories of images to use in selecting the Challenge 
winners. For example, directories DRJ and DCJ (J for Judging) contain 
reference and consumer-quality images similar to those in directories 
DR and DC provided to potential Challenge participants, and the ground 
truth matrix for judging MGTJ has MGTJ(i,j)=1 if DCJ(i) and DRJ(j) are 
photos of the same pill, else MGTJ(i,j)=0. The evaluation software 
calculates the mean average precision (MAP) for directories and ground 
truth matrix such as DRJ, DRC, and MGTJ. Mean average precision is a 
widely used measure for evaluating how well information retrieval 
systems (for example, search engines) retrieve results across a large 
number of queries. The MAP formula and examples of MAP calculations for 
this Challenge are at http://pir.nlm.nih.gov/challenge/MAP_example.
    The eligible submissions will be submitted to the evaluation 
software. The submission with the highest MAP score will be recommended 
as the first place winner, with the second, third, fourth, and fifth 
best MAPs, respectively being recommended to earn second place, third 
place, and two honorable mentions. In the event of tied scores, the 
tied submissions will be tested against additional DRJ and DCJ 
directories until a winner is determined. Dr. Terry Yoo will serve as 
the judge for the competition, thereby overseeing the evaluation 
software process and being responsible for its proper application.

Additional Information

    The NLM Computational Photography Project for Pill Identification 
(C3PI) is a research and development project in the Office of High 
Performance Computing and Communications (OHPCC) within the NLM Lister 
Hill National Center for Biomedical Communications (LHNCBC). C3PI 
computer scientists conduct computer vision R&D in text- and image-
based search and retrieval. C3PI's overall goal is to help improve the 
prescription drug information made available to health professionals 
and consumers. The NLM ``Pill Image Recognition Request for 
Information'' (PIR RFI) (https://www.fbo.gov/index?s=opportunity&mode=form&id=a1a694718366ea7cbaf8f715047d63e1&tab=core&_cview=0) was a pilot for this Challenge. The RFI was announced in 
February 2015 and responses were due in May 2015. RFI responses were 
used to test, evaluate, and as needed refine the components of the 
Challenge, including its instructions. NLM appreciates the work done by 
the parties that responded to the RFI.

    Dated: January 11, 2016.
Betsy L. Humphreys,
Acting Director, National Library of Medicine, National Institutes of 
Health.
[FR Doc. 2016-00777 Filed 1-15-16; 8:45 am]
BILLING CODE 4140-01-P