The Research, Technology and Development (RTD) Evaluation Topical Interest Group (TIG) within the American Evaluation Association has as its mission to advance the field of performance measurement and in-depth assessment of programs, particularly publicly funded RTD programs.
We provide a forum for the exchange of ideas, good practice, and emerging tools by:
- organizing the RTD portion of the AEA conference each year
- collecting and disseminating information related to good practice, as well as events of interest, and gray literature
Announcements
July webinar!
"Engaging the metascience community to bolster NIH stewardship activities"
Monday, July 8th, 11:00-11:50 am EDT
Abstract:
As the largest public funder of biomedical research in the world, the National Institutes of Health (NIH) has long engaged with the research community to study NIH processes and programs, and to develop robust measures for assessing its impact. To fully capitalize on contributions from the growing metascience field, NIH is exploring avenues to pursue study questions of mutual interest that could bolster existing stewardship activities. Towards this goal, NIH has partnered with the Office of Management and Budget to seek research partners through the Evidence Project Portal that would address the agency's evidence needs and further strengthen evidence-informed decision-making. Another effort under consideration is a Metascience Scholars Program that would provide selected researchers access to NIH proprietary data and the opportunity to work closely with NIH staff to assess and advance the impact of biomedical research.
Bio:
Ira Kukic Kuhn is a Health Science Policy Analyst in the Office of Evaluation, Performance, and Reporting, within the Division of Program Coordination, Planning, and Strategic Initiatives in the Office of the Director at NIH. In her current role, Ira works to better capture, communicate and enhance the value of biomedical research through stewardship activities including reporting of long-term impacts of NIH activities. Prior to her current role, Ira was a Science and Technology Policy Fellow of the American Association for the Advancement of Science in the Office of Science Policy at NIH. Ira did her postdoctoral studies at Yale University School of Medicine and holds a Ph.D. in Molecular, Cell and Developmental Biology from the University of Pittsburgh.
Join ZoomGov Meeting
https://nih.zoomgov.com/j/1601072363?pwd=TmtYY3JRYzNONVd5bVY3QndkZTJvQT09
Meeting ID: 160 107 2363
Passcode: 222887
One tap mobile
+16692545252,,1601072363#,,,,*222887# US (San Jose)
+16469641167,,1601072363#,,,,*222887# US (US Spanish Line)
Dial by your location
+1 669 254 5252 US (San Jose)
+1 646 964 1167 US (US Spanish Line)
+1 646 828 7666 US (New York)
+1 551 285 1373 US (New Jersey)
+1 669 216 1590 US (San Jose)
+1 415 449 4000 US (US Spanish Line)
Meeting ID: 160 107 2363
Passcode: 222887
Find your local number: https://nih.zoomgov.com/u/abxb39PupI
Join by SIP
1601072363.222887@...
Join by H.323
161.199.138.10 (US West)
161.199.136.10 (US East)
Meeting ID: 160 107 2363
Passcode: 222887
June webinar!
“All your base are belong to us”: Generative AI and research impact
June 20th from 11:00 am to 11:50 am US Eastern
The recent emergence of generative AI has many in academia scrambling to keep up. As more students and researchers apply these tools to their writing and research activities, questions about their accuracy and ethical use become more important. This is especially true in the field of research impact, where generative AI could transform the ways that we measure and describe the impact that research has on science and society. In this talk, I will discuss a project in which we used generative AI to generate impact stories for selected research projects and then validated those stories using the projects’ known research outcomes. Our results indicate significant weaknesses and potential biases in the current technology, suggesting that generative AI is not yet ready to perform these kinds of tasks. I will then close with some reflections on our work so far and on the potential future of generative AI in research impact studies.
Chris Belter is the lead analyst and evaluator for the Division of Extramural Research at the Eunice Kennedy Shriver National Institute of Child Health and Human Development (NICHD), a division of the US National Institutes of Health. At NICHD, he designs and conducts analyses to inform the Institute’s strategic planning and funding decisions, provides expert advice on the use of data and analytics in those decisions, and coordinates analytical activities across the Division. Prior to joining NICHD, he co-created and led bibliometric service programs at the NIH Library and the Central Library of the National Oceanic and Atmospheric Administration. Chris has a master’s degree in library science from the University of Maryland and a BA in religion from Shenandoah University.
April and May webinars!
Analysis of the National Cancer Institute’s Pathway to Independence (K99/R00) Award
April 24th from 12:00 pm to 1 pm US Eastern
The National Cancer Institute (NCI) K99/R00 award is intended to help postdoctoral scholars transition in a timely manner to research independence and to foster their development of an impactful cancer research program that is competitive for subsequent independent funding. We analyzed factors that impact peer review outcomes and evaluated whether NCI K99/R00 awardees have achieved the goals of the K99/R00 funding mechanism. Our analysis shows the NCI K99/R00 award facilitated the successful transition of postdoctoral scholars to research independence and enhanced the likelihood of K99/R00 awardees to secure subsequent R01-equivalent NIH grant support although not in an accelerated fashion as originally intended. An NCI K99/R00 award was not determined to be a prerequisite to obtain a faculty position, but for some awardees it was an asset in that transition. Our results suggest that the NCI K99/R00 award is an important component for training and retention of the next generation of independent cancer researchers and to increasing the percentage of women and promoting the diversity of the cancer research workforce. There are additional lessons learned from this type of analysis. The work was performed in collaboration with The Center for Research Strategy, and Digital Science, Inc. A large portion of its importance is due to this collaboration which promotes fair and unbiased collection and analysis of the data. This type of analysis also shows us that our data can be used to provide answers to a large breadth of research questions, not just those which were the original focus of our study. The same data can be used for different interpretations based on the questions asked and the needs of the researchers and audience.
Corinne Boulanger-Espeut PhD, MBA, is a Health Science Administrator/Program Director in the Cancer Training Branch (CTB) in the NCI’s Center for Cancer Training (CCT). Dr. Boulanger-Espeut is responsible for managing a grants portfolio of training and career development awards. She has a strong affinity for training, mentoring and teaching, and is excited to work with the next generation of cancer researchers. Before joining CBT, Dr. Boulanger-Espeut had a long career in intramural science at the NIH. During her time in the laboratory, Dr. Boulanger-Espeut co-authored numerous papers, spoke at many national meetings, and trained many students and post-doctoral fellows. She has also taught graduate courses for many years in biosecurity and biotechnology.
|
Join Zoom Meeting
|
|
One tap mobile:
|
US: +19292056099,,96532723849#,,,,*159889# or+13017158592,,96532723849#,,,,*159889#
|
|
Meeting URL:
|
https://rtiorg.zoom.us/j/96532723849?pwd=eFRidVJxS3lvTjFNUy9KUzREaHdsUT09&from=addon
|
|
Meeting ID:
|
965 3272 3849
|
|
Passcode:
|
159889
|
Evaluating Benefits from Publicly Funded Research using Patent-Data: Applications, Benefits, & Misuse
May 15th from 12:00 pm to 1 pm US Eastern
How does the public benefit from public investment in research and development? While there is broad agreement that new technologies have significant social and economic impact, tracing technologies back to federally funded research is challenging. To measure benefits, funding agencies, the national labs, and others often look to patent data. Recently, economists and legal scholars have raised questions about conventional measures (including simple patent counts and patent citations) linking public R&D to patents, and linking patents to social outcomes. Bhaven Sampat will review these concerns, and also describe and assess several new advances in patent bibliometrics that aim to overcome them.
Bhaven N. Sampat is an economist by training, and a Professor in Arizona State University’s (ASU’s) School for the Future of Innovation in Society and School of Public Affairs, and Research Associate at the National Bureau of Economic Research (NBER). He is a founding member of NBER’s Innovation Information Initiative (I3), a data collaborative for open innovation data and related analytics, tools, & metrics (https://iii.pubpub.org/).
Submit to the 2024 AEA Annual Conference!
We invite you to submit an abstract for RTD TIG program at the 2024 AEA Annual Conference. The conference will be held in Portland, OR, October 21-26. Submissions are due at 11:59 pm eastern time on March 22, 2024: https://www.eval.org/Events/Evaluation-Conference/Conference-Theme.
*Please note there is a new process for submitting abstracts this year.* To have your abstract reviewed by the RTD TIG follow these steps:
- First, select the *Stream*: 'Government and Public Policy'
- Only after that will the *Topical Interest Groups *appear. Select *'*Research, Technology, and Development Evaluation' TIG.
Eval2024 Pre-Conference Planning
For Eval2024 (Oct 21-26, 2024) in Portland, Oregon, the RTD TIG wants to re-establish a half-day pre-conference to allow more time to learn from one another. If you are based in the Portland area and/or are interested in serving as a TIG ambassador, or more generally in helping to program this pre-conference, please let the TIG leaders know. Watch this space for more information as the program develops.
RTD TIG Webinar Series
May Webinar
Evaluating Benefits from Publicly Funded Research using Patent-Data: Applications, Benefits, & Misuse
How does the public benefit from public investment in research and development? While there is broad agreement that new technologies have significant social and economic impact, tracing technologies back to federally funded research is challenging. To measure benefits, funding agencies, the national labs, and others often look to patent data. Recently, economists and legal scholars have raised questions about conventional measures (including simple patent counts and patent citations) linking public R&D to patents, and linking patents to social outcomes. Bhaven Sampat will review these concerns, and also describe and assess several new advances in patent bibliometrics that aim to overcome them.
Bhaven N. Sampat is an economist by training, and a Professor in Arizona State University’s (ASU’s) School for the Future of Innovation in Society and School of Public Affairs, and Research Associate at the National Bureau of Economic Research (NBER). He is a founding member of NBER’s Innovation Information Initiative (I3), a data collaborative for open innovation data and related analytics, tools, & metrics (https://iii.pubpub.org/).
March Webinar
“Lessons Learned from RTI’s Evaluation Support for NIH’s Translational & Commercialization Programs.”
Wednesday, March 13th at 12pm US Eastern
The webinar will feature Dr. Kathleen Rousche and Dr. Julia Berzhanskaya from the US NIH’s National Heart, Lung and Blood Institute (NHLBI) Innovation and Commercialization Office (I&C) and Dr. Lena Leonchuk of RTI International, a long-standing NIH/NHLBI evaluator. The speakers will share information on how NIH and NHLBI have incorporated evaluation methods into their program planning, measurement development, and decision-making, and discuss an example of the Update Tracker commercialization outcomes monitoring system for the NIH Research Evaluation and Commercialization Hubs (REACH) and its predecessor, the NIH Centers for Accelerated Innovation (NCAI), supported by the NIH Small business Education and Entrepreneurial Development (SEED).
Please join us at this link: https://rtiorg.zoom.us/j/93787277690?pwd=MDduc3FFRHAzTy94RmlmNDlSOFE5QT09&from=addon
The presentation will be recorded and posted on our Youtube channel here.
February Webinar
The RTD TIG hosted on Monday, February 12, a webinar that consisted of a series of lightning talks.
-
The Promise and Perils of Data Linkage by Andrew Miklos (NIGMS/NIH)
-
AI and Evaluation by Madeleine F. Wallace (Windrose Vision)
-
Predictive Data in Evaluations by B. Ian Hutchins (University of Wisconsin)
-
Simulation in Evaluations by Anand Desai (Clarivate), Cynthia Phillips (Decision Catalyst, LLC), and Lisa Gajary (Caspian Strategy and Analytics, LLC)
-
Evaluations of Research Infrastructure/Instrumentation by Jeff Alexander (RTI) and Josh Schnell (RTI)
You can view the recording of this and previous webinars at the following link:
https://www.youtube.com/channel/UCNZXXFRu6KHyIs_r2HWBZtQ. Additional information on future webinars will be forthcoming at our TIG business meeting and in future announcements.
------
The RTD TIG membership held its 2023 business meeting earlier last week, with a recap of TIG activities from this year, including Eval 2023 in Indianapolis, and re-electing the current officers for 2023/2024. Thanks to all Eval 2023 presenters and participants and to our proposal reviewers for making Eval 2023 another great conference. Congratulations to our 2023/2024 officers:
- TIG Co-Chairs: Robin Wagner and Josh Schnell
- TIG Program Co-chairs: Didi Cross and Gavin Reddick
- TIG Leadership Member-at-Large representative: Paula Fearon
Program Input Sought
As we are committed to providing the RTD TIG community with engaging and educational programming, we are seeking your input via a very short survey here. Please fill this out right away to have your input included in our planning for the year.
RTD TIG Webinar Continues
To expand our engagement beyond the annual conference, the TIG hosts webinars on relevant RTD topics and posts them on a Youtube channel here. Our next session will be on November 13th, 12pm US Eastern. Josh Schnell will host a discussion of recent developments, such as new conceptual models (e.g. convergence research) and evaluation of ARPA funding models. Another goal of this session is to hear from RTD TIG community members about issues/topics they are interested in that can be incorporated into a) the 2024 annual meeting call for proposals, b) the program of the proposed 2024 pre-conference meeting, and c) the monthly webinar series.
https://rtiorg.zoom.us/j/99736280248?pwd=dkZNdFl6cXJSY1pLNjBsenIyMUgyUT09
Meeting ID: 997 3628 0248
Password: 339771
Find your local number: https://rtiorg.zoom.us/u/aJmfYHu9R
Other TIG Resources
RTD Topical Interest Group (TIG) Paper, February 2015
Evaluating Outcomes of Publicly Funded Research, Technology and Development Programs: Recommendations for Improving Current Practice (FINAL)
White paper SUMMARY slides
Generic Logic Models, collection over time