ICYMI: From Insight to Impact: AI-Driven Decision Making for Leaders

ACT-IAC’s 2026 Health Innovation Summit featured a panel on mission-critical AI decision-making across CMS, FDA, and NCI. Panelists described AI’s growing day-to-day role, including secure NIH-wide tools, productivity gains (reported time savings), and reducing administrative friction like performance reviews, emphasizing impact occurs when AI is embedded in workflows with “human in the lead.” The session closed with vendor guidance: show products via video, be future-focused, lead with data-sharing and tool strengths, and bring a “total package.”

https://www.actiac.org/act-iac-event/fellows-friends-day-domaine-fortier 

Summary - A Hole in One with ACT-IAC

Voyagers Program | ACT-IAC

Subscribe on your favorite podcast platform to never miss an episode! For more from ACT-IAC, follow us on LinkedIn or visit http://www.actiac.org.

Learn more about membership at https://www.actiac.org/join.
Donate to ACT-IAC at https://actiac.org/donate.
 
Intro/Outro Music: See a Brighter Day/Gloria Tells
Courtesy of Epidemic Sound

(Episodes 1-159: Intro/Outro Music: Focal Point/Young Community
Courtesy of Epidemic Sound)

Transcript

Yohanna: [00:00:00] A few weeks back ACT-IACt held our 2026 Health Innovation Summit. On this episode of the Buzz, we showcase a discussion from that summit on AI decision making for leaders with panelists from the F-D-A-C-M-S and NCI. They highlight examples of how to move from data in. Sites to action and impact through AI workflows.

Yohanna: In case you missed it, here it is. 

Ratima Kataria: I guess I'm up again. Alright, so we welcome to the stage here, our panelists for this next session, which is a great segue from the previous discussion. And I'll tell you in a second how Andres Colón Pérez is the Chief Technology architect at CMS. We have with us Andreas Schick, the director of Data Science and Analytics at FDA.

Ratima Kataria: We have Dr. Ying Huang, the program officer at NCI, and we have with us also Ms. Jessica Berrellez, the Vice President of Strategy and [00:01:00] Operations DLS, but also former FDA. So thank you so much to our panelists for taking the time in joining us. So we talk all morning about innovation and uh, really appreciate Megan, the question that you asked our panelists in the last session.

Ratima Kataria: And you know, when we talk about data to insights. But it doesn't end there. What really is most meaningful is then taking those insights into action and impact, right? So that is something, if there is one thing you want to take away from the day to day, I think that's where our heart and energies need to be, is.

Ratima Kataria: Data to insights, but insights to action and then to impact. So with that, we are now shifting gears to that engine that actually is reshaping how the leaders make those decisions and, and drive that impact. Not the hype, not the headlines, the real stuff, the mission critical impact across [00:02:00] C-M-S-F-D-A-N-C-I and the industry here.

Ratima Kataria: So let's dive into that. Uh, we are going to do something slightly different to make sure all of you are also not passing out right before lunch. So what we are gonna do is instead of doing audience questions at the very end, we are gonna mix it up. We are going to do two questions. Um, that I would ask the panelists and then a question from the audience, and we'll keep mixing it up as we go.

Ratima Kataria: So please do think about your questions as we go along with that. Our first question, um, that we will start maybe with you, uh, Dr. Wong, is when you think about AI today, what is the one way it's already showing up in your day-to-day work? And I know it is different for everybody, but we'll start with you, Dr.

Ratima Kataria: Wong. 

 Dr. Ying Huang: Thank you. Thank you very much. Uh, first of all, I am so proud that HHS and un IH have rolled out the AI tools for everybody in our [00:03:00] institute. Everybody can have a taste what will change our life, and we could do that with the workforce who work the work that is the best. I think that's the signal with two parts.

 Dr. Ying Huang: One part is that our leaders would like. Everybody to use it and catch up with advances of technology. And two, that we do have a security system that we can use our institutional data within the institute and make a great impact to the Americans. 

Ratima Kataria: Andreas. 

Andres Colón Pérez: So a way that it's showing up. Uh, so at CMS we've been, uh, you know, working on AI now for several years.

Andres Colón Pérez: Um, and so, um, I've had the privilege of being very close to that, uh, as a co-founder of the AI Explorers Program, uh, which essentially, um, looked at what was coming in in ai, the change that it was gonna have in this ticket and, and further and, um, given the, the [00:04:00] organizational material that we had at the time.

Andres Colón Pérez: Um, start doing rapid experimentation for rapid learning, minimizing risk, and incorporating what worked and scaling it. Um, today, you know, we've seen dramatic advances from where we were, right? Um, so, um, AI showing up in my work from any, uh, helping, uh, augment workflows, right? We can do a lot. Think lot of things faster.

Andres Colón Pérez: We're hearing from colleagues, uh, across the organization through qualitative and quantitative research that we're saving anywhere from like five hours a week. For their work. And so, um, this is allowing, uh, increased productivity for the organization, not just for me, but uh, for many others. So 

Ratima Kataria: really appreciate that insight about it is showing up everywhere, right?

Ratima Kataria: And I think, um, as Dr. Butler and Patrick and several others pointed out, it is going to have that real impact at scale, only when it starts showing up in the workflow. And that's [00:05:00] where you will see the productivity change. So how about you, Andreas? And then we'll go to Jess after that. I, 

Andreas Schick: I don't know about everyone else, but anyone else have to do, uh, performance evaluations this month.

Andreas Schick: So, uh, where, where I've really been, uh, seeing it show up in, in the agencies, uh, with tasks where there's, uh, I call 'em friction points or frustration points. So, administrative work. I don't know about you guys. I was, I was born to push paper, but my, that, that's not a sentiment, uh, shared by a lot of my colleagues.

Andreas Schick: They, uh, they, they see a form and they're like, Ooh, I don't know about that. But, um, this was the first, you know, beginning of the year we had do performance evaluations, all these administrative forms, and people were just like, Hmm. Yeah, maybe, maybe it would be good if I consulted AI and had it write up some of this, this language for me, and then provide suggestions for me.

Andreas Schick: You know, the, the [00:06:00] tasks, you know, a lot of individuals kind of agonized over for hours and hour and hours. Um, but you know, they're able to expedite quite efficiently with AI once they, you know, they saw the options. They're like, okay, I like this one. And they added their edits and they went in that direction.

Andreas Schick: Um, this year. You know, I think, I believe like operations were enhanced quite, quite well. Uh, everyone I think finished everything in about a week, which is, you know, pretty, pretty impressive. Um, given that there was a, there was a, you know, litany of new requirements placed on us. This your good requirements, but you know, there, whenever that happens, you know, it's a little bit time consuming.

Andreas Schick: You figure it all out and. And, and try to understand like how best to implement them. But this year everyone just rolled it out in a week. It was, it was, it was pretty about as painless as painless can be when it comes to administrative work. I had fun, but my, again, my colleagues didn't enjoy it so much, but, you know, just, just that, those little tiny, [00:07:00] small things, just like, you know, but a year ago no one would've said, oh, well let, let me ask AI This year, the first, a lot of people are increasingly, even people who are kind of.

Andreas Schick: You know, not so on the AI bandwagon or, you know, the first thing that does sort of come to mind is, well, I could, I could at least look at this as a resource. See what options it provides to me. 

Ratima Kataria: Jess. 

 Jessica Berrellez: Yeah. I, I, I, I'm just gonna keep beating the same drum and I, I'm really speaking from the perspective of someone who was recently, uh, a federal IT executive and has now transitioned over to industry.

 Jessica Berrellez: And so speaking, come from that sort of dual vantage point. Uh, we're, we're at such a different place now than we were. A year ago, a year and a half ago. Um, you know, I, I think that this past year the focus was really on access, these rapid deployments, uh, and, and experimentation. Starting to talk about risks, starting to talk about.

 Jessica Berrellez: Uh, uh, [00:08:00] appropriate use. And we're increasingly seeing this shift, I think, across numerous different domains, whether it's, uh, research or operational regulatory, where we're starting to see AI a lot more embedded in our day-to-day workflows. And so the, the conversation has really shifted from, you know, should we use ai?

 Jessica Berrellez: Can we use ai, uh, to, um, not just how, but how, how can we do it better? Uh, and I think that's really exciting. Uh, I, I remember a year and a half ago, uh. Two years ago being on, you know, the government side and there, there was a lot of concern, there was a lot of pushback. And the pushback wasn't just from the business side of the house, it was also internal within our, you know, IT organization, uh, thinking about risks, thinking about all of the challenges.

 Jessica Berrellez: That we were faced with that I think are still very real around, uh, you know, basic things or, you know, issues that we were facing around infrastructure, data, uh, [00:09:00] cybersecurity. Um, those challenges are still there, right? And so how do we balance it all? But right now, I think, you know, there's this incredible moment that we've been talking about all day, but there's also this reality that as leaders are now starting to be very interested, uh, in ai.

 Jessica Berrellez: Expectations are also rising. And how do we reconcile, uh, the rising expectations with the reality that, you know, most AI projects fail, uh, that, you know, most AI deployments aren't, uh, really transformational. Uh, I think that's where forums like. These are really helpful because we get to learn about these really, really good, impactful use cases across government and, and hopefully try to apply those lessons learned.

 Jessica Berrellez: So it's a really exciting time, but there are also a lot of issues that remain, I think, around, um, uh, risk and fit for purpose and, um, and, and frankly, capacity and workforce that I think we'll dig into today. 

Ratima Kataria: You know, these are some great examples here, and given the hat that I'm wearing [00:10:00] today in this conference, I would like to say that AI actually also helps us prepare.

Ratima Kataria: For panel discussions on ai. You know, I'm not gonna lie about that, right. Um, for a lot of us who are engaged in a lot of the industry associations regularly help with planning, conferences, panels, and such. I'm gonna say that it has definitely helped reduce a level of lift, uh, when we are able to use ai, you know, instead of spending.

Ratima Kataria: An hour or hour and a half writing panel abstracts and draft questions and what have you. Late in the night after your day job is over, you're able to at least leverage that as a resource. But with human in the lead verse, Patrick, in the room, I'm already adopting human in the lead, right With human in the lead.

Ratima Kataria: So with that, our next question is, Andreas, we'll start this with you, is you all talked about going from discussion. To experimentation and now the next big step is going to scale. Can you talk about whether it's an [00:11:00] example or a story or challenges? How did your organization move, even if it's set a, it's, it's a small example, but going from that discussion to experimentation, to starting to deploy at scale a, any thoughts you have there?

Andres Colón Pérez: Yeah, absolutely. So got a couple of examples, but I'll, I'll, uh, um, with the emergence of generative ai, um, we saw within a matter of weeks early, uh, private experimentations between, uh, you know, I wrote code, a couple of colleagues wrote code, um, and we were all very excited about the promise of, you know, what, what would happen if we could use this for, um, uh, reducing burden of, of reduction, assisting a human in their work, um, uh, reducing, um.

Andres Colón Pérez: Uh, you know, finding knowledge across the organization, going to one place and asking questions. And so, um, in our, uh, organization, our current practice, uh, especially in the Office of Information Technology is that, um, we want to do small experiments. We wanna be [00:12:00] scrappy, we just wanna prove the idea, see if this is technical, technically feasible, where are some of the blind spots, especially that's an area that we want to see.

Andres Colón Pérez: Um, and then. Technology is not enough, right? You can have a really cool tool. Um, it, it could be feasible to actually, um, you know, deliver some products, but will actually impact the business, right? Um, we will deliver, uh, value. And so that's why, you know, we are, we're being rigorous internally of doing this proof of concept and scaling to a pilot.

Andres Colón Pérez: And then from there on, if we can prove the idea. In a business context, then invest further. So, um, an example of that was our first, uh, generative AI product. Um, we, we started with, um, uh, you know, that really, uh, uh, scrappy version. Um, and then we had, uh, very small investment. To, uh, further refine what, what it would actually look like if users were gonna authenticate to a system, interact with knowledge, what are some of the bare core [00:13:00] features that are needed?

Andres Colón Pérez: And, um, our approach, when we, we decided, okay, we're, we're actually gonna launch this for the organization was, um, that we were. We're all very excited, but we were gonna be very honest about its limitations, right? Uh, primarily because many of our users had never touched, uh, a system like chat, GBT. Um, and so while some of us area adopter were, you know, we're very familiar with these tools, never everybody was.

Andres Colón Pérez: And so what that meant is they might not be familiar with, uh, some of the drawbacks, uh, some of the gaps. And so we start seeing, um, our approach was instead of releasing it to everybody. We made it a join, uh, wait list. And so that essentially allowed those folks that were very excited to try like that, you know, they were already using in their private lives and we didn't know about it.

Andres Colón Pérez: Uh, they were reaching out and saying, you know, want to join the wait list. And so once they were approved, that started building momentum. 'cause they, you know, they would do things [00:14:00] faster, they would share with their colleagues and then they were like, oh, how do you did that? And, and so that started building momentum within the organization.

Andres Colón Pérez: Um, we still saw lapse in terms of understanding, right? Like, um, uh, it wasn't particularly, uh, clear initially, like why would a system that is from CMS not know all the facts about CMS? And so when we started, it wasn't augmented with CMS knowledge. We were using like, um, models as they came out of the box, as they were trained with, uh, on the internet.

Andres Colón Pérez: Um, and so we had to do. Uh, you know, engagement with folks, we had to educate them. And the, the ability to have them slowly roll in, really help with that, um, so that we could set up the right foundations for proper training, right? And then broader rollout. And today, you know, the entire organization has access.

Andres Colón Pérez: Um, 80% of the users, uh, have, uh, you know, are, uh, marketing as high satisfaction with the tool. And saving time. So that's one of the examples of how we're doing, um, you know, small experimentation [00:15:00] and scaling up for the enterprise. 

Ratima Kataria: What I loved over there is the technique and the tactic you guys are using instead of the technology folks pushing AI to the programmatic people.

Ratima Kataria: You are having them join the wait list. So it's a push versus pull mechanism and, and what a wonderful strategy there. I think we can adopt that in many cases. So really appreciate you sharing that example and the tactic. Um, Andreas, how about you in the FDA space? 

Andreas Schick: I, I was really impressed with how we rolled out Elsa.

Andreas Schick: Um, it was, it was kind of an interesting concept. They just, we took a lot of individuals who were very smart and we put 'em into a room. And they just coded all day. Um, so like at tech companies that, that's kind of normal. But, uh, we was, it was pretty profound in, in, in the, in the FDA, I don't think anyone ever imagined, you know, just one day it's like, Hey, uh, can you come to this room?

Andreas Schick: And then it's like, oh, we're just going to, we're just gonna build [00:16:00] and implement a chat bot for the FDA in about two months. Who, who's with me? Yeah. That was. That that was, that was that. I just still remember that. That was, that was, that was very funny. Um, and it was just interesting seeing that dynamic of, um, everyone just coming together every day and just kind of scrambling to get that done.

Andreas Schick: Um, kind of what, what struck me is interesting is, you know, we could have actually done that a long time ago. Right? Um, you know, just two years ago. People proposed, why don't we implement a chatbot? And you know, the, the discussion was just like, well, but there are risks to that. You know, we don't really know what this is gonna do.

Andreas Schick: There's a lot of talking, but there wasn't a lot of acting. And I think what I, what kind of the impression I got, you know, in the last year that really impressed me was they just, the, the new group just kind of decided to, they made a decision, they just kind of implemented it. It, [00:17:00] um, what a concept. But, uh, now we actually have a chat bot.

Andreas Schick: People are working with a chat bot. They, uh, used it to help them write P maps as I, sorry, I'm gonna keep going back to the, but, um, you know, it's, it is just, it is just interesting that from, as Jess mentioned, from a year to, to now, just like the huge, it's, it's not the big things that have changed. It's just those little things that you, you see.

Andreas Schick: You know, here and here and here sprinkled out throughout the day that have changed it. It's um. It's just very interesting. 

Ratima Kataria: So Andrea, for one, we all know that you're done with your P maps this year. I think we, we got that. We got that for sure. Um, we can imagine, and you know, innovation is not just about using technology.

Ratima Kataria: Innovation is also about rethinking how to do things. And I think what you just talked about here is the model that FDA followed of bringing all those people in a room. And saying, let's figure out how to do this together. That in itself is [00:18:00] shifting the culture to be more innovative, so really appreciate you sharing that story.

Ratima Kataria: Ying, how about you at NCI? 

 Dr. Ying Huang: First of all, I haven't finished my P map yet, 

Ratima Kataria: so Andreas talk to Andreas after this. Yes, we won't tell anyone, uh, he'll help you. I 

 Dr. Ying Huang: will go home and I will use strategy B. Um, I really loved how, uh, our HHS fellows are. Developing the tools for their in institute use like va, C-M-S-F-D-A admire you guys and MVU guys, um, to that age.

 Dr. Ying Huang: We have large system, we have a lot of tools developed it. We will not call them AI tools, but they are the system. Beautiful purpose. The fit for purpose at the time. We know those data information about our institute can generate more knowledge about us, just about us, [00:19:00] but we have barriers. The system do not always talk to each other or talk to each other in the same language.

 Dr. Ying Huang: We are at the point we have questions about our own institute. We know there are. Data and answers vary in all these kind of systems. How could we use AI to answer those questions and the brain, the administrative data to more knowledge? For example, the, uh, NH has launched the, uh, investigation of using AI to evaluate, help the pos, to evaluate the, uh, data sharing plan.

 Dr. Ying Huang: That was a great project. We see a lot of. For pos participate as a super user, provided their ideas and um, how even more what they wanted the tool to do that shows the data lies [00:20:00] the administrative work that we do day to day can be something be useful to the society. For example, one of the things we are tackling with is.

 Dr. Ying Huang: Data type, some core genomic omic that are something everybody seems to know, but where is the definition, how you are defining it? People are looking at N-C-I-N-I-H to find out how you guys think about a western blood. Is that a image or it is a jail or it is a functional assay? You can tag it in any way you want, but when you want to have all the institutional.

 Dr. Ying Huang: Information together and find trends and guide our PIs and pos what to do with the information. For example, we can find out who are using the similar methodology and generate similar data and put it to some repository. We can analyze the [00:21:00] cost, analyze the process, how easy it is, and advise the newcomers to say, Hey look, this is our chart to reflect.

 Dr. Ying Huang: Where are the data as a product of stay and how they are managed? Then you don't have to knock at the door to say, oh, I, I heard that your PhD is doing that. How did he do with data? You don't have to do that anymore. You will have a more, uh, comprehensive and more evidence-based. Uh, decision that you can make for your institute and for the researchers and society taking, uh, one downfall is that everybody's making their own AI tools.

 Dr. Ying Huang: I hope that industry can help us to streamline it and find the sweet spot that it won't be another fit for purpose tool just for this group of people. 

 Jessica Berrellez: Thank you. 

Ratima Kataria: Fit for purpose is the key [00:22:00] and how you're applying AI to helping with some of the policies around in the Office of Data Science at NCI is very, very impressive.

Ratima Kataria: Jess, would you like to add anything to this? 

 Jessica Berrellez: Yeah, absolutely. You know, I just, um, this is, this is such an exciting day and discussion and, you know, I, I, I've heard, I feel like a couple times now, well, like, this isn't new, right? Well, we, AI's been around for a long time, but I think what's special about this moment is that we have this like a.

 Jessica Berrellez: Top down push, right, that we're seeing from the new administration at the HHS, um, office of the CIO level. And then we have this groundswell of very organic interest. And so that's really been a forcing function, I think, across the, the federal health space in particular to, to help us push through. A lot of these barriers that we've been trying to chip away at for a really long time.

 Jessica Berrellez: And you know, I think what helps us to move the needle is isn't just, you know, uh, more or better tools. It's [00:23:00] really leadership clarity and workforce readiness, and a willingness to continue to address some of these silos that you mentioned and some of the organizational cultural barriers that have been there for a long time.

 Jessica Berrellez: Uh, you know, and I think when we talk about embedding AI into workflows and, you know, fit for purpose, identifying, uh, uh, and surfacing the, the best use cases, all of that requires a lot of work. So it isn't something that just magically happens where you start doing ai, and that's where I think a lot of, like the good old fashioned.

 Jessica Berrellez: Planning and strategy and, and, um, uh, effective it and data management are really important. So all of these like, you know, fundamental things that we've been doing and talking about for a long time are. Are still critically important. We've also been able to learn so much about, um, the, the [00:24:00] importance of, of, um, UX and cx bringing users, um, to the table, front and center, and all of this work that we're doing.

 Jessica Berrellez: And so there's really, I think, a lot of work that we're able to, to tap into and to leverage and a lot of understanding from years of doing, you know, digital transformation and modernization work. That have led us to this moment. And I think that in this, you know, in the excitement and in the rush to, to get this across the finish line, we still need to think about things like people and culture.

 Jessica Berrellez: Um, one of the areas that I really focused on a lot when I was at FDA, um, was around, um, uh, technology workforce upskilling and re-skilling. We started an agency-wide, uh, re-skilling a new skilling program that was really creating learning paths. Based on roles and, and jobs to be done. Uh, we also, uh, uh, launched an enterprise-wise, um, wide, uh, AI literacy program, [00:25:00] uh, called AI for Everyone.

 Jessica Berrellez: And it was, you know, trying to reach different types of, of users and practitioners and taking a very holistic approach, uh, and, and starting to, you know, kind of demystify some of these. Really challenging topics. Uh, being willing to create space, to ask questions, to have tough, tough conversations, to acknowledge that, you know, we don't necessarily have this all figured out.

 Jessica Berrellez: We don't necessarily have all of the answers. Uh, we also created a digital leadership program, and this was to help equip, uh, uh, leaders with the skills to really be able to, you know. Assess ai, assess emerging technologies and, and help to strengthen some of what I like to call smart skills. So not soft skills, but you know, uh, some of the, the skills around, um, collaboration, communication, uh, learning agility, growth mindset that are so important for leading through change.

 Jessica Berrellez: And so thinking about that, I think what Patrick called the, the total package, um, I think that's something that's [00:26:00] gonna be really important for us to continue focusing on. 

Ratima Kataria: I agree, Jess, and it is amazing that you have the experience on the federal side and now you're on the industry side, so your insights of.

Ratima Kataria: The pain points and challenges that you faced in your role in the federal government, I think would be very, very valuable for the industry to learn about. And I've gone through that experience and journey myself, so I really appreciate your insights there. And I think we are going to hear about total package theme throughout the day.

Ratima Kataria: It is definitely sticking as well as total experience. You know, you talked about the holistic thinking. That is extremely key. So as I had promised, our wonderful audience here, we are gonna mix things up, and now it is time for us to take one audience question as we continue through our panel discussion.

Ratima Kataria: All right. Does anybody have a question ready to go? Alright. Yes sir. Go ahead. I think they're bringing a mic over to you. Thank you Ra. 

AUDIENCE: Hey, uh, Andres, how are you mate? Um,

AUDIENCE: um, even [00:27:00] the last panel and so far actually I haven't really heard much about health outcomes, right? So for example, in Europe, which we all know is highly regulated and bureaucratic, et cetera, they collect very little data relative to the us. They're far ex far more excellent health outcomes in terms of population.

AUDIENCE: Well, England is an exception because we have far too many pubs, but, uh, but generally speaking, Western Europe has much better health outcomes relative to the amount of data they collect. On the contrary, us both federal and, you know, private sector of voracious. Consumers of public data, what are we doing with that data?

AUDIENCE: If our health overall, relative to Europe, Western Europe is actually in the other direction, we are not having good [00:28:00] health outcomes overall relative to Europe and they, they consume far little data. So I have really two questions. How much of this data is really relevant in terms of health outcomes? What are we doing with this data?

AUDIENCE: You can talk about interoperability and all of that. What does it really mean when it comes to the general public with diabetes and this and that? What are we doing with it? 

Ratima Kataria: Who wants to take that? That is a wonderful question, Ragu. We appreciate it. Go ahead, Andreas. All yours. 

Andreas Schick: It's like, I'll, I'll be brave and take that one.

Andreas Schick: I 

Ratima Kataria: think, yeah.

Andreas Schick: I kind of had this conversation actually with somebody else just a week ago, and I, I think it, it boils down against a, a common theme we've been hearing, which is leadership, right. There's, there's, I've noticed there's two kind of leadership styles when it, when it comes to like the implementation of ideas.

Andreas Schick: There's, there's a style, which is you look busy and you create widgets. Right? And data's really good for doing that. You have your little rapport, you crunch the [00:29:00] numbers, and you have a widget. Um, so a good example of that in a space I work in is you see an issue and then you're like, uh oh. And then you may you have a group trying to address that issue.

Andreas Schick: Right. So that sound, that's great. You have a good KPI And the good thing is you might prevent it an issue, right? But it's a lot of work and it doesn't lead to sustainable durable change. It doesn't lead to long-term improvements in health outcomes. Like you mentioned, if you really want to have a long-term health, uh, uh, improvement in health outcomes.

Andreas Schick: What you should do is you should take the information and try and see the pattern that's continually emerging, that always causes that reaction to it to occur. And then ask yourself, what is the underlying problem that is con perpetually causing this issue to emerge? And then address [00:30:00] it. Right. And so for, I can't go into too much detail of what it was, but it's a, it was a controlled substance and the, the real problem has to deal with how active pharmaceutical ingredients, the underlying ingredient of the actual product, how it's all scheduled.

Andreas Schick: Um, um, bought purchasing decisions are made a, a year out in advance. Um. And that can cause problems, uh, when there is a sudden shift in demand, because controlled substances, you can only make so much of 'em, right? Because they're, they're controlled substances. So, you know, the solution to that then is addressing the, the underlying, you know, scheduling of the API, which is actually very, very, very difficult.

Andreas Schick: It requires a lot of different planning. Um, but it kind of goes back to like, how is the data used? If it's used just to react, that's great, right? You know, at the, at the heat of the moment. You're, you're probably gonna help some individuals access their medication, which prevents a [00:31:00] lot of pain in their, in their life.

Andreas Schick: Um, but if you actually wanna make sure that perpetually happens and then they never have to go through any inconvenience in the first place, you have to deal with it. You have to use the information and data to lead a structural change in the actual overall process. Um, and that, that can be a lot harder.

Andreas Schick: I'm not sure if that's what, I'm not sure if that's the reason why there's a difference between America and and Europe. Um, but I think if we want to get to a, a, a, a state in the end where we are having much better health outcomes, we have to, we have to start using this data and information to really tackle some of those really challenging, underlying structural, uh.

Andreas Schick: Issues that we're starting to identify with our data and analytics. 

Yohanna: Who's doing it? 

Andreas Schick: Who's doing it? That's a good question. Are you doing it Andris? We do it every day. We do it. Oh, okay. [00:32:00] We do it every day. 

Ratima Kataria: Doing it every day. And now if I can add to this a little bit, is ra, when you asked that question and you had, part of the things baked into your question is things, we talk about interoperability and data silos.

Ratima Kataria: Yes, there is a challenge. Yes, a lot of data is being collected, but all of these agencies and all the constituents, the people who are working on this, we are not able to. Connect the dots across these fragmented silos of data. The data that comes from EHRs, the data that comes from research publications, the data that comes from other consumers, right?

Ratima Kataria: These are fragmented data sets, so it's not that there is, I don't think we have been talking about data and panels and with our clients and such for years. Now. This is not a new topic. I have not once heard that we are not collecting enough data. I've never heard that. That's not the problem. Volume of data is not the problem.

Ratima Kataria: It is what the previous panel talked about, the credibility of [00:33:00] the data, uh, the readiness of the data, the interoperability and the connectivity of the data across these various silos. I think that's where the challenge lies and, and I think that's what a lot of our federal partners and vendor partners are trying to break through.

Ratima Kataria: Um, I don't know. That's just my 2 cents on, on a part of the answer. Yes, please go ahead, sir. 

AUDIENCE: So the silos that you're referencing, is it internal or external or 

Ratima Kataria: reference? I believe it is both. There is a lot of intramural and extramural there, so I don't wanna take away from the airtime from the panelists.

Ratima Kataria: Is there, um, anybody else who would like to answer that question before we move on to the next? 

 Jessica Berrellez: Yeah, I mean, I, I think it's really important too for us to, you know, um, start to, to disaggregate AI in a research or clinical context, or in a re regulatory context and also in an operational context. And I think that many of us who are here are, you know, um, coming at this conversation from an operational [00:34:00] perspective.

 Jessica Berrellez: And I, I think it also. Speaks to varying levels of maturity across the, the federal health space, but also a need to convene not just conversations like this more frequently, but across all of these different, um, areas so that we can bring these varying perspectives together. And I think too, like, you know, we talked earlier on in the program about what a different conversation we're having now versus a year or a year and a half ago.

 Jessica Berrellez: Um, and I think that, you know, we're also starting to get to these more important. Questions around value and around impact. Um, but, but we're not there yet. And I do think, again, that's where strategy is so important and, and also governance. And that's probably, um, a good segue to our next question. 

Ratima Kataria: You are absolutely right.

Ratima Kataria: Um, I guess I'm gonna have to pay you for this one, Jess, for, for creating the segue. So I think everybody is aligned with. Leveraging the emerging [00:35:00] technology to create that speed to impact. And I think that's a no brainer. It's going to help us with that. But how do you balance that with governance, with the risk management part of it?

Ratima Kataria: In the prior panel, we heard about ensuring the data security, the privacy is protected, and all of that good stuff, right? So when we are using AI or any of this emerging technology to advance and continue to accelerate. The speed to impact. How do you balance it out with all these other elements which are going to make sure that the outcomes are going to be well informed and not biased?

Ratima Kataria: So maybe Jess, we start with you this time and we'll, we'll go this in this direction. 

 Jessica Berrellez: Sure. I, I, I think the reality is that, you know, we're many, many agencies are, are essentially. You know, they're, they're deploying AI while building governance in parallel or trying to, to retrofit it to a certain extent.

 Jessica Berrellez: And I think that's just the, the world that we're in right now. And the [00:36:00] reality of what in change management, we, we call like the, the messy middle. Um, I do think that a lot of, you know, existing governance and standards, uh, you know, obviously apply and are, and are critically important. But you know, when, when it comes to ai, um, in particular, I think the, the.

 Jessica Berrellez: The key point that I wanna emphasize around governance is that it's just really important to, to get started and to expect to have to iterate a lot. Um, and, uh, you know, I, I, I can share a little bit of insight from, from FDA in terms of how this, how some of like the technology and data modernization work that we did over the last, uh, several years helped to lay the foundation for now.

 Jessica Berrellez: The really exciting work that's happening at the agency around Elsa and other initiatives. And, you know, it was really started, like we, we started with something very simple. It was an AI playbook, which is essentially like an amalgamation of here are good practices for our early adopters and for our practitioners.

 Jessica Berrellez: [00:37:00] This was a couple years ago, and then that evolved into, uh, uh, you know, starting to talk about, uh. Policy and guidance in a much more thoughtful way, uh, summarizing all of the, uh, existing policies and guidance that were relevant, um, starting to, um, uh, exert some thought leadership. And, um, uh, the agency published a position paper that was essentially a summary of all of the applicable, uh, guidance, but also, uh, uh, what was really a dialogue around different considerations, acknowledging that we don't have all the answers.

 Jessica Berrellez: This is really complex. Uh, we're still working to understand here's what we're thinking right now. That then led to, um, an enterprise advisory board comprised of, uh, diverse stakeholders across the organization to start to identify and evaluate. Different use cases, uh, to start to think about evaluation.

 Jessica Berrellez: Um, and that then led to a much more [00:38:00] robust governance, uh, council that is in place today. And so it was essentially starting to mature governance in parallel to, you know, again, doing the thing and the NIH, uh, now that I'm in the NIH environment, it, it's taking a very sort of. Similar iterative approach where there's a lot of, um, uh, interest and there's this focus on, um, uh, you know, bringing stakeholders to the table from different institutes and centers and offices across the agency.

 Jessica Berrellez: There's, um, a responsible, uh, AI tiger team that's underway doing a lot of exciting work in this space. And I love that the focus is on collaborative governance and also making it understandable, uh, and bringing diverse perspectives to the table. 

Ratima Kataria: Thank you Jess Ying. 

 Dr. Ying Huang: Um, it's a great to think about the outcome and we transform the outcome to targets.

 Dr. Ying Huang: Then we u. [00:39:00] With the targets, each agency and offices have their own mission and their tasks responsibilities. Um, data sharing is a huge thing for our office, office of Data sharing. At NCI, we are the guardian of the genomic data that our NCI and other agencies, uh, collaborators generated. We are the one to make sure those data are used.

 Dr. Ying Huang: In the responsible way. So in years we are promoting, come over to our website, use our data, publish it right, make impact. We have targets. When AI come in, we learn with the society how to put responsible of using AI into our book and make sure the data are used. By the person should use it and really generate the knowledge to [00:40:00] meet our target.

 Dr. Ying Huang: It's not easy. I think this challenge is also lies with I-R-B-F-D-A. When you use the data to gen, to train your algorithm. What happened to the data when you publish your algorithm? How I'm going to hold. Who accountable for the youth and the downstream, a lot of things that is our target. I hope that our target integrated into the whole picture of health outcome and we could make changes and we, we could catch up with Europe people and I think we have a lot of data, as you said, they use even less data than we have.

 Dr. Ying Huang: We have great people, great ideas, great technology, um, passionate as you guys wanted to help. I think we [00:41:00] define the targets and we move towards that. We call for industries help to align our targets and to get to our targets faster. Safer and make more impact to our overall target. Uh, we are public. The NIH has, uh, RFI request for information for how we are going to modify the GDS uh, genomic data sharing policy.

 Dr. Ying Huang: Uh, through that you can put your comments in and let us know how you think and that age should align our. Mission, our task force towards our overall target. Thank you. 

Ratima Kataria: I'm going to ask audience a question with the show of hands. How many of you have got client facing meetings, sales target, anything in that space?

Ratima Kataria: Many. [00:42:00] Dr. Ying Wong here has got data use targets. How cool is her work? Oh my gosh. She has got targets about data use. That's the space to be in. Um, but that Andreas, your thoughts on it? 

Andreas Schick: Yes, I largely, I got that came out strong. I largely agree with every, what everyone else has set up to this point. Um, for, for me, with governance, uh, I think barring a lot for what, what you said, Chas is.

Andreas Schick: Um, you have to do it. You have to start, and you have to not worry about getting it right 'cause you're not gonna get it right. And, but, and even if you do think you get it right, it's gonna change the next day and it's gonna be wrong. So you just have to be as agile as the technology you're trying to govern.

Andreas Schick: Um, in my experience, I think the, the most effective way of dealing with governance is to talk to your, to as many people as possible. I think that the last, uh, I mean the last panel really. Illustrated very, uh, well, that something that's very valuable is having as many perspectives as [00:43:00] possible. 'cause you never know what that one unique perspective is gonna bring to the table.

Andreas Schick: And with governance, the challenge isn't, you know about something, you have to deal with it. You might not deal with it. Well, the challenge is you don't know about something at all. And then it, then it, then it comes, and then, then, then people are like, why didn't you see this? It's so obvious, but, um, it's only obvious once it happens.

Andreas Schick: Um, but, you know, I was talking to, uh, Ben Rogers, the, the CAO of CDC last week about governance of all things. And we were just asking was like, what are you saying? And the stuff he brought to the table. Like I had, I was like, I'd never even have heard of this. I thought I, I thought I talked to everyone. I thought I heard about, just about everything you can hear.

Andreas Schick: And I'd never heard any of the problems they were dealing with. Um, so you always just be amazed, like what you can hear, but I think that's the, the strategy you have to employ is like you do governance has to be treated like its own real [00:44:00] job. It can't be treated as some extracurricular side project, you know?

Andreas Schick: I know. 'cause it's not glamorous, right? Just like, no. Okay. I'm not gonna say it again. Yeah. I was so tempted. But it has to be its own, it has to be recognized as its own function. That requires, you know, its own dedicated staff that are on top of this and we have to be talking, uh, with as many people as possible.

Andreas Schick: Like not only that was an interesting break. Um, not only are other government partners, but also, you know, our colleagues in the private sector, um, because they're ahead of, you know, in many cases they're ahead of us and they're starting to see things that, you know, we can hopefully prepare for as we reach that level of maturity.

Andreas Schick: Um, but that's kind of, that's my perspective on it. 

Ratima Kataria: You're very well said. And Andreas. 

Andres Colón Pérez: So when it comes to AI governance, um, one thing that we, we notice. Yeah. Some areas of this technology are more mature than others, right? [00:45:00] Um, when you think about like, traditional ai, ML you know, the risk are well known.

Andres Colón Pérez: Um, whereas, uh, some of these more emerging like AI code generation, um, agent tech systems, like what are some of the implications? And there are some really smart people. Um, some of them are probably here in industry actively, uh, thinking about this colleagues in government, uh, as well, um, establishing standards.

Andres Colón Pérez: But the reality is that. You know that that maturity, there's a gap. And so, um, at the same time, we're living in exponential times, right? So the piece of progress is essentially accelerating. Um, in, in, when it comes to, um, artificial intelligence. There's a convergence that's happening. And so that with governance, which relates to having proper oversight, proper risk mitigation, um.

Andres Colón Pérez: These processes usually are, um, rely on not only like standards, best practices, [00:46:00] policy laws and things like that, and those processes tend to, you know, be relatively slower in government. Right? Um, and so how do we keep pace with that, right? Because we can establish rules that might be completely obsolete, um, might not be relevant.

Andres Colón Pérez: You can have a team that is, um, you know, working on governance based on what was happening 12 months ago. But not with the reality of what's happening now. Right. And so, um, I, the way that we're approaching it is, um, we have our established IT governance. Um, at the same time with the reality of not only the pace of change, but also, you know, we've seen executive orders coming out on artificial intelligence, uh, uh, directives that had come out like last year.

Andres Colón Pérez: There was a significant number, um, thinks 200% increase between, uh, you know, last year and the previous administration. And so what, um. There's, at CMS, one of the [00:47:00] things that we do is we're very proactive about, um, ensuring that we are on top of understanding what does this executive, uh, directive mandate, um, and we do leverage ai, right?

Andres Colón Pérez: Like we, we do analysis with that. We also have human analysis. Um, and we establish outcomes, right? So, um, we have our CIO, uh, Patrick Newbold, who you've heard about, um, his title of leadership is outcome Focused. And so when you have leadership that allows you to, um, you know, have clear priorities at the same time, we're.

Andres Colón Pérez: Um, you know, a accepting the fact that this pace is moving really fast. Um, how do we mitigate the risks? Um, how do we have proper oversight? So for oversight, uh, we're ensuring that we know what's out there. We are actively reaching out, engaging with teams. Also in doing data calls where we have like an inventory of all of our a I use cases, we try to ask the right questions to understand, um, what kind of AI they're using.

Andres Colón Pérez: Right. Is it agent, is it, um, generative ai? Is it, uh, [00:48:00] traditionally IML. Um, so that's that oversight piece. And then on the risk side, um, like I said, because there's, um, more clarity in certain areas of risk for, so a, some ai, ml, uh, type of initiatives and others a little bit more, um, ambiguous. Uh, we have the benefit at CMS of engaging very rapidly on, on proof of concept.

Andres Colón Pérez: So what we try to do is engage these teams as early as possible in their development life cycle so that we can, you know, have a conversation, um, let them know, right? We are, um, we've established a cross cutting initiative. So instead of just building one team, let's say in the Office of Information Technology.

Andres Colón Pérez: Looking at what AI governance and risks should be. We go out across the organization and we, um, invite subject matter experts that are passionate about, uh, the problem space, that know that governance will affect them, and that, that we have to be. Innovative and, and understand, think beyond just one single office or center.

Andres Colón Pérez: And then, um, we [00:49:00] work very rapidly to define frameworks and then put them to the test. So last year, um, we, you know, develop a, a risk framework. We, by August, we were already testing it with teams. Um, 10% of all, of all our, all of our high impact use cases, uh, were already, uh, vetted by, um, uh, before the end of the fiscal year last year.

Andres Colón Pérez: And is it perfect? It's not perfect. Um, but we need to build that expertise. We need to know where the blind spots are. And one of the things that we ask our colleagues when they're coming and we're saying, Hey, listen, we're trying this out. We wanna learn from you. We want to help you, but we also need your feedback, right?

Andres Colón Pérez: If we're not asking the right questions, uh, if there are things that are, is not working, uh, about this, uh, process, give us that input. And so that has helped us further refine that. Now we are, uh, we have. Clear outcomes for later in this year to a hundred percent of our, all of our high impact use cases, uh, properly met by April of this year.

Andres Colón Pérez: And so that's an example of how, you know, we're addressing governance in [00:50:00] at 

Ratima Kataria: sales. That is super impressive. And you talked about smart people. I can assure you, Andreas, all the smart people you want to talk to are in this room here. There is no problem you have that cannot be solved by some combination of the people here who's with me on that.

Ratima Kataria: Yeah. Yeah. There you go. Alright. I am looking at the time here, given the time, we are going to do a wrap with a super quick rapid fire. This is gonna be a rapid times rapid fire, 30 seconds age. When vendor partners and folks come to talk to you, how do you want them to show up with what? So we will start with you Andres, and go down 30 seconds or less.

Andres Colón Pérez: For me, if you have a tech product, a video is the best way to get it to us. So rather than asking for a 30 minute meeting, record a video of the product showing how it actually works, and then ship it to us. If it's promising, you'll start to see that we share it across broadly. Um, some colleagues focus a lot more on the business development part, like, this is who we [00:51:00] are and this is what we do.

Andres Colón Pérez: A little bit of that is. Good. But you know, if you want us to look at your product, focus on that videos, share via email. It'll, uh, it'll get the most, uh, rapid eyes and, uh, and, uh, a broader sharing within the organization, if relevant. 

Ratima Kataria: Well noted. Thank you, Andreas and Andreas. 

Andreas Schick: I, I definitely like the video.

Andreas Schick: I, I agree with that wholeheartedly. For me, I'd like seeing solutions for 2027 and 2028 and onward, not solutions for today or yesterday. So I, I highly recommend individuals. Be forward thinking, think about what the solutions for the future, not for the present. 

Ratima Kataria: Excellent. Ying, 

 Dr. Ying Huang: I would like everybody come over and say, I love sharing data.

 Dr. Ying Huang: So that it 

Ratima Kataria: fabulous 

 Dr. Ying Huang: that it gives the tickets to all the conversation. And tell me what are you good at? What are your, the tricks of your tool? Um. I will expand the door and I will connect people who are in need of those [00:52:00] tools and we start from there. Thank you. 

Ratima Kataria: Awesome. And Jess. 

 Jessica Berrellez: Yeah, I, I'm gonna have to, uh, borrow Patrick, uh, your comment about the total package.

 Jessica Berrellez: I, you know, I, I think come to the table with the total package. One of my favorite days at FDA was a deep racer, uh, race day where we partnered with AWS. We leveraged our existing, uh, contract and, you know, we were able to host this incredible day of, of AI and ML learning. It was so much fun. It was awesome.

 Jessica Berrellez: They have so many great resources and this was really just, um, a. Where we're leveraging resources that are already available to us. And that was so great. So I think that, you know, when we're thinking about what that total package means, it's finding ways to deliver value within existing, uh, products and services.

 Jessica Berrellez: And then beyond that. 

Ratima Kataria: You know, I think I'm gonna announce on Patrick's behalf [00:53:00] this afternoon, anytime we use the word total package, we have to drop a dollar in a jar. And, um, and Patrick, we can split it in the evening. All right, with that, thank you so much to our great panelists here. Before you guys go, we'll do a quick picture.

Ratima Kataria: So, so stay here for another second, but let's give them a big round of applause. 

Yohanna: If you're passionate about technology and eager to explore more incredible events, make sure to visit act iac.org/upcoming-events. Keep your curiosity sparked and your calendars marked. Until next time, stay inspired and connected.