OpenAI Interest Groups

AI in Healthcare

The FDA just dropped a new guidance on January 7, 2025, with its new draft guidance on AI in medical devices. Anyone involved in developing or even just monitoring AI in healthcare should be prepared for these changes, as they address multiple critical aspects. If these new guidelines are overlooked, AI projects could face significant challenges. Below is a summary of the key points, and a link to the full document is  

https://www.fda.gov/regulatory-information/search-fda-guidance-documents/artificial-intelligence-enabled-device-software-functions-lifecycle-management-and-marketing

. However, these are critical points that change from last guidance about Artificial Intelligence-Enabled Device Software Function. Still, I also recommend you read the full paper if you need to work directly with Artificial Intelligence-Enabled Device Software Functions in the future.

In the past, many assumed that an AI system simply worked “as advertised” because the evaluation of Medical AI may not be a thing that the health profession did it easily, but the FDA is moving away from mere assumptions. It’s no longer just a recommendation; the agency is effectively saying, “Demonstrate clearly how your AI works and what are limitations, or it will not be approved.”

: The FDA now expects disclosure at a level akin to academic research, or at least a substantive summary with sufficient model details (comparable to a “Model Card” in the computer science field). Inserting a brief explanation or omitting critical details is no longer acceptable.

: The FDA provides an example of how a Model Card should be structured, including key information such as data sources (specifying whether real-world data, synthetic data, or a combination are used), performance metrics across different demographic groups, and known model limitations.

: Model Card information will be integrated into the product’s labeling, so users can clearly understand both the limitations and the expected outcomes.

Ultimately, the FDA wants these documents—especially Model Cards—to help physicians, relevant healthcare professionals, and even patients decide which AI solution is appropriate for specific cases, without requiring them to be AI experts who must manually delve into technical papers.

The FDA has long emphasized bias issues, but this latest guidance is more stringent:

: Developers must disclose detailed demographic breakdowns (e.g., race, ethnicity, gender, age) for both training and testing datasets. Broad summaries will no longer suffice.

: The AI system’s performance must be reported for each demographic group. If performance is suboptimal in certain groups, there must be a clear plan to address these shortcomings, and this information must be communicated to end users.

These steps aim to ensure AI systems provide equitable healthcare for all, not just those well-represented in the training data.

Previously, it was recommended to monitor products after they hit the market, but now the FDA is enforcing this more strictly:

: There must be a plan to detect “data drift,” or the degradation of model performance over time, since real-world conditions are constantly changing.

: If the AI system can continue learning on the fly, developers must demonstrate how they will detect errors, notify users, and correct problems swiftly. The positive side is that a robust self-updating mechanism may reduce the need for entirely new approvals whenever the model is updated.

The FDA’s previous advisories on cybersecurity now include more explicit details on the types of threats AI might face and how to protect against them:

: Attackers could subtly alter inputs (e.g., adding noise to medical images) so that the AI produces incorrect diagnoses without the changes being evident to the human eye.

: Attackers might bombard the model with inputs to glean how it makes decisions, potentially reverse-engineering or creating adversarial examples. Proper “red team” testing plans must be in place.

: Deliberately crafted inputs can cause AI to make mistakes, potentially impacting patient care.

While these threats may seem hypothetical, robust defenses will help address real scenarios where criminals could manipulate AI systems to produce harmful or erroneous results.

5. Clarification of “Validation” vs. “Validation Sets”

Confusion often surrounds the term “validation.” The FDA now clarifies:

: This is the final, independent assessment to confirm the AI’s performance.

: These are internal development processes and do not constitute final validation.

This clarification aims to prevent misleading practices where a model appears highly accurate but is merely overfitted to a repeatedly used “validation set.” In the medical domain, true “validation” typically means external or independent validation rather than merely reusing the same data.

The new draft guidelines are more detailed, user-centric, and rigorous than ever before. They represent a significant step in ensuring that medical AI systems are 

. From straightforward algorithms to deep learning and advanced transformer models, developers will need to align with these higher standards.

Though the final details may still evolve, it’s evident that the future of medical AI will be increasingly transparent, stringent, and credible—good news for everyone involved, especially patients. However, developers who are unprepared could face serious complications. As regulatory expectations rise, those in the Medical AI field will need to take on more of a researcher’s mindset in their work.

Draft Guidance in Artificial Intelligence-Enabled Device Software Functions from FDA

The OpenAI Forum connects members to discuss AGI’s future through events, webinars, and networking. The community aims to help us achieve our mission of building AGI that benefits all of humanity.

OpenAI Forum

Job Title

Login to the OpenAI Forum community to connect with others, attend community events, and more!

Login or Sign Up for even more

To unlock all parts of the community and get the best experience, complete your profile.