FDA Clinical Decision Support: New Rules for Healthcare AI

The U.S. Food and Drug Administration’s “Clinical Decision Support Software: Guidance for Industry and Food and Drug Administration Staff,” finalized in September 2022, explicitly defines which CDS software is considered a medical device under Section 520(o)(1)(E) of the Federal Food, Drug, and Cosmetic Act (FD&C Act), and which software is exempt from FDA oversight[1]. This long-awaited clarification comes after years of ambiguity, during which AI-driven CDS tools proliferated across health systems, often without clear regulatory guardrails. The new guidance is not just a technical update—it is a regulatory inflection point that will reshape how healthcare organizations, AI developers, and compliance officers approach the deployment and governance of clinical AI.

The Regulatory Boundary: What the FDA Now Regulates

The FDA’s guidance pivots on four statutory criteria that determine whether a CDS tool is a regulated medical device. The most significant change is the FDA’s insistence that, to be exempt, CDS software must allow healthcare professionals to “independently review the basis” for any recommendations it generates[1]. This means that AI models functioning as “black boxes”—where the logic, data sources, or reasoning behind a recommendation are opaque—will almost certainly be regulated as medical devices. The FDA’s rationale is clear: if a clinician cannot understand or validate the reasoning behind a recommendation, the software is effectively making the decision, not supporting it. This distinction is especially relevant for machine learning models that ingest vast datasets and output risk scores or diagnostic suggestions without providing interpretable explanations. The FDA guidance states: “If the user cannot independently review the basis for the recommendation, the software does not meet the fourth criterion and is subject to FDA oversight as a device”[1]. This is a direct response to the proliferation of AI tools that promise clinical insight but offer little transparency, a trend that has alarmed both regulators and patient safety advocates.

The FDA’s definition of CDS software that is not regulated as a device is now tightly circumscribed. To qualify for exemption, the software must: (1) not acquire, process, or analyze medical images or signals from in vitro diagnostic devices; (2) be intended for the purpose of displaying, analyzing, or printing medical information about a patient or other medical information; (3) support or provide recommendations to a healthcare professional about prevention, diagnosis, or treatment; and (4) enable the healthcare professional to independently review the basis for the recommendations so that it is not the intent that the professional rely primarily on any such recommendation to make a clinical diagnosis or treatment decision[1]. If any of these criteria are not met, the software is regulated as a device, triggering the full suite of FDA requirements, including premarket review and quality system regulation.

Explainability and Transparency: The New Compliance Imperative

The FDA’s guidance elevates explainability from a best practice to a regulatory requirement for CDS software seeking exemption. This is a marked shift from prior years, when many AI vendors touted performance metrics while downplaying or ignoring the need for model transparency. The FDA now expects that exempt CDS tools must “identify the patient-specific data, the medical information relied upon, and the rationale or support for the recommendation” in a way that is “understandable to the intended user”[1]. For AI developers, this means that the days of deploying inscrutable deep learning models in clinical settings without clear documentation or rationale are over—at least for those seeking to avoid FDA device regulation.

This requirement has profound implications for the design and deployment of AI in healthcare. Developers must now build systems that not only generate recommendations but also provide clinicians with the ability to interrogate the underlying logic. For example, a CDS tool that flags potential drug interactions must show the specific medications, the interaction risk, and the evidence base supporting the alert. For more complex AI models, such as those predicting sepsis risk, the software must allow clinicians to see which patient features (e.g., lab values, vital signs) drove the risk score and how the model weighted those features. This level of transparency is nontrivial to implement, especially for models trained on high-dimensional data, but it is now a baseline expectation for regulatory compliance.

The FDA’s stance aligns with broader trends in AI governance, including the European Union’s AI Act and the U.S. White House Blueprint for an AI Bill of Rights, both of which emphasize explainability and human oversight as key safeguards[2]. For health systems and vendors, this means that explainability is no longer just a differentiator—it is a regulatory necessity. Failure to provide it risks not only FDA enforcement but also loss of trust among clinicians and patients.

Compliance Pathways: What Regulated CDS Software Must Do

For CDS software that does not meet the exemption criteria—either because it processes medical images/signals or because its recommendations are not independently reviewable—the FDA’s device requirements are triggered. This includes premarket notification (510(k)), premarket approval (PMA), or De Novo classification, depending on the risk profile of the software[1]. The FDA has signaled that it will apply a risk-based approach, focusing its oversight on software that could directly impact patient safety if it malfunctions or is misused.

Manufacturers of regulated CDS software must also implement quality system regulations (QSR) under 21 CFR Part 820, covering design controls, validation, risk management, and postmarket surveillance. This is a significant operational burden, especially for startups and smaller vendors accustomed to the more permissive environment of “non-device” software. The FDA’s guidance is explicit: “Manufacturers of device CDS software must comply with applicable requirements, including but not limited to, registration and listing, premarket submission, labeling, and adverse event reporting”[1]. For many AI vendors, this will require a fundamental shift in product development, documentation, and postmarket monitoring practices.

The FDA has also clarified its expectations for software modifications and updates. If a CDS tool is updated in a way that changes its intended use, logic, or risk profile, the manufacturer may need to submit a new premarket notification or supplement. This is particularly relevant for AI models that are retrained on new data or that incorporate adaptive learning. The FDA expects a robust change management process, including documentation of how updates are validated and how their impact on clinical performance is assessed[1]. This requirement dovetails with emerging best practices in AI lifecycle management, such as model versioning, audit trails, and continuous performance monitoring.

Operational Implications: What Healthcare Leaders Must Do Now

The FDA’s new CDS guidance is not just a technical document—it is a call to action for healthcare executives, compliance officers, and technology leaders. For CTOs at health systems, the immediate priority is to inventory all CDS software in use or under consideration, mapping each tool against the FDA’s four exemption criteria. Any software that fails the “independent review” test must be treated as a regulated device, triggering a review of vendor compliance with FDA requirements. This may necessitate renegotiation of contracts, enhanced due diligence, and, in some cases, discontinuation of noncompliant tools.

CISOs and compliance officers must also revisit their risk management frameworks. The FDA’s guidance underscores the importance of transparency, traceability, and postmarket surveillance for CDS software. This means that security and compliance teams must work closely with IT and clinical stakeholders to ensure that software procurement, deployment, and monitoring processes align with FDA expectations. For regulated CDS tools, this includes verifying that vendors have completed the necessary premarket submissions, maintain robust quality systems, and have processes in place for adverse event reporting and software updates.

For AI developers and vendors, the new rules demand a shift from performance-centric development to compliance-centric design. This includes building explainability features into the core architecture, documenting model logic and evidence sources, and establishing rigorous validation and change management processes. Vendors must also be prepared to support health system clients in FDA audits and to provide detailed documentation on software functionality, risk assessment, and postmarket monitoring.

Looking ahead, the FDA’s guidance is likely to drive consolidation in the CDS software market, as smaller vendors struggle to meet the regulatory burden and health systems gravitate toward established players with proven compliance track records. It will also accelerate the adoption of explainable AI frameworks and tools, as both regulators and clinicians demand greater visibility into how AI-driven recommendations are generated.

For this quarter, CTOs and CISOs at regulated health systems should take the following operational steps: (1) Conduct a comprehensive review of all CDS software to determine regulatory status under the new FDA guidance; (2) Require vendors to provide detailed documentation on explainability, validation, and FDA compliance; (3) Update procurement and risk management policies to reflect the new regulatory landscape; and (4) Initiate staff training on the implications of the FDA’s CDS rules for clinical workflows and patient safety. These actions are not optional—they are now essential for legal compliance, risk mitigation, and maintaining trust in the use of AI in healthcare.