AI in Regulated Environments

What We Inherit

Your data science team built a model. It performs well in testing. Now you need to deploy it in a HIPAA-covered environment, or get it past an FCA review, or demonstrate NIST AI RMF compliance. The model works. The compliance architecture doesn't exist. Most AI vendors don't know what FedRAMP means. You need engineers who understand both the model and the regulatory framework.

The compliance architecture that does not exist is not just a documentation problem — it is an evidence production problem. When a HIPAA-covered entity deploys an AI system, the Security Rule requires audit controls that generate records of system activity sufficient for forensic review. For an AI system making inference decisions on PHI, the audit record must capture what data was accessed, by which model version, at what time, and what decision was produced. Most AI platforms do not generate this record automatically. Retrofitting audit logging to a model serving infrastructure not designed for it requires significant engineering work that changes the architecture of the serving system.

FDA's Software as a Medical Device guidance adds a classification requirement that surprises most healthcare AI teams: if the AI system meets the definition of a medical device — typically when it is intended to aid clinical decision-making for a specific condition or population — deployment may require FDA clearance or a Pre-Submission to establish that an exemption applies. The determination requires analysis of the intended use, the patient population, and the clinical significance of the AI output. Teams that deploy clinical AI without performing this determination are taking regulatory risk that may not be visible until the FDA asks a question.

Model drift is the compliance risk that most AI governance frameworks are only beginning to address. NIST AI RMF requires organizations to monitor deployed AI systems for changes in performance and behavior that could indicate drift from the validated state. For a clinical AI system, drift could mean incorrect recommendations at scale — with clinical consequences — before the drift is detected. The monitoring infrastructure required to detect drift before it becomes a clinical safety event must be designed before deployment, configured during validation, and producing baseline measurements before the system goes live.

Data science teams build models. They are not responsible for the compliance infrastructure that governs those models in production. The organizational gap between the team that builds the model and the team responsible for regulatory compliance is where most AI compliance failures originate. The data science team delivers a performant model. The compliance team discovers after deployment that the data access patterns do not satisfy the minimum necessary standard, that the model's outputs cannot be explained at the granularity clinical staff require, and that the training data did not go through the de-identification process required for HIPAA-compliant AI training.

AI vendor marketing outpaces AI regulatory maturity by several years. Healthcare and financial services organizations buy AI platforms marketed as 'compliant' based on infrastructure certifications — FedRAMP authorization, HIPAA eligibility, SOC 2 Type II. These certifications cover the platform's infrastructure, not the AI application running on it. A HIPAA-eligible cloud platform running a model that logs PHI to an uncontrolled output stream is not a HIPAA-compliant AI deployment. The infrastructure certification is necessary but not sufficient — and the gap between them is the AI application's compliance architecture.

Explainability is treated as a machine learning problem rather than a regulatory requirement. The technical question of how to explain a neural network's output is a research problem. The regulatory question of what explanation is sufficient for a specific clinical or financial use case is a domain question that must be answered by the regulatory framework, not by the capabilities of the explainability library. An explanation that satisfies a data scientist's curiosity about model behavior is not necessarily an explanation that satisfies an FDA reviewer's requirement for clinical decision support transparency.

Startup Compliance Foundation Guide

The minimum viable compliance posture for early-stage companies building AI in regulated industries — before the enterprise sales cycle exposes the gaps.