HTN is pleased to share this piece from Keith Figlioli, Managing Partner and Lise Courtney D’Amico, Principal at LRVHealth and Alan Rosa, Chief Information Security Officer & Head of Infrastructure and Technical Operations, Health and Srikant “Sri” Narasimhan, Vice President, Enterprise Customer Experience & Insights from CVS Health.

Keith and Sri will join Kevin for a live discussion on Thursday, October 22nd at 12p eastern time. You can register to join.

Health systems and payers are on the cusp of an AI agent explosion. Every major enterprise vendor is embedding AI “agent” capabilities into their platforms: Epic, Microsoft, Workday, Salesforce, and others are all introducing tools to build or deploy AI assistants directly within clinical, financial, and operational workflows. Creating a specialized agent no longer requires a team of data scientists. Soon, clinicians, administrators, and analysts will be able to build their own AI helpers by configuring prompts and rules inside the systems they already use.

It’s increasingly clear that this proliferation of agents is inevitable. We believe the vast majority of upcoming healthcare AI agents will emerge from these primary platforms: the EHR, ERP, CRM, and office suites that hospitals and payers already rely on, while the rest will be provided by domain-specific vendors or startups. For health system and payer executives, the key question is no longer if AI agents will permeate their organizations, but how to effectively manage and govern them when they do.

Amid widespread enthusiasm for agenic AI, the industry is underestimating a practical challenge: how to monitor, control, and audit a growing portfolio of autonomous agents operating across critical workflows. Unlike traditional software tools, AI agents can make decisions or take actions with a degree of autonomy. In the near future, a health system could have agents assisting in clinical care, revenue cycle, and operations. Some might be purchased solutions, others built in-house or configured by frontline staff. Who will have visibility into how all these agents are performing and interacting across departments?

Today’s IT monitoring and governance structures are not built for this scenario. Healthcare leaders should start asking several unresolved oversight questions now:

An uncoordinated agent surge could create fragmentation, blind spots, and risk for healthcare organizations. It is not hard to imagine that simply keeping an inventory of all active agents might overwhelm existing IT processes. In the worst case, critical decisions might be influenced by “rogue” agents that leaders did not even realize were running.

However, most of the early conversations about agent oversight envision a dashboard: an inventory of agents, performance monitors, and a kill switch to protect the organization in an emergency. Those capabilities are useful, but they share a limitation. Dashboards observe; they do not enforce. Healthcare organizations already have the core infrastructure needed to govern AI agents: identity.  

Every agent is a new non-human identity with the emerging ability to act like a human. It authenticates to systems, holds entitlements, and takes actions under those entitlements. That means the disciplines healthcare organizations already apply to their workforce can extend directly to agents: unique credentials rather than shared service accounts, least-privilege access scoped to the agent’s function, periodic certification of what each agent can touch, and a defined lifecycle from provisioning to retirement. The difference is scale, speed, and tireless execution. Agent populations will grow faster than any workforce ever has, and agents act at machine speed. An over-entitled employee makes mistakes at human pace; an over-entitled agent can make them by the thousand.

Framing agent governance as identity governance resolves several open questions at once. Inventory stops being a census project: if an agent cannot authenticate, it cannot act; if it must authenticate, the organization knows it exists and can audit it. The kill switch stops being a new product category: credential revocation is a control every enterprise already operates, or should know how to operate. Shadow agents become detectable in the same way shadow accounts are detectable.

Executives do not need to wait for a new market category to act. Four baseline questions can define the foundation of oversight today: 

An organization that can answer yes to all four has already built the foundation of agent oversight with capabilities it owns and understands.

The oversight discussion often treats agents as systems that might make mistakes: a scheduling mistake, a billing misfire, or a drafted note that gets a fact wrong. That is only half the risk. Agents are also systems that will be attacked, and their attack surface is unlike anything traditional application security was built to defend.

Three exposures deserve particular attention. First, prompt injection. Agents act on the content they read, which means a malicious instruction embedded in an inbound fax, referral document, claim note, or email can become a command. The data an agent processes must be treated as untrusted input, the same way security teams learned to treat web input two decades ago. Second, tool abuse. An agent authorized to schedule visits, release records, or resolve payments is exactly as dangerous as the tools it can call. Its blast radius is defined by its entitlements, which is another reason identity scoping is the first control rather than an afterthought. Third, supply chain. As marketplaces of pre-built agents and connectors emerge, organizations will import agent behavior the way they import open-source code, with all of the provenance, versioning, and tampering questions that follow.

Two design principles fall out of this. Irreversible or high-consequence actions should route through human approval regardless of how reliable the agent has been, because reliability against error says nothing about resilience against manipulation. Logging should also be built for forensics, not just performance. When an agent behaves badly, the first question will be whether it malfunctioned or was made to. That investigative requirement is even more critical for an agent than for a human, because humans can be interviewed after an incident and agents cannot.

One further point rarely appears in these discussions: Oversight will be bidirectional. Security operations teams are adopting agents to triage alerts and to investigate incidents. The watchers will need watching. Any control layer worth the cost must govern the defensive fleet with the same rigor it applies to the clinical and administrative one.

Most agent governance discussions assume agents live inside a single legal entity. Increasingly, they will not. Integrated enterprises and health systems that sponsor their own plans face a constraint standalone providers may not encounter: data that lawfully informs one line of business may be restricted from another. Regulatory firewalls between insurance, care delivery, and pharmacy functions were designed for people and documents. An agent with cross-entity access can breach one in milliseconds, invisibly, and at scale.

The control implication returns to identity. Agent entitlements must encode entity boundaries, not just job function, so that an agent serving one line of business is structurally unable to read data it has no legal basis to touch and so the audit trail can affirmatively demonstrate that separation. This is also where governance meets the member and patient experience directly. Every data segregation failure is a trust failure, and trust is the asset that determines whether patients and members accept AI in their care at all. The organizations that can show their agents respect the same legal walls their people do will have earned the right to deploy agents where they matter most.

For patients and members, AI governance will not be experienced as a policy, dashboard, or control framework. It will be experienced in the moments that matter: whether a prescription is filled correctly, a claim is explained clearly, a clinician has the right information, or a person can get help without repeating their story. In health care, experience is not separate from trust. It is how trust becomes visible.

Trust also works differently across the health care system. In pharmacy, it is largely functional: accuracy, reliability, and consistent execution preserve confidence. In care delivery, it is relational: people need to feel heard, known, and safe enough to share information that may affect their care. In health insurance, it is more contractual: coverage decisions, costs, and whether commitments are honored shape the relationship. A single standard for “trustworthy AI” will miss these differences. Agent governance should reflect the experience and trust expectations of the setting in which each agent operates.

That changes how organizations should define success. An agent can meet technical performance targets and still damage the experience if it creates confusion, adds steps, gives an answer without a clear explanation, or removes access to a person when the situation is complex or consequential. Leaders should therefore evaluate the quality of the experience alongside accuracy, uptime, task completion, and model performance. That means monitoring ease, clarity, resolution, continuity, escalation to human support, and whether the agent helps people feel more confident taking the next step in their care.

The stakes are higher in health care because experience can influence behavior. Friction can contribute to delayed care, missed refills, abandoned tasks, or disengagement. A well-governed agent should reduce that burden by making care easier to understand and navigate while preserving meaningful choice. It should support, not replace, the human relationships that patients rely on in vulnerable moments.

This leads to a practical principle: every agent should have both an operational owner and an experience owner. The operational owner is accountable for access, security, performance, and retirement. The experience owner is accountable for the effect on patients, members, caregivers, clinicians, and service teams. Together, they should be able to answer four questions: Did the agent make the experience easier? Did it improve confidence or understanding? Did it preserve an appropriate path to a person? And did it behave in a way that earned the right to be used again?

The organizations that answer those questions with evidence will do more than govern AI safely. They will build confidence through the experience itself. In health care, that is the standard that matters: not whether people trust AI in the abstract, but whether each interaction gives them a reason to trust the organization using it.

These challenges point to an emerging need for what one might call “AgentOps”—a layer of management and governance above the individual AI agents and their host platforms. Rather than replacing core systems like Epic or Microsoft, this control layer would span across them, providing enterprise-wide visibility, oversight, and optimization for the entire fleet of agents. In many ways, it is analogous to how DevOps emerged to manage the software development lifecycle. AgentOps would manage the agent lifecycle from creation to retirement.

We are already seeing early attempts at this control concept. Epic, for example, recently previewed an “Agent Factory” that lets health systems build, customize, and monitor AI agents directly within the Epic environment. This platform is meant to give hospitals a “sandbox” to invent and manage their own AI assistants with guardrails and oversight built in. Workday’s acquisition of Flowise underscores a similar trend: Enterprise vendors want to offer safe agent-building tools with integrated governance.

But these vendor-specific controls may not suffice if an organization runs agents across many systems. Healthcare organizations may need a unified, cross-platform control plane of their own. The core capabilities would likely include a central inventory of all agents, identity and entitlement management, policy-based runtime control, audit logging, cost tracking, and risk tiering. In short, “good” agent oversight is less a passive dashboard than an operating model: Every active agent has a known identity, a defined owner, scoped permissions, observable actions, and a clear path to suspension or retirement.

Critically, governance must be balanced with innovation. Over-centralizing could stifle the operational improvements these agents promise. The goal of AgentOps is not bureaucracy, but safe innovation at scale: enabling teams to adopt AI tools faster because guardrails are in place.

The hardest governance question is also the one that often gets the least attention: How will an organization prove to a regulator, auditor, or court what an agent did, why it did it, and under whose authority? No regulator will accept “the model decided” as an answer. Healthcare organizations should assume that agent activity records will be examined the way access logs are examined today in OCR investigations, payer and delegation audits, and litigation discovery.

That assumption dictates the design standard. An agent audit trail must be contemporaneous, immutable, and attributable. It must capture what the agent saw, what it did, which version and configuration was running, and which human authority it acted under. If the record cannot support reconstruction of a specific decision months after the fact, it will not support a defense.

Accountability requires the same discipline. Every agent should have a named human owner in the same way every system of record has an owner. That owner should be accountable for the agent’s behavior and outcomes. The agent should also be attached to a role rather than just a specific individual so that ownership survives turnover. Finally, the agent’s owner must be empowered to retire the agent. When a scheduling agent causes a missed visit, the answer to “who is responsible?” should exist before the incident, not be negotiated after it.

Existing law also reaches further into this space than the current conversation often acknowledges. HIPAA’s minimum necessary standard applies to an agent’s data access just as it applies to a person’s. An agent that performs one narrow function but can read an entire record fails that standard by design. When a single agent’s operation spans a model provider, platform vendor, and third-party developer, the business associate chain must match the actual data flow. Procurement categories tend to lag architecture, and in this area that lag is a compliance exposure.

If a new management layer is needed, who is best positioned to deliver it? This remains a live question in the industry, and there are a few contenders:

The challenge is not simply controlling what agents cost. It is also creating a governance model that can scale with their proliferation without slowing innovation. Three principals should apply: 

AI agents will test healthcare’s operational readiness in new ways, and sooner than many realize. The true bottleneck to unlocking value from AI will not be model accuracy or user adoption alone; it will be management complexity. Health systems and payers that get ahead of this challenge can reap the productivity and patient-care benefits of AI safely. Those that lag may find themselves with a hodgepodge of ungoverned bots or stuck in pilot purgatory.

Healthcare executives can start now with a practical set of moves:

The bottom line: Health systems and payers cannot avoid the agent proliferation that is coming. The major software platforms and user demand will drive it regardless. Instead of fighting it, leaders must embrace these tools with eyes wide open to the new governance demands they bring. The organizations that harness dozens of AI agents safely and cohesively will turn hype into real value. Those that do not embrace AI agents may find fragmentation and risk outpacing the rewards. It is time to decide who will oversee the agents now, before they are everywhere.