Executive Summary
The rapid integration of generative AI into enterprise workflows has created a blind spot in corporate governance. When organizations conduct due diligence on AI vendors, the focus is almost exclusively on the "model"—the algorithmic architecture and its versioning. However, recent events, including Meta’s experimental deployment of a "human concierge" service, highlight a critical governance failure: AI services can undergo fundamental changes in privacy, security, and operational authority without a single line of code or model version changing.
Regulatory adviser Maja Kurek argues that for compliance officers and third-party risk management (TPRM) leads, the priority must shift from tracking software versions to tracking the "human layer" of AI services. If a service’s functional reality deviates from the conditions under which it was originally approved, the organization faces significant, often unmonitored, risk.
The Chronology of the "Concierge" Controversy
On September 22, 2026, a Reuters report brought to light a significant internal oversight at Meta. The tech giant had been conducting internal tests of its "Muse" AI agent, augmented by a "human concierge" layer. In this setup, human contractors were surreptitiously handling phone calls initiated by the AI agent without clear disclosure to the users.
- The Discovery: Meta employees raised internal alarms regarding privacy and transparency.
- The Admission: A Meta vice president acknowledged the failure, describing the lack of disclosure as "a miss."
- The Remediation: The feature was promptly rolled back.
- The Context: While the testing was limited to internal staff rather than enterprise customers, the incident served as a wake-up call for the industry. It proved that a service could remain technically "the same" while its operational and privacy profile shifted drastically due to human intervention.
The Governance Gap: Why Model-Change Notices Fail
For years, procurement and legal teams have relied on contractual "model-change notices." These clauses mandate that a vendor inform the customer when the underlying LLM (Large Language Model) is updated—for instance, moving from GPT-4o to a hypothetical GPT-5.
However, as Kurek points out, a model release is only one narrow vector of change. Consider a scenario where an AI service is approved for "redacted output sampling." Six months later, the vendor shifts the role of its external reviewers: they are now using full, unredacted records to resolve customer cases. The model is identical. The vendor is the same. But the data exposure surface area has expanded exponentially.
The "Role, Not Roster" Philosophy
Tracking every individual contractor or employee at a vendor is an administrative nightmare and a futile endeavor. Staff turnover is a standard operational reality. Instead, governance must focus on the role. If the authority, access, or essential function of the human layer changes, the compliance status of the AI service must be re-evaluated.
Supporting Data and Risk Factors
Research from Gartner suggests a significant disconnect between business units and compliance functions. When relationship owners—the staff who manage the day-to-day interactions with vendors—observe a change in the scope of a service, only 52% report that change to the compliance or risk team.
This implies that nearly half of all material changes to AI-enabled services go unvetted. These changes often fall into three categories:
- Access Escalation: When a human intermediary moves from metadata to PII (Personally Identifiable Information).
- Authority Shifting: When a human role moves from a consultative position to an execution position (e.g., the ability to trigger a transaction).
- Dependency Growth: When a human-in-the-loop process moves from "experimental" to "critical path," creating a massive business continuity risk if that human network is disrupted.
Structuring Governance: The Four Pillars of Approval
To bridge the gap identified by the NIST AI Risk Management Framework (specifically the MANAGE 3.1 guidance), organizations should document four specific "facts of the role" at the point of initial service approval:
- Scope of Access: What is the exact data set the human role can view? (e.g., redacted vs. raw records).
- Scope of Authority: Does the human role have the power to execute, approve, or alter customer outcomes?
- Nature of the Human-AI Interaction: Is the human role incidental (monitoring) or essential (co-processing)?
- Operational Handoffs: Does the human role depend on third-party subprocessors, and what are the exit strategies if that team is replaced?
By recording these facts, the organization creates a baseline. If, at any point, the answer to these four questions changes, it triggers an automatic review by the risk team. This is not a "full vendor audit," but a surgical review of the specific, changed component.
Implications for Legal and Procurement Teams
Legal departments must update their standard service-level agreements (SLAs) to reflect this reality. Current contracts are often hyper-focused on technical uptimes and versioning. Future-proofed contracts should include:
- Expanded Notice Obligations: Vendors must provide notice not just for model changes, but for material changes to human-in-the-loop workflows.
- Definition of Materiality: Contractual language must define what constitutes a "material change" in the human layer, ensuring the vendor cannot unilaterally decide that a change in privacy protocols is "minor."
- Audit Rights: Procurement teams must ensure that "right to audit" clauses extend to the human-process workflows, not just the software logs.
Integrating with Internal Audit (The IIA Standard)
The Institute of Internal Auditors (IIA) introduced new third-party topical requirements effective September 15, 2024. This provides a perfect framework for testing these risks. Instead of asking "Is the AI safe?", internal auditors should ask:
- Does the service delivered today match the documentation at approval?
- Where a deviation exists, was it escalated to the appropriate risk owner for a documented decision?
If the approval record states that "Human A" performs "Task B" using "Data C," and today "Human D" performs "Task E" using "Data F," the internal auditor does not need to be a machine learning expert to identify a compliance failure. They simply need to see the "gap" in the decision-making record.
Conclusion: The Human Element in a Machine-Driven World
The "Meta Concierge" incident demonstrates that while we focus on the black box of artificial intelligence, the most dangerous variables are often the humans operating behind the curtain. Organizations that fail to account for the human layer of their AI vendors are leaving themselves exposed to privacy breaches, regulatory fines, and reputational damage.
Effective governance is not about building a separate, bloated "AI-governance workflow." It is about embedding these four simple factual checks into the third-party risk management processes that already exist. By tracking the role rather than the roster, and ensuring that business owners have a clear, actionable trigger for reporting changes, companies can maintain control over their AI deployments—even when the technology itself remains an evolving target.
In the final analysis, the most sophisticated AI service is only as secure as the human processes that support it. If those processes shift, the governance must shift with them.
