TL;DR
- AI governance is not a committee — it's five working answers: what data AI can touch, who can access what, what gets logged, what your vendors may do with your data, and where humans must review.
- Your existing data classification is the foundation. If you wouldn't email a document to an outside contractor, it shouldn't flow into an unapproved AI tool either.
- Vendor terms matter more than model choice: training-on-your-data clauses, retention, DPAs/BAAs, and security attestations belong in writing before rollout.
- Human review is non-negotiable wherever an error is hard to reverse or lands on a person — money, contracts, hiring, health, anything customer-facing.
- Governance questions are a core section of MadXR's AI Readiness Audit ($4,500) — resolved before a build, not after an incident.
Every company rolling out AI eventually has its governance conversation. The lucky ones have it before deployment, in a conference room. The unlucky ones have it after an employee pastes a customer list into a free chatbot, or an AI-drafted email quotes a price that doesn't exist. This guide covers the basics that separate the two groups — in plain language, without pretending governance requires a task force.
What AI Governance Actually Means
Strip away the buzzwords and AI governance is a set of standing answers to practical questions: What data may AI systems touch, and under what conditions? Who is allowed to use which tools? What records do we keep of what AI did? What have our vendors promised about our data? And where must a human stay in the loop? An organization that can answer all five, in writing, has governance. An organization with a policy PDF nobody follows does not.
The good news: you are not starting from zero. Your existing security practices — data classification, access control, vendor review — extend naturally to AI. The work is applying them to a new category of tool, not inventing a discipline.
Data Boundaries: Decide What AI Can Touch
The first and most consequential decision is a data one. A simple three-tier classification serves most mid-sized organizations:
- Open: public or harmless internal content — marketing copy, published docs, generic templates. Fine for any approved tool.
- Restricted: internal business data — financials, strategy, code, customer records. Allowed only in tools with contractual protections: no training on your data, defined retention, access controls.
- Prohibited: regulated or high-sensitivity data — health information, payment data, credentials, anything a client contract forbids sharing. Allowed only in systems explicitly approved for that data class, or not at all.
The heuristic that survives contact with real employees: if you wouldn't send it to an outside contractor without a signed agreement, don't put it into an AI tool without one either. Make the approved-tool list short, visible, and easy to add to — a fast approval path is the best defense against shadow AI use.
Access Control and Least Privilege
AI systems should inherit the permissions of the person using them — never exceed them. This sounds obvious and is violated constantly, usually by giving an assistant a service account that can read everything so it "just works." An assistant grounded in company documents must respect document-level permissions, or it becomes a search engine for secrets: ask it about salaries, and it helpfully summarizes the HR folder.
The same principle governs AI agents that take actions. An agent should hold the narrowest permissions that let it do its one job — draft but not send, stage but not pay, propose but not delete. The engineering side of this is covered in our companion piece on LLM security best practices, which pairs with this article: governance decides the rules, engineering enforces them.
Audit Trails: If AI Did It, You Can Show What Happened
When an AI-assisted decision is questioned — by a customer, an auditor, or a court — "the AI did it" is not an answer. The systems you deploy should log what was asked, what data was retrieved, what was produced, and who approved it. For assistants, that's conversation and retrieval logs with retention aligned to your records policy. For agents, it's an action log: every step taken in an external system, timestamped, attributable, and reviewable.
Logging is cheap at design time and nearly impossible to reconstruct after the fact. It also pays for itself operationally: the same logs that satisfy an auditor are the ones you'll use to debug quality problems and measure adoption.
Vendor Terms: The Contract Is the Control
Most companies don't run their own models; their AI security posture is largely inherited from vendors. Four questions should be answered in writing before any tool touches restricted data:
| Question | What to look for | Red flag |
|---|---|---|
| Training on your data | Business/enterprise tiers that contractually exclude your inputs from model training | Consumer tiers with training on by default, or vague "we may use data to improve services" language |
| Retention | Defined retention windows, deletion on request, clarity on where data is stored | Indefinite retention, or no answer at all |
| Legal agreements | Willingness to sign a DPA — and a BAA where healthcare data is involved | "Our standard terms cover it" for regulated data |
| Security posture | Recognized attestations (e.g., SOC 2), documented access controls, breach notification terms | No attestations and no security documentation available |
Terms vary by vendor and tier and change over time — verify current contracts rather than relying on any summary, including this one.
Where Human Review Is Non-Negotiable
Governance is not about slowing AI down everywhere; it's about knowing exactly where you refuse to remove the human. Our rule of thumb: review is mandatory wherever an error is expensive to reverse or lands on a person. Money movement. Contract language. Anything customer-facing sent under your name. Hiring, promotion, and termination inputs. Medical and legal contexts. Permanent changes to records. Elsewhere — internal drafts, research, categorization with downstream checks — review can be sampled rather than total, and tightened or loosened as measured accuracy earns it.
Note what this framing does: it turns "should a human review AI output?" from a philosophical argument into a risk-based policy anyone can apply. That, more than any document, is what makes governance stick — a theme we expand on in why AI rollouts fail, because rules that teams don't understand are rules they route around.
Start Proportionate, Then Grow
A five-person company and a five-thousand-person enterprise need different amounts of governance, but the same shape: data rules, access rules, logging, vendor terms, and review points. Write the one-page version before your first pilot. Expand it when the second and third systems arrive. Revisit it when regulations or vendors change. Governance that grows with deployment always beats governance imposed after an incident — in cost, in credibility, and in how fast it lets you move.
Frequently Asked Questions
Do we need AI governance before our first pilot?
You need a proportionate slice of it. A single-workflow pilot doesn't require a committee or a forty-page policy, but it does require knowing what data the pilot touches, which vendor processes it under what terms, who can access it, and where a human reviews output before it matters. Those four answers fit on one page, and writing them down before the pilot is far cheaper than retrofitting them after something goes wrong.
What should an enterprise AI use policy include?
A workable first policy covers five things: which AI tools are approved and for what data classes, what must never be pasted into unapproved tools, which outputs require human review before they reach customers or records, how AI-involved decisions are logged, and who to ask when a case isn't covered. Short policies that people actually read outperform comprehensive ones that they don't.
What terms matter most in an AI vendor agreement?
Four clauses do most of the work: whether the vendor trains its models on your data and how to opt out, how long inputs and outputs are retained and where, whether the vendor will sign the agreements your obligations require such as a DPA or, in healthcare contexts, a BAA, and what security attestations the vendor holds. If a vendor cannot answer these in writing, that is your answer.
Where is human review of AI output non-negotiable?
Anywhere an error is expensive to reverse or lands on a person: money movement, contract language, anything customer-facing under your name, hiring and employment decisions, medical or legal contexts, and permanent record changes. Review can be sampled rather than total once accuracy is measured, but removing it entirely is a decision you earn with evidence — never a default setting.