How to Ask a Vendor What They Did NOT Check

Three questions, one message, and a decision about what was left out

You leave this guide with a written message, ready to send, and a way to read what comes back. That's all. It doesn't teach you to audit anyone, and it can't tell you whether a vendor is telling the truth. It gets you, in writing, what you need in order to decide.

This reading doesn't judge the person making the claim. It measures one thing: which of your three questions already have an answer in what you were handed. A claim can be complete and wrong, or open and right. What changes is how much weight you can put on it without knowing.

If you haven't read the article that goes with this guide, start there. This is the part that didn't fit: the questions with their edge cases, the message line by line, how to read the reply, and what to do with what was left out.

A claim is a phrase someone hands you so you can lean on it. "Verified." "Reviewed by our team." "Certified." "Tested." "Approved." "Safe." It might come from a software or AI vendor, a contractor, a bank or an insurer, a seal on a website, or your own team when they say "I already looked at it."

The word sounds like a measurement and isn't one. A measurement has two parts: what was measured, and what it was compared with. The word "verified" on its own carries neither, and even less the edge: what wasn't measured.

That gives you the three questions. They're always the same, in the same words:

1. What exactly was checked?
2. What was it compared against?
3. What was left out, and who says so?

And a fourth, only when it applies: if the claim is in one language and your work or your contract is in another, was it checked in both?

The answer has to be written by whoever makes the claim. If you fill it in yourself, from what you assume, you haven't read the claim. You've completed it.

One example of someone who does write all three. On October 22, Anthropic published research on formal verification of code. The page itself says "formally verified" does not mean "bug-free," and that they didn't verify all the entry points for each library. That doesn't mean it applies to your business. It's one case where the edge is written by the person who drew it. The usual situation, when you get a bare word, is that the edge isn't written anywhere.

"Exactly" means by name and with scope: which part, how far.

The tool that goes with this guide has three possible answers:

• By name and scope. "We reviewed all 40 pieces in the fall catalog, one by one, against the materials sheet." You know what went in.
• General. "Everything has been reviewed." It sounds like a lot and doesn't say what "everything" is.
• They don't say. The word shows up alone.

Imagine a watch boutique that sends you a report with the word "authenticated" on every listing. What you need to know is what got authenticated: the case, the movement, the papers, all three? An answer of "the watch" is general. An answer of "the movement's serial number against the maker's registry, and the case from the outside, not opened" is by name and with scope, and it already half-answers question 3.

A common edge case: they reply with a long list of things that were done, but none of them is what matters to you. That's a named answer to a different question. Note what they said and ask again about yours.

To check is to compare. If they tell you what against, you can go and look at it.

Three possible answers:

• Against a source or standard they name and you can open. A standard, a registry, a contract, a document. This is the only one that counts as complete, because it's the only one you can look at yourself.
• Against their own criterion, which they don't show you. "We checked it against our internal standards." It isn't a bad answer. It's an incomplete one: it tells you there was a criterion and doesn't let you see it.
• They don't say what against.

Imagine an event producer who tells you the setup is "approved" for your conference. Approved against the room's capacity, against a list of requirements you put in the contract, or against what they usually do? All three are comparisons, but you were only in one of them.

Open whatever they name. A source appearing in the answer doesn't make it true or applicable. The tool marks "named" when they name one; you're the one who opens the document and sees whether it says what they say it says.

One nuance Anthropic's page leaves in plain view, and it isn't a criticism: the specifications the proofs were checked against came out of tests the developers already had, and the page says a proof checker "can't tell you whether your spec matches reality." That's question 2 in its cleanest form: a check is worth what the thing it was checked against is worth.

This is the one that goes unanswered unless you ask, and the one that helps you most. Every check has an edge. What varies is whether anyone wrote it down.

Three possible answers:

• They say it themselves. The person making the claim tells you what didn't go in.
• A third party says it. Someone else, another source, a separate report.
• Nobody says it. Not in anything you were handed.

The first two count as an answer. The third is the gap.

An example with a source. On October 22, METR published a pilot stress test of the system OpenAI uses to monitor its internal agents. On that page METR writes that its findings describe the system as it was "on and before July 21st," that OpenAI redacted exploit details and that a redacted 38-page version went to the leadership teams at METR and Redwood, that they're publishing only a subset of the report, and, about what OpenAI says it fixed: "We have not evaluated these claims." There's a written edge here, said by a third party. It isn't a verdict on OpenAI or on METR. It's what you see when someone writes the edge down.

What matters in question 3 is the "who." An edge written by the person making the claim is an honest answer, but it's from one side. An edge written by a third party weighs differently. The tool doesn't rank one above the other: it lets you know which one you have.

If you send it with a tone of complaint, the reply will be defensive. What works is one sentence that says what you'll do with the answer: lean on it.

The message the tool builds always has the same shape, and this guide gives it to you so you can write it yourself even if you don't use the tool.

Opening. "About what we received from a software or AI vendor, marked with the word 'verified': before I rely on it, I need three answers in writing."

Why it works: it quotes the word, says where it came from by type (no names), and says what you want it for: to rely on it. "In writing" matters, because a hallway answer can't be opened or forwarded.

One line for each question still missing. Only the missing ones. If they've already told you what they checked, don't ask again.

1. "What exactly was checked, and with what scope?"
2. "What was it compared against? I need a source or standard I can open." If they told you it was their own criterion: "It was your own criterion: can you show it to me, or point me to a source or standard I can open?"
3. "What was left out of the check, and who says so: you or a third party?"
4. (Only if two languages are involved) "The claim is in English and my work is in Spanish: was the check done in both?"

Closing. "If something was not checked, tell me plainly: knowing is more useful to me than not knowing."

That closing line does the work of lowering the guard. It gives the other person permission to say "we didn't cover this" without it sounding like a confession.

If you don't use the tool: copy the opening and the lines you're missing, change the word and the type of source, and send. If yours is a WhatsApp message to one person, shorten it; what you don't shorten is the questions.

Write in the language of the claim. If it reached you in English, ask in English. The tool builds the message in the claim's language even when you're reading the page in another.

You'll get one of three kinds of reply.

1. It names. They tell you what, against what, and what was left out, even in two lines. Go to Part 8: you now have something to decide with. And open whatever they named.

2. It repeats the word. "Yes, it's verified." "We reviewed it thoroughly." They answered the question you didn't ask. That isn't necessarily bad faith: the person may not have what was done at hand. Ask again about one thing only, the most important, in the same words. If what matters most to you is what was left out, ask only that.

3. It goes silent. No reply. Silence proves nothing: it could be a full inbox, someone on vacation, or nobody knowing. Don't read it as a confession. It is information about how much you can lean on this today: you have an open claim and no way to close it. Decide with that (Part 8).

A reply that names and surprises you (because they left out something you assumed was covered) is the reply that was most worth getting. That's exactly what question 3 was for.

If your vendor sends you the claim in English and your contract, your client or your team work in Spanish, the question is whether the same thing was checked in both languages. It isn't an odd question: a text, an interface, a tool's output or a legal document can work in one language and fail in another.

Imagine a keynote speaker who hires a captioning tool and receives a report in English that says "tested." Their audience is in Spanish. Tested in which language? The answer that helps is "we tested with Spanish audio and text, and here are the cases"; the one that doesn't is "it's a multilingual model," because that's a different kind of claim: it says what the tool can do, not what was checked.

The tool flags the language gap when the claim and your work are in different languages and they don't tell you they checked in both. The reply that closes the question is the one that names both languages, not the one that says "it works in all of them."

You have the answers, or you're missing some. You decide, and the decision depends on what being wrong would cost you. The tool asks for that (low, medium or high) and uses it only to order what to ask first.

Four paths, from loosest to strictest:

• You accept it. What was left out doesn't touch what you're about to do. Write down where the edge is, so you remember when what you do changes.
• You ask for more. There's a part that does affect you and isn't covered. You ask about that part, one thing.
• You use it only where being wrong is cheap. For what you can fix yourself in an afternoon, yes. For what affects a client, a contract or another person, not yet.
• You don't use it. If what was left out is exactly what you need and they can't cover it, that's the answer.

None of them is right in general. The right one is the one that matches your risk.

• It doesn't measure whether the claim is true. Naming a source doesn't make it true: open it.
• It doesn't measure whether the vendor is honest. A vendor can lack what you ask for and still be honest. Another can reply beautifully and be wrong.
• "Complete" isn't "good." It means the questions have answers, not that you like the answers.
• Silence is not proof of anything.
• It doesn't compare vendors or give a score. There is no good rate of complete claims. With one claim and three questions, you can't say anything about anyone else.

One claim, one message. This week you don't need more. If the reply comes, read it with Part 6 and decide with Part 8. If it doesn't, you have one open claim written down, which is already more than you had.

The fill-in note

Copy this note once per claim:

What I received: ______________________ - From whom (type): ______________ - The word: ______________

What was checked (question 1): [by name and scope] [general] [they don't say]

Against what (question 2): [a source or standard I can open] [their own criterion] [they don't say]

What was left out (question 3): [they say it] [a third party says it] [nobody says it]

Language: the claim is in ______ - my work is in ______ - [checked in both] [only in one] [they don't say]

If I'm wrong, what happens? [I fix it myself in an afternoon] [it costs me money or a customer] [it affects another person, a contract or the law]

What I'm going to ask for: __________________________________

What I decide about what was left out: [I accept it] [I ask for more] [I use it where it's cheap] [I don't use it]

What's outside this guide

Counting how much of what AI hands you that you use, and deciding who should review whoever builds. Those are other questions, and this guide doesn't answer them.

The adding up. That's what The Three Questions for "Verified" is for, the tool that comes with this guide: for each claim you mark what they answered on each question, and it hands back what's missing and the message, ready to send.

One last thing

To start, one claim is enough. Find the latest "verified" or "reviewed" that reached you this week, and answer the three questions. If any is left without an answer, you know what to write.

A word tells you someone looked. The three questions tell you at what.

Want help reading your claims? Bring them to a free call and we'll start with whatever you have in writing from your vendor.

Book a free call

On Fridays at 4 p.m. Eastern, The AI Edge goes out: one AI idea you can use that same week. Subscribe here.