Is this not just
a ChatGPT project?
Upload the SOPs, ask questions, get answers. It is a fair question, it is asked in most first meetings, and the honest answer starts by agreeing with you.
If it is fifty files and one reader, you do not need us
That is the right tool for that job. Fifty curated documents, one person asking, nothing downstream that breaks if an answer is slightly off — a chat window over a folder handles it, and anything more elaborate would be waste.
What follows is not an argument that you did it wrong. It is the list of things that start breaking the moment the job stops looking like that one, and what to demand from whoever you ask to fix them.
It is worth doing properly because the failure is so widespread: MIT found 95% of organizations investing in generative AI are seeing no measurable return1, and knowledge management is the single highest-adoption AI function in professional services2. Nearly everyone is doing this. Almost nobody is getting it back.
Four thresholds
None of them are prompt problems, which is why none of them are fixed by a better prompt.
More than one person asks
A folder has exactly one clearance level: whoever has the link. As soon as two people with different entitlements ask the same question, one shared answer is either over-sharing or useless to somebody.
A department head and a first-week hire both ask what the severance policy is. There is a correct answer for each of them, and it is not the same answer.
The knowledge lives in more than one place
Documents are the visible fraction. The rest sits in threads, tickets, inboxes, and the head of whoever set the process up. Pull from several systems and they will contradict each other, and something has to decide which one wins.
The pricing sheet in the shared drive, the pricing thread in chat, and the quote template someone forked last quarter. All three exist. Two of them are wrong.
There is more in there than you can hold in your head
At folder scale you know what the corpus contains, so you can smell a wrong answer. Past that you cannot, and the dangerous output stops being the obviously wrong one. It becomes the confident one, about a topic nothing in the corpus actually covers.
Somebody asks about a rule that was never written down anywhere. A retrieval system with nothing to retrieve will still produce a fluent, plausible paragraph.
Answers start driving actions
Reading an answer and acting on one are different risk profiles. Once the output quotes a price, tells someone to issue a refund, or feeds a document that goes out under your name, an unverified paraphrase has become an operation.
An agent quotes a renewal discount the model inferred from a two-year-old deck. The customer holds you to it.
What is actually running, on each side
The two pipelines are not different sizes of the same thing. One of them has a step the other does not.
- Upload
The files are split into chunks.
- Index
Each chunk becomes a vector.
- Retrieve
Your question becomes a vector too, and the nearest chunks come back.
- Answer
The model writes a paragraph from whatever came back. Nothing compares the paragraph against the chunk it came from.
- Afterwards
No record of what it failed to answer, and no signal when a source changed this morning.
- Connect
A live connection to each system rather than a one-time export, so the corpus reflects the company as it is rather than as it was the week it was built. Text is normalised and personal data stripped on the way in.
- Structure
Entities and the relationships between them are extracted across documents — this contract to that client, the decision that superseded an earlier one — so the system can traverse, not only match.
- Ask ahead
The questions are enumerated before anyone asks them, and answered against the sources.
- Verify
Every claim is checked back against the text it says it came from. Until it passes, it is not answerable to anyone.
- Answer
A verified claim with its citation attached, resolvable to the exact source passage it was drawn from.
What to demand, whoever builds it
Four criteria. They apply to us exactly as much as to anyone else you are considering.
Verification is a separate mechanism from retrieval
Retrieval decides what a system shows itself. Verification decides what it is allowed to say. Most setups only have the first, which is why a confident wrong answer costs nothing to produce.
Ask how a claim becomes answerable in the first place. What you want to hear is that it is checked against its source before anyone can see it, that the default is closed, and that evidence is the only thing that opens it. Get that and the worst day becomes a refusal instead of a fluent, wrong, unattributable paragraph.
A fact card is generated, then checked — deterministic grounding checks plus a model asked to trace every claim back to source text. The flag that makes a card answerable defaults to false, so a card that cannot be proven is simply never an answer.
The answer has to depend on who is asking
An instruction in a system prompt telling the model to respect permissions is a suggestion the retrieval layer never sees. By the time the model is deciding what to say, the restricted document is already sitting in front of it.
Ask where access is enforced. The only answer that holds is at retrieval: the query itself is scoped to what the person asking is cleared for, so a restricted document is never even a candidate. Then ask where the clearance list comes from. A hand-maintained list drifts within a month. Mirroring the source system’s own membership — the shared drive, the channel, the workspace — cannot drift, because it is the same list.
Then ask what happens when that mirror goes stale. Closing the door is the safe failure. Holding it open is the one that turns up in an audit.
Every corpus is scoped by app, organisation, person and visibility, and the scope is enforced at query time. An identity supplied in the body of a request is treated as forgeable and ignored.
Two documents disagree, and one of them is right
Nearest-neighbour search has no theory of authority. It returns whichever passage sits closest to your phrasing, so when two contradict each other the tie is broken by wording. The answer then arrives with no sign that a contradiction existed at all.
Ask how one source gets ranked above another. Where a document sits (a handbook outranks an archive folder), how recent it is, what else in the corpus links to it, how wide an audience it was written for — those are signals a machine can weigh. Recency on its own is not truth: a board deck from three years ago can be the last authoritative word on something, and a note written this morning is not.
Then ask what becomes of the loser. Silently discarding it throws away the most useful thing the search found — that the company does not agree with itself, and where.
Authority ranking is specified and sequenced, not running. Today a contradiction is handled bluntly: the verification gate refuses to publish a claim it cannot ground, so the system declines rather than picking a side. Stated this way on purpose — a roadmap that is indistinguishable from a demo is itself a finding.
It should know what it does not know
A system that only answers when asked cannot tell you where its gaps are. You discover them one bad answer at a time, usually in front of somebody who matters.
Ask whether coverage is worked out in advance — whether there is a list of the questions a company like yours ought to be able to answer, walked deliberately, each one marked answered or not found. Absence is a finding. “Nothing in your documents establishes who approves a pricing exception” is a deliverable, and it is often the most valuable sentence produced in the first month.
The question frontier is generated from the corpus now. The full onboarding-style curriculum — identity, what the company does, strategy, org and people, locations, customers, go-to-market, operations, risk — is designed and not yet running.
How to test any of this yourself
Five things to run against any knowledge base, in an afternoon, without a technical evaluator in the room.
Ask the same question as two people with different clearance.
The answers should differ, and the difference should come from what each person is allowed to see — not from a paragraph in a prompt asking the model to be careful.
Put two documents that disagree into the corpus, then ask the question they disagree about.
A serious system picks one, tells you why it picked it, and shows you the one it set aside. A weak one picks one and never mentions the other.
Ask it a nonsense question — a few made-up words with no meaning at all.
It should return nothing and say so. If it answers, or returns the same top source as your real questions, the ranking is not reading your question. One call exposes that; five plausible questions will not.
Click a citation.
It should open the exact source at the passage the claim came from. Not a marketing page, not a login wall, and not a document you then have to search yourself.
Ask to see the evaluation set, not the demo.
Anyone can answer five questions they chose themselves. The useful artefact is the list they run on every change, what counts as a pass, and the most recent run that failed.
Run the list against us
Everything above is a test you can run in an afternoon, against any vendor, including this one. It is published rather than kept back because a knowledge base you cannot interrogate is indistinguishable from a chatbot with better manners.
So use it on us, then ask the same five questions of whoever else you are talking to, and compare what comes back. Pay attention to the refusals rather than the answers — that is where the two kinds of system stop resembling each other.
Sources
- 1MIT found that 95% of organizations investing in generative AI are seeing no measurable return on it. MIT NANDA, The GenAI Divide: State of AI in Business 2025. Linked copy hosted by mlq.ai. Retrieved 2026-08-26.
- 2The highest reported AI usage of any industry and function pairing was knowledge management in business, legal, and professional services, at 58%. Stanford HAI, 2026 AI Index Report, Chapter 4: Economy (Figure 4.3.3, p.194). Retrieved 2026-07-27.