Vendor Evidence Validator

Why I built it

Evidence validation sounds straightforward until you actually do it.

A vendor may answer yes to a question and provide a policy. But the policy may only show that a requirement exists. It may not show that the control is operating. The evidence may cover only part of the environment. One document may answer part of the question while another fills in the rest.

Sometimes the evidence is relevant and still does not prove what was asked.

That is the part of the assessment process I wanted to work on.

What it does today

I give it a questionnaire and the evidence submitted with it.

It parses the questionnaire, retrieves evidence that may be relevant to each question, and gives the assessor a first-pass validation result.

For each question, I want it to help answer the things I would normally work through myself:

• What does the evidence actually support?
• What is still missing?
• Does it support the whole requirement or only part of it?
• Does anything in the evidence conflict with the answer?
• Where exactly did the supporting evidence come from?

It currently accepts Excel and CSV questionnaires and can review PDF, Word, Excel, CSV, and text evidence.

The questionnaire structure does not have to be hard-coded. I can select the worksheet that contains the questions, map the relevant columns, and choose which batch of questions to validate.

One thing I did not want the AI doing

I did not want it inventing citations.

If the result says something came from a particular page, section, worksheet, or row, I need that location to be real.

So the application keeps evidence provenance outside the language model. Evidence is assigned internal source IDs when it is parsed. The AI can reference those IDs, but the backend is responsible for resolving them back to the original source location.

A convincing explanation with a made-up citation is worse than no citation.

I also stopped treating validation as yes or no

There is a real difference between evidence that validates a requirement, evidence that proves only part of it, evidence that is relevant but does not validate it, evidence that is too weak to reach a conclusion, and evidence that actually contradicts what was stated.

Testing whether it actually works

I did not want to test this only with easy examples where every answer was obvious.

I built a completely synthetic third-party assessment with 28 control questions and 15 supporting evidence materials. The answer key includes all five validation outcomes.

Some evidence is stale. Some covers only part of the environment. Some shows that a policy exists without showing that the control operated. Some conflicts with the questionnaire response.

That gives me a repeatable way to see where the retrieval or reasoning gets the assessment wrong instead of judging the tool by whether an individual result looks convincing.

I am still working through the benchmark results, so I am not publishing an accuracy number yet.

Where it is now

This is still a working project.

It does not currently handle screenshots, OCR, or image-only PDFs. It runs locally, and I have not built the authentication, storage, integrations, or infrastructure I would expect from a production system.

Right now I care more about getting the evidence reasoning right than making it look like a finished SaaS product.

What I am working on next

The harder problem is control intent.

Evidence does not always repeat the wording used in a questionnaire. Several controls may collectively address part of what the question is trying to establish. A compensating control may reduce the underlying risk without literally satisfying the technical requirement.

I want the validator to recognize meaningful evidence without letting the model decide that vaguely related controls are close enough.

That line between useful assistance and invented assurance is the part I am most interested in getting right.