Methodology

How we assess software. Published, versioned, and applied consistently to every vendor. Currently v0.1.0.

Jobs to be Done

Every assessment is structured around Jobs to be Done — the specific capabilities a buyer needs from a product area. JTBDs are the atomic unit of assessment: each one is independently scored across multiple domains, evidence-graded, and versioned.

How it works

AV defines the JTBD framework for each product area — what buyers need to accomplish.
Each JTBD includes a job statement, scope, buyer context, illustrative approaches, and scoring guidance per domain.
The vendor is assessed against each JTBD with scores across 7 domains and an evidence level.
Rubrics are maintained as versioned Markdown files in a public GitHub repository.
When a JTBD version changes, vendors are invited to retest at a reduced rate.

14 product areas currently defined: Third-Party Risk Management, AI Governance, Audit Management, Business Continuity, Compliance Management, Cyber Risk, ESG, Enterprise Risk Management, Governance, Incident Management, Policy Management, Privacy, Regulatory Change.

Scoring Domains

Each JTBD is scored across 8 domains. This prevents a single score from masking strengths or weaknesses — a vendor might score highly on depth but poorly on usability for non-experts.

📐

Breadth

The scope of JTBD coverage — how many of the defined jobs the vendor addresses within the product area.

📏

Depth

How comprehensively the vendor covers each job — completeness of capability within the scope they address.

💎

Value

The business impact the capability delivers — measurable outcomes for the buyer, not just features on a spec sheet.

👤

Usability (1st line)

How effectively non-expert users — analysts, compliance officers, risk owners — can operate the capability day-to-day.

👥

Usability (2nd line)

How effectively expert users can set up methodology, push out change, and get answers to questions — without needing vendor support.

⚙️

Configuration Simplicity

How easily administrators can set up and maintain the platform without creating another job within the business — directly impacts total cost of ownership.

🏗️

Architecture

The technical foundation — scalability, integration capability, data model quality, and platform maturity.

🔒

Governance

Audit trail, role-based access, approval workflows, and controls around the capability.

Scoring Levels

A 1–5 ordinal scale where each level represents a qualitatively different degree of capability. These are not percentages — they describe architectural depth, not a ranking.

Exemplary

Vendor fully achieves the job beyond baseline expectations. Capability is mature, integrated, and works at scale. Real-world outcomes evidenced with no material gaps. Demonstrates thought leadership in approach.

Proficient

Vendor substantially achieves the job. Core capability is present and works well. Minor gaps do not impair the buyer's ability to accomplish the job. Evidenced through demo or documentation with positive customer outcomes.

Adequate

Vendor achieves the job functionally. Capability covers main requirements but has noticeable gaps or limitations. Less mature than leading vendors. Documentation may be incomplete.

Partial

Vendor partially achieves the job. Relevant capability exists but is incomplete. Significant gaps require supplemental tools or manual workarounds. Limited evidence. Capability only works in constrained scenarios.

Insufficient

Vendor does not meaningfully achieve the job. Capability is absent, negligible, or misaligned with the job requirement. No credible evidence. May be on roadmap but not in product.

Evidence Levels

Every score is paired with an evidence level that communicates how the capability was observed. Higher evidence levels unlock higher maximum scores — you cannot score 5 on something that was only described in a slide deck.

VerifiedMax score: 5

Confirmed through hands-on testing of a GA-released product version by an AV analyst.

ShownMax score: 5

Demonstrated via live product demo, screenshot, or video walkthrough.

ToldMax score: 2

Described by the vendor in conversation, slides, or concept materials. Not independently confirmed.

PromisedMax score: 1

Roadmap commitment — capability is not yet in the product. Scored at maximum 1.

Assessment Tiers

Two tiers of assessment, using the same published rubric. The difference is depth and method of evidence collection.

AV Verified

Evidence Based

Vendor completes a structured questionnaire with supporting evidence. Evaluated by AI agent against the published rubric, with human review. Maximum evidence level: Shown.

AV Assessed

AV Verdict

Full hands-on testing by an AV analyst against the JTBD framework. Published as a full verdict with domain scores, evidence levels, and vendor right of reply. Maximum evidence level: Verified.

Core Principles

These principles govern every assessment we conduct. They are non-negotiable.

Evidence-based scoring

Every score is derived from observable evidence, not vendor reputation or market position.

Vendor-neutral assessment

We assess outcomes, not features. The same rubric is applied to every vendor regardless of size or relationship.

Right of reply

Vendors receive published opinions before they go live and have the opportunity to respond. Responses are visible alongside AV conclusions.

Transparency of methodology

Our JTBD rubrics and scoring criteria are published and versioned in a public GitHub repository.

Dispute process

Vendors have a 14-day review window to formally dispute scores with supporting evidence.

Conflict of interest

All commercial relationships are declared. Assessment fees are fixed and identical for all vendors.

Explore our rubrics

Full JTBD rubrics and scoring criteria are available on GitHub. For specific questions about methodology, get in touch.

Browse Assessed Vendors Get Assessed