Case study3 min read
A cybersecurity audit platform where the AI writes, but never grades
Paladin Assurance runs DOL EBSA cybersecurity assessments for retirement-plan fiduciaries. Optivantix designed and built the platform end to end — the scoring engine, the roles, the AI report layer, and the verification tooling underneath it.
The problem
The incumbent lets vendors grade their own homework
Retirement-plan fiduciaries have to verify their service providers' cybersecurity under DOL EBSA guidance. The existing option in this market is a self-graded questionnaire — the vendor being assessed also scores itself. Paladin's answer is a published, deterministic rubric that scores every submission the same way, plus an AI layer that only explains the result, never sets it.
Proof, not a pitch
One real assessment, start to finish
This is a real specimen report from the live platform. Step through it — intake, all twelve weighted domains, and the vendor's own explanation for every gap.
All twelve domains — Every domain, each tagged High or Medium weight and scored against how many controls in that domain are actually in place.
Every domain, weighted, explained
High-weight domains like Encryption and Business Resiliency count more than lower-weight ones. Where a control isn't in place, the report doesn't just dock points — it shows the vendor's own written explanation, which is what turns a score into an actual remediation conversation instead of a number nobody can act on.
Score meets override
A high number doesn't save you from a red flag
ASSESSMENT RESULT — REAL DATA FROM THE LIVE PLATFORM
Kingsbridge Fiduciary Services
Escalated to High Risk regardless of the numeric score. One or more Section D red flags were triggered — these override the weighted average per the DOL scoring methodology.
Red flag triggered: participant data not encrypted at rest or in transit (Domain 10).
84.7% would pass almost any weighted average
This is a real result from the live platform: a vendor scored nearly 85% across the twelve domains but is still marked High Risk, because unencrypted participant data is one of Section D's automatic-override rules. The methodology is explicit about this on purpose — a strong average shouldn't be able to bury one disqualifying gap.
(Recreated from the live dashboard for legibility — the score, band, and red-flag reason shown are the real, unedited result for this vendor.)
Beyond self-report
Checking the vendor's claims against their own domain
Real scan, real findings
Alongside the questionnaire, Paladin can run an external scan against a service provider's actual domain — email spoofing protection, DNS exposure, TLS configuration, and standard browser security headers — and translate the results into plain English instead of a raw security-tool report.
(Recreated from a real scan run against a live domain during this project — not a mockup of hypothetical findings.)
External scan
High RiskWhat this service provider already exposes to the public internet.
- Email claiming to be from this company could be forged. SPF missing · DMARC missing
- This domain's internal DNS map is not exposed to outsiders. DNS zone transfer refused
- This site's connection encryption is current and healthy. TLS 1.3
- The website is missing 1 standard browser protection. content-security-policy
How it was built
In the order it actually happened
01
Deterministic scoring engine first
Before any UI existed, we built the scoring engine and its test suite — the twelve DOL EBSA domains, the weighted Section C formula, and Section D's seven red-flag rules. This is the part that can never be an AI's opinion, so it shipped as plain, tested code before anything a customer would see.
02
Real accounts, not a demo shell
Auth (Clerk), a Postgres database (Neon), and payments (Stripe) went in early and for real — not mocked, not stubbed. The first user account, the first saved assessment, and the first webhook all ran against live infrastructure from week one.
03
The assessment itself
A twelve-domain intake form wired to the real scoring engine — every answer computes a live weighted score as you go, not after submission. Later restructured around how the business actually works: vendors are shared across sponsors, so one vendor's relationship with each client had to stay private from every other client.
04
Roles the org chart actually has
A plan sponsor (Owner), a firm the sponsor delegates day-to-day audit management to without giving up ownership (Manager), and the vendor being assessed, reached through a tokenized link with no account required (Service Provider). Three account shapes, not one generic “user.”
05
The AI layer — after the rules were frozen
Only once the scoring contract was locked did an AI report-writer go in. It receives the computed score and writes the prose explaining it — it never touches the arithmetic. Every recommendation it makes carries the exact question number it's grounded in, and a validator rejects output that invents a question, contradicts a red flag, or states a risk band the engine didn't compute.
06
Proof beyond the form
An external attack-surface scanner was added so a service provider's self-reported answers can be checked against what their domain actually exposes — SPF, DMARC, TLS configuration, DNS zone-transfer exposure, security headers — read directly from the public internet, not asserted.
What makes it different
Six decisions that shape the product
The AI never computes the score
Every claim on the platform traces to a rule, not a model's judgment. The scoring engine is deterministic code — the same twelve answers produce the same score every time, and a sponsor can check the arithmetic by hand. The AI's only job is writing the explanation after the number already exists.
Red flags override the average
A strong weighted score doesn't matter if multi-factor authentication is missing — that's an automatic-fail rule, not a deduction. The platform shows its work: the numeric score, the override, and the specific rule that fired, all in one place.
Reports are grounded, not generated
The AI report-writer can't invent a finding. Every recommendation carries the exact question number it came from, and a validator rejects any output that references a question that doesn't exist, contradicts a triggered red flag, or states a risk band the engine didn't produce.
Verification, not just self-report
A built-in scanner checks what a vendor's domain actually exposes to the public internet — email spoofing protection, DNS exposure, certificate health — and surfaces it in plain English next to their self-reported answers.
Delegation without losing ownership
A plan sponsor can hand day-to-day audit management to a law firm or IT consultancy without transferring ownership of the assessment itself. The org chart most fiduciaries actually have, not a single admin account shared over Slack.
A vendor never needs an account
Service providers respond through a single tokenized link — no sign-up, no password, no seat to provision. The friction that kills third-party assessments in practice is the account creation step, so there isn't one.
Under the hood
The stack
AI for the report layer, routed through Vercel's AI Gateway, model pinned and never a floating alias, called once per report at temperature zero.
- Next.jsApplication framework
- VercelHosting & deployment
- NeonPostgres database
- ClerkAuth & organizations
- StripeBilling
- AI GatewayReport generation
“This is the same pattern behind Showroom Lens — a narrow, specific problem, solved with as much determinism as the problem allows, and AI applied only where it earns its place.”