Questions about bias audits

An outside check on whether a piece of software treats some groups differently from others. For a hiring tool it means working out how often each group gets picked. We compare those rates, then test whether the gaps are big enough and steady enough to point to discrimination.

No employment relationship, no financial stake in the tool being audited and no consulting engagement that would be threatened by an unfavourable result. VerifyWise takes no equity or revenue share in the systems it audits. An audit run by the vendor's own team is internal testing, which is useful but is not what these laws mean by an audit.

Ask four things. Whether they hold equity, revenue share or a remediation contract tied to the system, because any of those compromises the independence the law is asking for. Whether the signed report names them as auditor of record. Whether the method is fixed before they see your data. And whether they will show you a redacted sample report before you commit, which is the fastest way to tell a statistical audit from a checklist.

A report is accepted on what it contains, not on who wrote it. For Local Law 144 that means selection rates, impact ratios, every sex-by-race combination, the date you began using the tool and any excluded category with the reason. Our reports carry that as standard. No auditor can promise a regulator will never ask a follow-up question, and treat anyone who does with suspicion.

About two weeks for one tool, if the data is clean. A few days to hand over the data and agree the scope. Roughly a week of testing. Then a review of the method and the results before we sign the report. More models or more settings take longer.

Quoted per engagement and scaled by the number of systems in scope, the jurisdictions you need covered and the state of the data. Talk to us with the tools you need audited and we will scope it.

Applicant-level records covering the outcome the tool produced and the demographic categories the relevant law names. Where you hold no demographic data, we will discuss what the law expects and whether inference or a candidate self-identification flow is appropriate for your case.

Three artefacts. The signed report naming the auditor of record, the system audited, the data used, the method and the results. The methodology notes recording scoping decisions, thresholds and any excluded group with its justification. And a publication-ready summary where a jurisdiction requires posting. Keep the methodology notes even where nothing obliges you to publish them, since they are what answers a regulator asking how a ratio was derived.

Selection rate per group, impact ratios comparing each group against the most-selected one, and the four-fifths threshold as a screen. Significance tests then check whether a gap is large enough and stable enough to mean something rather than being an artefact of a small sample. Scoring tools use scoring rate against the sample's median instead of selection rate. Every combination is tested, not only the largest groups.

Usually. The maths is much the same across Local Law 144, Colorado, Illinois, California, Title VII and the EU AI Act, so one round of testing can feed several reports. What changes is the report format, which groups are covered and whether you have to publish.

You get the finding before a regulator or a claimant does, which is the point of testing. The report sets out where the disparity sits, how large it is and which features drive it. Remediation is a separate conversation, and remediation work does not change who signs the audit.

Yes. The statistical core carries over to lending, insurance pricing, housing, education admissions and any other consequential decision. What changes is the legal frame around it. Lending models sit under ECOA and the Fair Housing Act, where the disparate impact framework is long established and adverse action notices bring their own explainability duty. Healthcare and insurance pricing carry state rules that vary. Hiring is where the law is most explicit today, which is why the regulation pages lead with it.