FHIBE: Fairness Evaluation Dataset for Human-Centric Computer Vision

Summary

FHIBE breaks new ground as the first publicly available, consensually-collected, and globally diverse fairness evaluation dataset specifically designed for human-centric computer vision tasks. Created by Sony AI in 2024, this dataset addresses a critical gap in AI development: the lack of ethically sourced, diverse data for testing fairness across different populations. Unlike many existing datasets that raise consent and representation concerns, FHIBE provides researchers and developers with a clean, comprehensive benchmark for evaluating whether their computer vision systems perform equitably across different demographic groups.

What makes FHIBE different

The dataset stands out in three key ways that address longstanding issues in fairness evaluation:

• Consensual data collection: Every data point was collected with explicit consent from participants, addressing growing concerns about datasets built from scraped or non-consensual sources. This ethical foundation makes FHIBE suitable for commercial and research applications without the legal and ethical risks associated with many existing datasets.
• Global diversity by design: Rather than retrofitting diversity into an existing dataset, FHIBE was constructed from the ground up to ensure representative coverage across different demographic groups worldwide. This intentional approach provides more reliable fairness evaluations across populations that are often underrepresented in computer vision datasets.
• Human-centric focus: The dataset specifically targets computer vision applications that involve human subjects—facial recognition, person detection, demographic classification, and similar tasks where fairness concerns are most acute and consequential.

Core applications and use cases

FHIBE enables several critical fairness evaluation scenarios:

• Bias auditing: Test existing computer vision models against diverse populations to identify performance disparities across demographic groups before deployment. This is particularly valuable for companies needing to demonstrate due diligence in fairness testing.
• Benchmark comparisons: Establish standardized fairness metrics across different model architectures, allowing researchers to compare approaches using consistent, ethically sourced data rather than ad-hoc evaluation methods.
• Training data assessment: Use FHIBE as a reference to evaluate whether training datasets have sufficient diversity and representation, helping teams identify gaps in their data collection strategies.
• Regulatory compliance: Provide documentation for fairness testing that meets emerging regulatory requirements around AI bias testing, particularly valuable given the dataset's transparent consent and collection methodology.

Implementation roadmap

Getting started with FHIBE requires understanding both the technical integration and the evaluation methodology:

• Phase 1 - Dataset familiarization: Download and explore the dataset structure, understanding the demographic annotations, data format, and evaluation protocols. Sony AI provides documentation on the specific fairness metrics and evaluation procedures designed for the dataset.
• Phase 2 - Baseline evaluation: Run your existing computer vision models against FHIBE using the standard evaluation protocols to establish baseline fairness performance across different demographic groups.
• Phase 3 - Iterative improvement: Use the evaluation results to identify specific fairness gaps, then iterate on model architecture, training procedures, or data augmentation strategies to address disparities.
• Phase 4 - Ongoing monitoring: Integrate FHIBE evaluations into your model development pipeline to catch fairness regressions as models evolve and to benchmark new approaches against previous versions.

Who this resource is for

• Computer vision researchers developing facial recognition, person detection, or demographic analysis systems who need rigorous fairness evaluation data that won't create ethical or legal complications for their research.
• AI/ML engineers at companies deploying human-centric computer vision applications in production, particularly those in regulated industries or global markets where fairness and bias concerns directly impact business risk.
• AI governance and ethics teams who need to establish standardized fairness testing procedures and demonstrate due diligence in bias evaluation to stakeholders, regulators, or customers.
• Academic institutions and students researching fairness in computer vision who require access to ethically collected, diverse data that supports reproducible research and meets institutional ethics standards.
• Regulatory compliance teams at organizations subject to emerging AI bias regulations who need documented, standardized approaches to fairness testing that can withstand regulatory scrutiny.