Hybrid Confirmation Trees

Should we rely on AI advice or rather combine indepenent human and AI decisions?

Abstract: Combining human and artificial intelligence (AI) provides new avenues for improving decision accuracy, but effective and cost-efficient integration strategies that maintain human involvement in the decision process remain crucial. This paper introduces and comprehensively evaluates the Hybrid Confirmation Tree (HCT), a simple sequential heuristic where an initial human decision is confirmed by an AI and disagreement triggers a second human tie-breaker. Through analytical derivations, we show that the HCT can match and exceed the accuracy of a three-person human majority vote while requiring fewer human inputs, particularly when AI accuracy is comparable to or exceeds human accuracy. Extensive numerical simulations further explore the HCT’s performance landscape, demonstrating that its ability to achieve complementarity — outperforming both individual humans, machines and the majority vote — is maximized when human and AI accuracies are similar and their decisions are not overly correlated. Empirical re-analysis of six diverse real-world data sets (skin cancer diagnosis, deepfake detection, geopolitical forecasting, criminal re-arrest) validates these findings, showing the HCT improves accuracy over the majority vote by up to 10 percentage points while reducing the cost of decision making by 28-44\(\%\). Furthermore, the HCT provides greater flexibility in navigating true and false positive trade-offs compared to fixed human-only heuristics like hierarchies and polyarchies. The HCT emerges as a practical, efficient, and robust heuristic for hybrid collective intelligence while maintaining human agency.