Hybrid Confirmation Trees

Should we rely on AI advice or rather combine indepenent human and AI decisions?

Abstract: We study hybrid confirmation trees, a simple heuristic for producing hybrid intelligence in melanoma classification. Hybrid confirmation trees first elicit the decision of one human expert and one algorithm. Whenever the two agree a decision is immediately made. In case of disagreement, a second human expert is called in to break the tie. We apply this approach to data on skin cancer detection in dermoscopic and non-dermoscopic images across two studies. Study 1 establishes our approach to be a powerful alternative to human baselines, even outperforming up to three humans combined at lower costs. Study 2 compares hybrid confirmation trees against the industry-standard: humans taking AI advice. Crucially, we find humans supported by AI advice perform worse than hybrid confirmation trees. We attribute this benefit over human-only decision making and AI advice to uncorrelated errors between independent decisions by human and AI. AI advice lead to humans copying AI mistakes while hybrid confirmation trees already exhibit a generally lower agreement on AI mistakes, which can be successfully resolved by human tie breaking. Taken together, our results highlight the potential for combining independent choices of humans and AI for medical diagnostics and beyond as hybrid confirmation trees are a simple but powerful general-purpose method for arbitrating choices.