AI advice or independent aggregation?

Should we rely on AI advice or rather combine indepenent human and AI decisions?

Abstract: Artificial Intelligence (AI) systems are usually deployed as advisers: a system suggests a decision that a human may accept or reject. This influence of humans during their decision making process hinders a clear attribution of errors to human or machine and risks human automation bias and potentially deskilling humans. We evaluate an alternative to combining human and AI decisions: the hybrid confirmation tree — built on the principle of independent judgments. The hybrid confirmation tree elicits one human and one AI choice independent of each other. When they agree, that decision is accepted. If not, a second human breaks the tie. We compare this approach to the AI-as-advisor approach across 10 data sets, from a wide range of domains, including medical diagnostic and misinformation judgments. The hybrid confirmation tree outperforms the AI-as-advisor approach in all data sets. The benefit stems from the hybrid confirmation tree capturing a larger share of correct AI choices than advice-takers. Similar performance gains above AI-as-advice were observed even when AI advice was explainable unless humans alone were at chance performance. While the hybrid confirmation tree may cost more human decisions, this additional cost can be offset by the increased accuracy we observe. Overall, hybrid confirmation trees provide a robust, accurate and transparent alternative to the AI-advice, and offers a simple mechanism to tap into the wisdom of hybrid crowds.