ChatGPT falls short in accounting exams, students outperform OpenAI’s chatbot

A group of researchers from Brigham Young University (BYU) and 186 other universities conducted a study to determine how OpenAI’s technology would perform on accounting exams. The results of the study were published in the journal Issues in Accounting Education.

According to the study, students scored an average of 76.7 percent on the accounting exam, while ChatGPT achieved a score of 47.4 percent.

The study revealed that ChatGPT outperformed the student average in 11.3 percent of the questions, with particularly strong performance in accounting information systems (AIS) and auditing. However, ChatGPT’s performance was weaker in tax, financial, and managerial assessments. This could potentially be attributed to the bot’s difficulty with the mathematical calculations required for these types of questions, according to the researchers.

Moreover, the study found that ChatGPT, which utilizes machine learning to produce natural language responses, performed better on true/false questions (68.7% correct) and multiple-choice questions (59.5%), but struggled with short-answer questions (with correct answers ranging from 28.7% to 39.1%).

Furthermore, the researchers observed that higher-level questions were more challenging for ChatGPT to answer, and it sometimes provided detailed responses for incorrect answers or responded inconsistently to the same question. Despite frequently providing explanations for its answers, ChatGPT occasionally chose the wrong multiple-choice response despite offering accurate descriptions.

It was also found that ChatGPT provided false information, including generating a seemingly valid reference for non-existent sources. Additionally, it made errors in mathematical calculations such as adding instead of subtracting or dividing numbers incorrectly.

To contribute to the ongoing debate on the role of models like ChatGPT in education, David Wood, a BYU professor of accounting and the lead author of the study, enlisted numerous professors to compare ChatGPT’s performance with that of university accounting students. 

The social media pitch by his co-author garnered a huge response, with 327 co-authors from 186 educational institutions in 14 countries contributing 25,181 classroom accounting exam questions. They also involved undergraduate BYU students who provided ChatGPT with 2,268 questions from various accounting domains of varying complexity and types, including AIS, auditing, financial accounting, managerial accounting, and tax.

(With inputs from PTI)