Why ChatGPT Won’t Revolutionize Accounting and Assurance

ChatGPT has taken the internet and news cycle by storm. It can write essays, blogs and poetry. It can code. It can help you come up with ideas for your seven-year-old’s birthday party. And, from what we’ve been led to believe, it will transform every profession, including accounting.

But how exactly will accountants use ChatGPT? 

Obvious use cases are for general inquiries as a conversational replacement for a Google search for things like, “Can you depreciate land?” 

However, for seemingly general accounting- or assurance-related queries, ChatGPT’s usefulness is currently limited. For example, if you ask a more realistic question like, “Can you explain unique aspects of Delaware corporate tax?” it will generate such a list. But if you then follow up with a question such as, “Which states have no corporate income tax for corporations that do not have significant business in the state?” — an aspect that was included in the list — ChatGPT inexplicably gives you a list of other U.S. states which does not include Delaware. 

It can certainly help accounting departments and firms generate initial templates for simple business email messages, blogs and client requests. For example: “Can you write an engagement letter to Mr. Smith to complete his 2022 audit engagement?” provides a starting point with appropriate headings and boilerplate text if a previous template isn’t readily available.

It can also help with client market research. Instead of relying on Google searches and manually following multiple links to deliver an accurate market assessment, you can simply give ChatGPT natural directions like, “Describe the construction industry and Skyscrapers Inc.’s position in that market” and receive a useful, context-rich summary.

Data ownership and liability questions

For more complex accounting tasks, though, ChatGPT has definite drawbacks. For starters, it can be confidently incorrect, such as in the tax example above, just like Google’s ChatGPT competitor, Bard (aka: Pied Piper). It’s not the end of the world if a natural language processing (NLP) solution like ChatGPT or Bard flubs a question about the James Webb Space Telescope. But it’s an entirely different matter if it generates a faulty audit opinion — or even is used as a part of the process which results in a faulty opinion.

Another big issue for accounting and assurance professionals is data ownership. Typically, the data these professionals rely on belongs to their clients. Every client gives their accounting or assurance professional access to their private and sensitive data. The professional will add metadata or inferences on top of the data — but the underlying data ownership remains with the client.

In order to train machine learning (ML) models like the one ChatGPT relies on, you need to aggregate a lot of data. And to build effective predictive models, only minor data anonymization is possible. How many clients provide informed consent for their data to be used in large-scale aggregation and algorithm training purposes by their accounting firm’s internal and third-party vendor or software systems?

ChatGPT also raises a host of liability questions. For example, what does informed consent look like for clients and their professional firms in order to enable data aggregation for ChatGPT training purposes? Who owns the intellectual property of ChatGPT’s output, when everything the output is based on is unsourced? If ChatGPT pulls source code from a licensed source, what liability does the company leveraging the ChatGPT output take on?

The future of innovation in accounting and audit

For any innovator in the accounting and assurance space, it’s essential to understand the stringent accuracy requirements of the profession, the importance of client privacy and data security, and the critical role professional standards play in accounting and audit processes. It’s a premise Caseware has always deemed fundamental when developing our solutions.

For example, in 2017, we came out with our own publicly available machine learning model that predicted the likelihood of future financial statement restatements and a forecast range of key financial figure adjustments as a result. We based this predictive model on all SEC filings to date and all subsequent financial restatements on record, analyzing relative industry performance metrics, as well as the language used in financial statements and how they evolved over time. 

Surprisingly, while many predictive factors were identified, it turned out the highest correlation for restatements was the names of the corporation directors, rather than the content of the statements or the language used. We offered this as a free public information service, rather than as a product, because we understand professional accountants and auditors demand stricter accuracy requirements than we are able to provide with this type of offering.

The products we do bring to market help accountants and auditors work more efficiently and improve their clients’ businesses by delivering fresh insights based on accurate, real-world data. Our solutions are backed by industry-recognized security certifications and include controls that help ensure client data can be accessed only by those employees who need to use it. And we work closely with organizations such as the American Institute of Certified Public Accountants (AICPA) and CPA.com to deliver solutions that meet professional standards and guidelines.

The NLP road ahead

So what does the future hold for NLP solutions like ChatGPT in the accounting and assurance industry? 

First, we believe clients of accountants and auditors need to be educated about the value of training machine learning models on aggregated data sets, as well as the importance of providing informed client consent in regard to using their data. By enabling the creation of large-scale models, predictive algorithms will be able to detect all types of errors automatically across full data sets, providing higher-quality audit opinions at lower costs. Additionally, clients could benefit from insightful benchmarking analysis against the data set across far more aspects of their business than is currently possible.

Next, we believe large firms will begin leveraging partners like Caseware to host the data and manage the ML model lifecycle for their wholly managed data sets for the benefit of their informed clients. 

Finally, we see the broader accounting and assurance market benefitting from leveraging models as they mature over time and the development of marketplaces for these models to be used appropriately and with the potential for global scale.

A better question might be: If you’re not already, when are you going to start leveraging this technology of the future?