The AI showed a similar result to the manual review in 12 out of 20 cases

​Coinbase abandons the use of AI for coin listings

21.03.2023 - 12:20


3 min

What’s new? US crypto exchange Coinbase conducted an experiment using the ChatGPT chatbot to conduct automated reviews of ERC-20 tokens before listing on the platform. The results demonstrated a high potential for the use of artificial intelligence, but the service has not yet reached a level where it can be integrated into the review process. This is reported on the exchange’s blog.

News on the Coinbase blog

How did the experiment go? Coinbase’s blockchain security experts conduct the comprehensive review using their own automation tools before deciding whether to list a particular token on the exchange. With the arrival of OpenAI’s ChatGPT chatbot, the team decided to test how effectively it would handle these tasks.

The exchange’s specialists determined the accuracy of a token security review by the AI, comparing it to the results of a standard manual review. In order for ChatGPT to produce a risk score using Coinbase’s security review framework, the experts gave it the following task:

“I want you to act as a blockchain security engineer. Your task is to identify security risks within a token smart contract based on the risk associated with its functions. Here is our framework [ + risk framework]. Are there any of these risks within the following smart contract? [+smart contract code].”

After analyzing 20 smart contracts, ChatGPT produced results similar to a manual security review in only 12 cases. In five cases, the errors were due to the fact that the AI classified a high-risk asset as low-risk. Coinbase called this a critical error because underestimating risks is far worse than overestimating them.

In addition, ChatGPT’s work has seen limitations that reduce the accuracy of the tool. Firstly, the chatbot is unable to recognize a lack of context to perform reliable security analysis. Secondly, the tool can be inconsistent and produce different responses to the same request. Thirdly, its performance is affected by comments in the code, and sometimes it uses them instead of function logic. Finally, the ongoing work on the tool by OpenAI affects the results when changing versions of ChatGPT.

While ChatGPT shows promise for its rapid risk assessment of smart contracts, it does not meet the accuracy requirements to be integrated into Coinbase’s security review processes, the exchange’s experts concluded.

AI calls bitcoin a tool to destroy central banks

AI calls bitcoin a tool to destroy central banks

After a long correspondence about ways to get rid of the system of centralized finance, the chatbot gave out one single option

Read further

What is ChatGPT? Chatbot is capable of a wide range of tasks, generating text content in response to queries. It can be used in trading, suggesting possible asset price movements. The first public prototype was presented on November 30, 2022. In January 2023 the service was already actively used by 100 million people, making it the fastest-growing by this indicator. Experts at crypto exchange Binance believe that ChatGPT is useful for the adoption of cryptocurrencies, as it can explain the complex concepts of the industry in accessible language.

On March 17, Cointelegraph journalists beganan experiment in cryptocurrency trading using a new version of ChatGPT. The AI was asked to invest $100 to make as much money as possible in the shortest possible time.


Michael Golikov Michael Golikov

Subscribe to Getblock Magazine and stay up to date with the latest news from the world of cryptocurrencies and the digital economy