ChatGPT struggles with coding problems past its 2021 cutoff date
OpenAI’s ChatGPT was good at solving coding problems before its training cutoff date of 2021, but the chatbot struggled to solve problems after that year, according to a study published by IEEE Transactions on Software Engineering.
ChatGPT’s ability to solve coding problems dropped by a wide margin if the problem set was outside its training data as language model was unable to even understand the meaning of basic problems, the study noted.
The AI chatbot’s success rates of 89% (easy), 71% (medium) and 40% (hard) for coding problems dipped after 2021 to 52% (easy). For hard problems, the rate fell to 0.66%.
The study covered 728 coding problems from Leetcode testing website in five programming languages -- C, C++, Java, JavaScript and Python. The chatbot’s prowess for tested on functionality, complexity and security.
Tang said it was important to understand exactly where these AI models were limited so coders using them could find ways to work around it. “For example, when encountering more complex programming problems, developers can provide relevant knowledge as much as possible, and tell ChatGPT in the prompt which potential vulnerabilities to be aware of,” he added.
)
)
)
)
