Decoding ChatGPT's Math Woes: A Technical Analysis

0:00Can ChatGPT do math? This is a question a lot of people have had, especially people in school, or even just if you're trying to get some quick back-of-the-envelope math done right on the fly. A lot of people want to know, you know, can ChatGPT do math? It can do coding, it can do writing, it can do a lot of things, but is it good at math? And this is actually one area that ChatGPT is not good at, I would say. In fact, to the point that it's pretty well recognized, it is pretty notoriously bad at math. And the reason for this is because the way ChatGPT works. ChatGPT is a predictive language model, meaning all it's doing is typically predicting the next word that should come in a sentence.

0:39And so it's not really sitting there and computing a lot of what's going on. And again, we also see that problem with fact checking, you know, it'll come up with lots of different facts that are not accurate. Typically, when I find it writing stories, it's going to get the dates wrong, it's going to get the amounts wrong, it's going to get the dollars wrong and so it's definitely not very good at math but the problem is that it can be very convincing in its responses so today we're going to dive into a little bit about how chat gpt works in regards to the math side of things looking at some strengths it has looking at some weaknesses and overall at the end of the day what it can do and then i'm going to run an actual test through chat gpt to see if it's able to solve my math problem test you guys can follow along see if you guys can get it in the back of your head.

1:24So I guess the short answer is yes, ChatGPT can perform basic arithmetic operations such as addition, subtraction, multiplication, and division. However, its mathematical abilities are pretty limited to just simple arithmetic, and it does not extend to more advanced mathematical concepts such as calculus or linear algebra. That being said, if you ask it a question about calculus or linear algebra, it will typically try to answer it. And it's pretty interesting. Just recently, ChatGPT came up with a new update, which they said their goal on that new version was to make it better at doing math explicitly.

2:03That's pretty much the only thing that happened in the update. And yet, even with that, I'm still finding it not doing an amazing job. So that is definitely a downside. But one of the other strengths of ChatGPT is that when it comes to math, its ability to understand natural language inputs and provide an output in a conversational format is kind of useful. So for example, if you were to ask it, what is 2 plus 2, chat GPT would respond 2 plus 2 is equal to 4, right? So you don't necessarily have to put the symbol to a plus symbol in 2. You could just say the word and you could say similar things, right?

2:36You can explain the situation. And theoretically, it's supposed to try and solve that using simple human English. That being said, it's not always very good at that, to be honest. The other thing that they've tried to build into this is the ability for it to handle multi-step calculations and word problems. So for instance, if you were to say, you know, if John has five apples and he gives away two, how many does he have left? ChatGPT should respond, you know, John would have three apples left because it can understand, you know, the text and you can make some multiple step things. That being said, I have seen cases where it was unable to do those types of problems.

3:16Something that's very interesting is if you literally ask ChatGPT, what is your weakness in regard to math? It's going to say something like, one major weakness of ChatGPT is its lack of understanding of mathematical notations and symbols. While it can handle basic arithmetic and operations, when expressed in natural language, it struggles to understand mathematical expressions when in symbolic form. So for example, if a user were to ask, what is the value of 2x plus 3y when x equals 2 and y equals 4, Chad Shippini would not be able to provide a correct answer. So it literally, it explicitly states, you know, it's not able to solve problems that are, you know, using symbols in them.

3:59But if you did ask it that question, it will give you an answer and it will break down the whole answer, whether that's correct or not. You know, I think it's usually not correct, but it literally is trying. So I guess the problem there is there isn't much of a disclaimer on math on ChadGPT and it's typically getting it wrong. So another one of its, I guess, big drawbacks is it's, you know, does not have a good understanding of higher level mathematical concepts. So it can perform basic, you know, operations in math, but it does not have the ability to solve more complex things such as different equations or vector calculus.

4:37So I mean, it kind of limits its usefulness. But all in all, I think it's got a long way to go before it's going to be replacing calculators for people to just talk to. So I actually went ahead and tried giving it a math problem that I heard someone online in a forum say it struggled with. And my math problem was, if it takes five machines, five minutes to make five devices, how long would it take 100 machines to make 100 devices? And this is kind of like a silly word problem that if someone wasn't really paying attention they'd say five machines, five minutes, five devices, 100 machines, 100 minutes, 100 devices.

5:13I guess that's like the trick you're trying to get someone to say. In reality, five machines in five minutes making five devices just mean that each machine takes five minutes to make a device. So obviously that would mean that it would take 100 machines five minutes as well to make 100 devices. Nothing would change. but if you ask chai chippie t this is the response it gave me it says if mich if five machines can make five devices in five minutes then we calculate the rate of production for each machine five devices divided by five minutes equals one device per minute since the rate of production is constant for each machine we can use this value to calculate the total production rate of 100 machines 100 machines times and then it's got in brackets one device divided by a minute equals 100 devices per minute.

5:59So if 100 machines were working together, they'd be able to produce 100 devices in one minute, which is not true, it would actually take five minutes. So, you know, it's not and it's not like a super complex problem. So I guess I just kind of gives a little bit of an overview of chat GPT and math, it can do basic stuff. I said, what's 100 plus 234. And it said, the results of 100 plus 234 is 334. But like, honestly, that's like, the most basic math problem you could get you type the same thing in google and it would be able to give it to you you know just with their like on in search calculator function so honestly obviously not very impressive um so i guess yeah in conclusion on this whole thing chat gpt has some mathematical abilities they're pretty limited to basic arithmetic operations and very simple word problems but it lacks the ability to understand high level stuff it lacks the ability honestly to even do a lot of several word problems and so it's really not a good tool for math all in all I would say they got a long ways to go they obviously are aware of it they just did a big update so it will be to be determined how they improve this in the future