Trusting ChatGPT: the water bottle error that exposes its limits

Trusting ChatGPT: the water bottle error that dismantles confidence

Last week, a user posed an elementary physics problem to ChatGPT: how much does the water in a commercial 33-centilitre bottle weigh at 80 degrees? The answer, calculated with apparent precision, was 321 grams. The problem is that the question was a trick: the bottle is sealed, so the mass does not change with temperature. The model, however, not only got it wrong, but offered an incorrect explanation and even volunteered to detail why the water 'loses' almost 9 grams as it cools. A glaring error that serves as a metaphor for a broader problem: blind trust in generative artificial intelligences.

Why a probabilistic model is not a truth engine

ChatGPT is not a database that looks up correct answers. It is a system that predicts the next most likely word based on context. This means it can generate text with total confidence, even when the content is false. In the bottle case, the model applied the density formula for water at 80 degrees (0.972 g/mL) and calculated 321 grams, ignoring the fact that mass is independent of temperature in a closed system. AI does not reason: it reproduces statistical patterns. And when the pattern is wrong, the result is an elegant but incorrect answer.

The time-of-day factor: does it affect response quality?

One hypothesis circulating among advanced users is that ChatGPT's quality varies depending on the time of day. The argument goes that between midnight and 6 a.m., during off-peak hours, the model accesses more powerful versions with a larger context window. In contrast, during peak hours (from 6 a.m. to 10 p.m.), lighter and more generic models would be assigned, which would explain more superficial or downright stupid answers. There is no official evidence, but the observation is recurrent: at 3 in the morning, ChatGPT seems to remember previous conversations better and offers more refined answers. At midday, it becomes a generic 'sibling' that states the obvious.

The need for verification: prompt engineering is not infallible

Prompt engineering —providing context, requesting a format, limiting the scope— reduces uncertainty but does not eliminate it. A good prompt can prevent the AI from making things up, but it does not guarantee it won't make mistakes. The water bottle error is a perfect example: the user did not write a bad prompt; they asked a trick question that the AI failed to interpret. The conclusion is that human verification remains essential. Anyone who relies on ChatGPT for calculations, data or analysis without cross-checking is taking on a risk they shouldn't.

The risk of overconfidence: when AI makes things up about people

Another revealing episode: when asked about user profiles from online communities, the model produced generic and, at times, fabricated summaries. Instead of acknowledging that it had no information, it manufactured a plausible characterisation. This is especially dangerous when AI is used as a source to learn about someone's fruta or to make decisions based on unverified information. AI does not distinguish between what it knows and what it imagines; it simply generates coherent text.

The conclusion is uncomfortable: AI is a useful tool, but not an oracle. The water bottle error is trivial, but it reveals that trust must be conditional. The system works because there is a human who corrects it. Without that correction, AI is just a generator of plausible text, not of truth.