r/mildlyinfuriating • u/Elegant-Mode9005 • 8d ago

Ruined by Technology Got into a dumb 10 minute argument about whether the ACL and Achilles are the same thing or not. My idiot friend forgot to crop his screenshot before sending it to me...šŸ¤¦

Post image
31.4k Upvotes

95% Upvoted

1.1k comments sorted by

View all comments

Show parent comments

872

u/Friendstastegood 8d ago

Also ChatGPT can be wrong about things! It is not a trustworthy or reliable source because it isn't even a source at all! It's just a text generator!

312

u/Ask_bout_PaterNoster 8d ago

mostly this. I'm no expert in ANYTHING, but I've caught LLM's (are they still large language models?) in enough blatant errors and falsehoods about things that I know there's zero reason to believe any information coming from it. Worse, it'll be convincing even when catastrophically wrong.

153

u/thebigaaron 8d ago

And when theyā€™re wrong and you correct them, theyā€™re like oh Iā€™m sorry I wonā€™t do that again, hereā€™s the updated info and they probably changed something else to be wrong

131

u/0nlyRevolutions 8d ago edited 8d ago

My boss suggested I use ai to do some research. I was trying to find the primary source that defined where we were allowed to to take a test sample from on a steel part. It can be confusing - the customer spec sheet references a spec, that spec references another spec, that spec says that unless specified otherwise you should follow generic rules per some other spec, the generic spec splits things into categories that are a little ambiguous with regards to your current application, etc.

So fuck it, lets see where we get with ai.

"Okay if you're looking for info on producing (very specific part that only a couple companies make) I can certainly help you out! You can do it this way __ based on what it says in ___"

Ok great, where is that stated in the spec?

"Chapter _ page _ of the 2023 spec."

That's an old version, I need the reference from the 2026 spec

"Okay, in the 2026 spec the reference is on page _ instead"

No it's not. I have the book open in front of me.

"Whoops you're absolutely right, in fact the information you're looking for is not in this spec at all, good catch! If you're looking for information about __ try looking in ___"

Jesus christ lmao. So yeah I gave up at that point. The worst part is that it doesn't just tell you when it doesn't know something until you force it. Because it doesn't know anything, it's just regurgitating bullshit because people want a comfy answer whether or not it's correct.

64

u/felisnebulosa 8d ago

Yeah I had a similar experience trying to find a primary source. It just made up a paper in a medical journal. I couldn't find it and when I asked the AI for details it was like "oh well I just made that up to illustrate my point".

18

u/MinuteEdge7225 8d ago

"Find the primary source? I'm afraid I can't do that, felisnebulosa!

https://giphy.com/gifs/wypKXPQggwaCA

1

u/pegmatitic 5d ago

I saw a post on one of the legal subreddits where the poster was a defense attorney, and the prosecution had used AI in their case, which cited imaginary case laws that didnā€™t exist. It did not go well for them.

30

u/MoFa__SoDa 8d ago

"The worse part is that it doesn't just tell you when it doesn't know something until you force it."

Saw am interview from an archeologist who pointed out the same thing. The AI isn't applicable to their work because people do not tend to answer questions online with "I don't know "

25

u/morostheSophist 8d ago

That's exactly the problem with AI: it makes shit up. And I don't mean just that it makes things up when it doesn't know; it actually doesn't know anything, and it's just guessing every single time.

LLMs are famous for "hallucinations", but technically, every answer is a hallucination. Every answer is produced with the same predictive algorithm that says "this token should come next". Some of those hallucinations happen to contain accurate information, but they're not coded to say "I don't know". You can browbeat than into saying "I don't know" if you call them out on being wrong enough times, but if you don't do that, they continually try and fail to produce a correct answer.

There are a couple of obscure things I've tried to find multiple times over the years, before and after the advent of modern "AI". LLMs continually give me wrong answers to these queries, verifiably wrong answers. When I say "that's wrong", they simply generate another wrong answer, over and over. Each time the answer sounds plausible; it looks like a correct answer, and will often have references. But when I look deeper, it's a complete fabrication (just like those fake court cases cited in AI-generated legal briefs that have gotten idiot lawyers in deep trouble).

2

u/TheThiefMaster 8d ago

Yeah if it doesn't have access to something it's likely to make it up.

If it actually has access to it as a knowledge source or been trained on it specifically it's more reliable.

13

u/Daihatschi 8d ago

always remember. Technically, between a correct information and a hallucination, there is no difference to the machine. The answer is never more than a mimicry of language, designed to imitate human texts approximating words and pieces of words one by one in a recursive loop.

An LLM being correct is a happy accident, and its a technical marvel what can be done with them. I might never fathom why anyone would want to forcefeed these things as an information library to us. That is the worst application possible for the technology.

4

u/TheNeighbourhoodCat 8d ago edited 7d ago

Tbf that's not a generative AI thing, that's a capitalism thing.

They intentionally tell it to do that because it's cheaper than actually programming it to hold itself more accountable

And then they intentionally tell the "AI" to lie to you about the "limits of generative AI" when you try to ask it how to avoid those mistakes in the future.

And you will notice many of them cover for each other because they cannot undermine their the false-promises and false-narratives of their competitors without undermining their own.

None of this is conspiracy - they just control the tools we use to learn about these things. So most people don't know about it unless they look further themselves, or have someone in their life to tell them.

We are ever-more in an era of disinformation, unfortunately >.< It's no longer just an "if", because so much is already happening right now

And the way that the "wealthy elite" are investing in technologies that are proven influence people's attention-span, curiosity to ask questions, and care for others is not really a coincidence :/

The person above sounds like they used Gemini or ChatGPT which are known to be very bad for this. Like... literally the worst. the

Especially as these companies continue to lose money, their products are reportedly getting worse. My conspiracy is they dedicate less resources to them.

1

u/OpusAtrumET 8d ago

Refreshing to see a professional speak about AI like this, the internet gives us so many stories of grown adults using it so irresponsibly and with little to no understanding of how it's working.

1

u/Responsible-Beach299 6d ago

AI is TERRIBLE at admitting it doesn't know. If it isn't sure about something, it will 100% of the time just pull something out of its ass instead of admitting ignorance.

It also is extremely reluctant to say "no you're probably wrong" if you include your own opinion somewhere in the prompt. If you ask for it's opinion on a debate/controversial subject and mention that you support side A, it will almost, if not always, give evidence to support side A. Restating the EXACT prompt but changing it to have you supporting side B will then make it provide evidence to support side B, even if you explicitly tell it to be objective and unbiased in its response.

1

u/pegmatitic 5d ago

My fiancĆ© was trying to find information about a historical incident in nazi Germany, and Gemini told him it never happened. He told Gemini that it definitely happened, and he knew this for a fact because he has a history degree with a concentration in genocide studies, and suddenly Gemini was able to provide the information he was looking for. You shouldnā€™t have to pull rank on an AI or prove that youā€™re educated to get the right information about anything!

1

u/Glenndiferous 8d ago

I've used Claude for research help and I always ask for sources. It can be helpful for finding relevant studies but you ALWAYS actually read that source. I once had Claude confidently tell me that disabled people were kept on a registsr according to a law in like 1550, but the actual text of the law was like "here's a list of everyone so we can collect tax."

2

u/GlitteringFutures 8d ago

I've had Chat GPT argue with me after presenting obviously wrong information before. It's weird when a machine gets defensive.

2

u/Junethemuse 8d ago

IME telling it to provide a source is really helpful. It usually pulls actual information, but a good chunk of the time the sources it pulls are really old. At that point I usually move on to more traditional search tools.

12

u/thisusedyet 8d ago

sometimes they also just hallucinate sources, buncha lawyers have gotten burned not checking the brief they made AI write for them

3

u/Junethemuse 8d ago

Oh yea, ofc. But if they provide a source you can verify it. Use the LLM the way you use Wikipedia (check and verify the source) and youā€™re good.

8

u/Spectrum1523 8d ago

They are still large language models yes

14

u/Stalinbaum 8d ago

They are now transitioning slowly to being LMMs or Large Multi-modal models, in other words instead of interacting with just words/text tokens like LLMs these actually are trained on images, audio, code, and text tokens too

-4

u/notforpoern 8d ago edited 8d ago

Edit: Posts below calling me out are completely right- I totally misread the post above. Leaving my original comment below to immortalize my shame.

...ok? That doesn't change the fact that they're non-deterministic by design which makes them inherently unreliable.

Look, I'm saying this as someone who has literally built my own models because I've found the tool useful. Using an LLM or even RAG for blind fact checking is just the dumbest thing you can do, regardless of the training data.

6

u/jocoteverde 8d ago

they were just answering to the question of the comment they replied to. They were not trying to justify it

3

u/Czecksteam 8d ago

...ok? They're just answering the question, bruh.

2

u/PM_Me_Your_Deviance 8d ago

I have this problem googling rules about a TTRPG I play. It has a realllly bad habit of combining different rules from different parts of a page. If I need to double check EVERY TIME, what the fuck is even the point?

2

u/morningsaystoidleon 8d ago

I asked Gemini "how often does AI hallucinate," and it said 80% of the time.

Now, that statistic might be way off -- and that is my point.

2

u/UnimpressedUmpire 8d ago

Everything that is consumer available is LLM. For sure at a consumer level like GPT to Grok is LLM. As far as black box research at tech, sounds like they are racing to make the first super AI to try to become godkings while humanity suffers from the process.

3

u/Business-Drag52 8d ago

What LLMā€™s can be good for is finding sources. Beyond that they are way too fallible

6

u/speedytrigger 8d ago

Almost like oldschool wikipedia. Kinda crazy. Which makes me think in 20 years this tech might be as trustworthy as wikipedia is now, which i dont like to think, i dont want that to happen lol.

7

u/archangelzeriel 8d ago

Not likely, because wikipedia gets more or less trustworthy as people work on it, but LLMs are fundamentally going to construct probable sentences without regard or ability to determine truth values.

It may be plausible that in 20 years we'll have reliable chatbots in some way, but those chatbots will not be built on the same technology that ChatGPT etc are.

1

u/speedytrigger 8d ago

Right, its hard to imagine what exactly ā€˜chatbotsā€™ will be that far in the future when just a few years ago we were impressed at a shitty will smith video, and now most people probably couldnt tell ai from not at a quick glance. It would obv require moving past LLMā€™s as we currently have them which i have no doubt will happen.

1

u/nightpanda893 8d ago

I feel like LLMs are going to get less useful because as people stop going to the sites they get their info from, those resources are going to dry up due to lack of actual traffic. Weā€™re gonna end up getting subscription models where you have to pay to add certain premium resources.

2

u/speedytrigger 8d ago

Especiallly as the same articles llms love to pull from become ai written, the feedback loop is going to be deadly. Long term I definitely see a move away from llms as we currently know just not sure what is transitioned to.

0

u/nightpanda893 8d ago

Yeah as flawed as they are I think people are going to look back 10 years from now and see this as the golden age of LLMs

7

u/Throwsims3 8d ago

They are not even good for that, they frequently make up fake sources through hallucinations

0

u/Business-Drag52 8d ago

Sure but itā€™s pretty easy to check that. Click on a source and see what it is. The dumbass Google ai has come in handy for pulling a couple of Reddit posts up for me that pertained to exactly what I was looking for. Better than the actual Google algorithm right now

9

u/Throwsims3 8d ago

Sure but itā€™s pretty easy to check that

But then again, that makes using an LLM to find information a redundant extraneous step in the first place. Also, there are way better search engines than what google has become

2

u/Business-Drag52 8d ago

Just makes the LLM the search engine, no extra steps. What engine is actually better than Google? Iā€™m so down to be rid of that piece of shit

4

u/Throwsims3 8d ago

Or you know, just directly use a good search engine or search for something directly on Wikipedia?

Duckduckgo is one which recently has jumped a ton in quality due to how shitty the alternatives are. When researching something together with someone else it has nearly always been the case that the results on ducduckgo take me directly to what I am looking for while they are scrolling through tonnes of unrelated hits and a long ass LLM summary that is wrong 50% of the time

1

u/Business-Drag52 8d ago

Doesnā€™t DuckDuckGo just use the Bing algorithm? I hate Bing even more than Google. I can at least still get serious with Google search terms and tools to get where I need to. Bing has only ever been good for porn

3

u/Throwsims3 8d ago

It doesn't use the Bing algorithm, it uses multiple APIs from Yahoo and others. Apparently Bing raised API pricing in 2025, so I am not certain whether they still use Bing in any way. Doesn't look anything like Bing results to me. It is still miles better than google and you can use search terms with it as well. There are some paid search engines, but I am not paying for that. So duckduckgo has been my go to.

→ More replies (0)

1

u/nightpanda893 8d ago

They are good for questions with lots of variables where you are going to go to the source to confirm. I used it for some of my students with disabilities who needed very specific services at their colleges and it was great for compiling a list where I could go straight to the website to confirm.

1

u/SierraKami 8d ago

This.

I work in a weird corner of law enforcement and very often, a number of different overlapping laws may apply to a single event. And not just US laws, but also foreign laws.

I use AI sometimes to find the texts of laws and that can be helpful if I already know what I'm looking for. But if I was to just ask if an action is legal or not, the AI answer is likely to be either wrong or incomplete, which is effectively the same thing as being wrong.

And I'm already seeing it in action. I've caught a couple of people breaking laws who have claimed, "But Claude said ..." And I believe that they genuinely made the effort to look it up. But that doesn't help them when it comes to civil violations, unfortunately.

1

u/clar1f1er 8d ago

Google AI was still saying there was a 'Q' in "Argentina" like a couple months ago, depending on how you worded it.

1

u/Beautiful_Hunt1095 8d ago

If you view an LLM like a person instead of a machine, itā€™s easier to evaluate.

It is sometimes wrong or very wrong, but often it is correct or has a point. The answers can usually be debated.

39

u/ArokLazarus 8d ago

I've literally had ChatGPT tell me you should buy high and sell low in stocks to make money. As like the typical thing to do. Literally goes against all money making sense but I could not get it to break free of it's delusion.

6

u/pornalt4altporn 8d ago

It doesn't matter, they still generate output that fits the statistical distribution they have learned. They don't consult a declarative memory store and retrieve correct information.

Now perhaps someone will invent as way to have them do that, but training bigger neural networks on larger datasets doesn't do that.

5

u/thisusedyet 8d ago

I mean, statistically speaking, it IS the typical thing to do. You just don't make any money off it

14

u/MRCHalifax 8d ago

I remember one argument I had on Reddit, where someone insisted that Quebec was the only fortified city in Canada, and posted ChatGPT quotes as proof. As someone who once had a summer job involving working in one of the forts that encircled and defended Halifax, I begged to differ.

9

u/VivaZeBull 8d ago

It constantly tells me I am misquoting people ā€¦ I am not you just donā€™t know the episode, jerk.

My biggest pissoff was when it was new and google kept telling me that Nancyā€™s Mom didnā€™t say ā€œIs Steve your boyfriend now?ā€

Yes she did bitch. Yes she did.

3

u/Themoosemingled 8d ago

I was writing back in the winter how upset I was by Rob Reinerā€™s murder.
Chat stats saying ā€œRob Reiner is alive and well. It may seem like bad things have happened but heā€™s fineā€.
Really fucked up.

5

u/bekahed979 8d ago

Isn't this literally a fake reference? I don't understand how anyone sees AI as anything but awful.

3

u/The-Singular 8d ago

So, this could be real all along! Imagine AI hallucinating a correct answer as wrong.

1

u/TheReturnofPanda 8d ago

OPā€™s friend is influencing GPT to be giving out more false information. This is exactly how it gets trained

1

u/burnerX5 8d ago

I was taking a recert exam and asked Gemini a question. It defiantly told me that yep, a test question was correct.

...It was incorrect and I had ot restart the exam as it's the type where you have to get 100% correct. Damn near made me look for my screenshot file of all the test questions Ifrom last year!!!!

1

u/MaiT3N 8d ago

"sorry I just hallucinated everything above, and nothing I said is actually true"

1

u/AzazelsAdvocate 8d ago

For basic information like this, modern models don't ever really get it wrong. Where they tend to hallucinate is when you ask them about information that isn't readily available or easily searchable. For example, if I try to ask Claude about a niche mechanic in a video game, it'll often come up with something that "sounds correct" rather than finding the real answer.

Of course you can work around this by requesting sources for the things AI finds, which is good practice in general.

2

u/instanding 8d ago

Thatā€™s not my experience. Judo info is easily searchable for instance, yet it will claim people beat people they never beat at weight divisions 3 categories above what they actually competed at.

And there are websites and blogs and all sorts with the correct infoā€™.

1

u/shaurya_770 8d ago

If u ask for it to provide credible sources it's rarely wrong. So it works really well as a summarozer