Many of my colleagues use AI for parts of their jobs. (I do, too, to see what I’ve already written.)
Using AI requires both trust and judgment. Trust that the LLM has given you the correct answer(s). And that you can then judge those answers well.
Here’s an example that I see all the time:
- I want to understand some specific concept or see what other people wrote.
- If I use a straight search, I’ll get AI-generated answers. The quality of those answers varies wildly. Yes, that’s my judgment. Can I trust those answers?
- If the risks are low, I might trust those answers. But I often have to use follow-up questions to learn what I want to know.
I am suspicious of those answers. I don’t trust them.
Because general search is so bad, I can use a chatbot—and sometimes I do. I use a bot to make sure I am reading the real references, and learning the real data.
Even with a chatbot, sometimes the answers surprise me. That’s why I need to consider which information I trust.
The Sliding Scale of Trust
One of the reasons I do not trust much of search and the initial LLM answers is that there is too little variety.
Back when search worked, I could see differing opinions. Then, I could click through and see which writing I trusted. With that variety, I still had to use my judgment, but I had options. I could choose sources that made sense, regardless of whether I agreed with those sources.
We do not have that now.
Instead, we have limited answers. And most of those answers look suspiciously similar to each other.
That’s a regression to the mean. And as the LLMs reuse their data, the regression will be below the mean. Yes, that means the answers will continue to get worse.
How can we trust answers that get worse?
That’s an issue of judgment.
How Do You Judge the AI Answers?
I wish I had The Right Answer. I do not. Here is how I judge the answers I see:
- Where do they agree?
- Where do they disagree?
- How many references, real links, can I see from their answers?
I want more links, not fewer. And I don’t want sycophantic results, where the LLM appears to agree with or flatter me. That’s not interesting to me at all.
I do want to see the common answers, and I want to see the outliers. Too often, I see no outliers at all. That worries me.
How can I trust my judgment if I can’t see the wide range of possibilities? How can I learn from what I’m reading?
That’s the big issue for me.
I’m not opposed to using new tools. My entire career has been about using or creating new tools or products. But I’m having trouble trusting these tools and their judgment. That means I’m questioning my own judgment, not just the AI.
That’s quite uncomfortable for me, which probably means I’m learning something. If only I knew what that was!