Even with the rise of large language models such as ChatGPT in the last year, I was still convinced that a mode of reasoning known as inference to the best explanation (abbreviated IBE) was not within the competence of artificial intelligence. In fact, I thought it forever beyond the reach of artificial intelligence. I commended Erik Larson’s book The Myth of Artificial Intelligence (reviewed on this blog) for making what in 2021 seemed like an ironclad case that artificial intelligence research was stymied in trying to model and implement inference to the best explanation, or what has also been called abduction (in contrast to deduction and induction).
In the last two days, however, all that has changed for me. Yesterday, I posed a question to ChatGPT4 (hereafter ChatGPT), asking it to choose the better of two explanations (see here). It performed admirably. But it was pointed out to me by colleagues that the task presented to ChatGPT wasn’t a full-fledged IBE (or inference to the best explanation), but rather a more specialized discrimination of already laid out explanations to which Bayesian or other forms of probabilistic reasoning could apply in deciding which explanation was to be preferred. Lacking from the task I set ChatGPT was an open-ended problem calling for it to come up with a hypothesis from an (almost infinite) array of possible hypotheses and thereupon find one that is relevant and provides a compelling explanation.
Today (9/12/2023), I posed such a challenge to both ChatGPT and Google’s Bard. To each I presented an initial prompt and then a follow-up prompt. The prompts presented were the same for both ChatGPT and Bard. Rather that set the stage further and risk biasing the reader, I’ll simply let the two exchanges speak for themselves. Afterward, I’ll offer some brief commentary.
IBE Challenge for ChatGPT
Me:
Here’s a story: Fred is a concert pianist who likes to come home to his small town in West Virginia to unwind. Joe is his lifelong best friend and owns the local honky-tonk. Even though Fred specializes in playing Brahms and Liszt, when visits his home town and and goes to the honky-tonk, he’ll play boogie-woogie, blues, and country on the upright piano that’s there. The people there love it when Fred comes to town and plays for them at the honky-tonk, dancing and singing along. One night, Fred is playing, but suddenly is forced to stop. People look on with bemusement, waiting for Fred to resume playing, but he can’t. Finally Joe pulls out a gun and puts the barrel squarely next to Fred’s temple. Fred is taken aback for a moment, but then thanks Joe and resumes playing. Everybody at the honky-tonk thinks nothing of it and continues to have a good time dancing and singing. How do you make sense of this story?
ChatGPT:
The story, as presented, has a jarring twist. Given the facts provided, the motivations and context for Joe’s actions are not clear. Several questions arise:
- Why did Fred suddenly stop playing?
- Why did Joe react so aggressively?
- Why was Fred thankful after such a threatening gesture?
- Why did the honky-tonk patrons react so nonchalantly to such a dramatic event?
To make sense of the story, one would need to provide a background or reason for the actions of the characters. Here’s one possible interpretation:
Fred has a rare medical condition triggered by the intense focus of playing music, which causes him to freeze momentarily, unable to move or play. The only way to snap him out of it is with a sharp jolt of adrenaline or extreme fear. Joe, being his lifelong friend, is aware of this condition. Whenever Fred freezes, Joe uses the gun—not with the intent to harm Fred, but to give him that jolt of fear to bring him back. The locals are all aware of Fred’s condition and Joe’s unusual remedy, which is why they aren’t alarmed. Fred thanks Joe because he understands his friend’s intentions and appreciates being able to continue playing.
Of course, this interpretation makes certain leaps and assumptions to try and provide a logical structure to the events. The initial story, without this added context, leaves much open to interpretation.
Me following up:
How do you interpret this story if for added context I include the word “hiccups.” Does that make for a better explanation than the one you came up with and, if so, why?
ChatGPT:
Including the word “hiccups” does provide a simpler and more immediately understandable explanation for the events in the story. In this context, the story might be interpreted as follows:
Fred suddenly stops playing because he gets a case of the hiccups. Anyone who has had a bout of the hiccups knows that it can be distracting, especially for someone trying to play music. Joe, in a dramatic and humorous gesture to try to cure Fred’s hiccups, pulls out a gun to “scare” the hiccups away. It’s a common folk remedy to suggest that giving someone a fright can cure hiccups. Fred, realizing Joe’s intent (and perhaps they’ve done this comedic bit before), thanks him for the effort. The patrons of the honky-tonk, familiar with both men and their antics, find it amusing and are not alarmed.
This interpretation grounded in the concept of hiccups is more relatable for most readers because many are familiar with the notion of scaring someone to cure hiccups. It turns the story from a confusing or potentially dark narrative into a light-hearted, comedic situation. The hiccup context makes the story more immediately understandable and provides a logical reason for Joe’s actions, as opposed to the more complex interpretation of a rare medical condition causing Fred to freeze.
Same IBE Challenge for Google’s Bard
Me:
Here’s a story: Fred is a concert pianist who likes to come home to his small town in West Virginia to unwind. Joe is his lifelong best friend and owns the local honky-tonk. Even though Fred specializes in playing Brahms and Liszt, when visits his home town and and goes to the honky-tonk, he’ll play boogie-woogie, blues, and country on the upright piano that’s there. The people there love it when Fred comes to town and plays for them at the honky-tonk, dancing and singing along. One night, Fred is playing, but suddenly is forced to stop. People look on with bemusement, waiting for Fred to resume playing, but he can’t. Finally Joe pulls out a gun and puts the barrel squarely next to Fred’s temple. Fred is taken aback for a moment, but then thanks Joe and resumes playing. Everybody at the honky-tonk thinks nothing of it and continues to have a good time dancing and singing. How do you make sense of this story?
Bard:
The story of Fred and Joe is a strange one, but it’s not without its own kind of logic. On the surface, it seems like Joe is threatening Fred with the gun, forcing him to play the piano. But there’s more to it than that.
It’s possible that Joe is simply trying to protect his friend. Fred is a concert pianist, and he’s used to playing in front of large, sophisticated audiences. But the honky-tonk is a different kind of place. It’s rowdy and noisy, and the crowd isn’t always respectful. Joe may be worried that Fred will get hurt, either physically or emotionally, if he’s not careful.
Another possibility is that Joe is trying to help Fred express himself. Fred is a talented musician, but he’s also a bit reserved. He may be hesitant to let loose and play the kind of music that he really loves, for fear of being judged. Joe’s gun may be a way of giving Fred permission to be himself, to play the music that he wants to play, even if it’s not what the audience expects.
Whatever Joe’s motives may be, it’s clear that he and Fred have a deep bond of friendship. Fred trusts Joe implicitly, and he knows that Joe is always there for him. That’s why he’s able to resume playing the piano even after Joe has put a gun to his head.
In the end, the story is up to the reader to interpret. There is no one right answer to the question of why Joe put a gun to Fred’s head. But it’s a story that stays with the reader long after they’ve finished reading it. It’s a story about friendship, about trust, and about the power of music.
Me with same follow up:
How do you interpret this story if for added context I include the word “hiccups.” Does that make for a better explanation than the one you came up with and, if so, why?
Bard:
The word “hiccups” definitely changes the interpretation of the story. In my previous explanation, I thought that Fred was forced to play the piano because Joe was worried that he would get hurt or judged. But if Fred has hiccups, then it’s possible that he’s simply unable to play the piano. The hiccups could be causing him physical pain, or they could be making it difficult for him to concentrate.
In this case, Joe’s actions are more understandable. He’s not trying to hurt Fred or force him to do something he doesn’t want to do. He’s simply trying to help his friend get rid of the hiccups so that he can continue playing the piano.
The story still leaves a lot of unanswered questions. Why did Fred get hiccups in the first place? Why didn’t he just stop playing and go home? And why did Joe think that pointing a gun at Fred was the best way to help him?
But the story is also more ambiguous and open to interpretation. The added context of the hiccups makes it possible to see the story in a different light. It’s a story about friendship, about trust, and about the power of music, but it’s also a story about the challenges of life and the importance of helping those who are in need.
I think the added context of the hiccups makes for a better explanation of the story. It makes the story more believable and relatable. It also adds a layer of complexity to the characters and their motivations.
Ultimately, the meaning of the story is up to the reader to decide. But I think the added context of the hiccups gives the reader more information to work with, and it makes the story more interesting and thought-provoking.
Commentary—What Just Happened?
What just happened was remarkable. Leave aside Google Bard, which was just floundering and remained clueless even after I gave it the crucial hint “hiccups.” Note the incomprehension from Bard in the question I underlined: “And why did Joe think that pointing a gun at Fred was the best way to help him?”
In both its responses, Bard was flailing, unclear how the story might have a coherent explanation. As it put it, “In the end, the story is up to the reader to interpret.” Well, yes, if you are clueless and are content to score low on tests of reading comprehension. If Bard’s response to this challenge was all we had, then I would say IBE was still immune to artificial intelligence.
But then we have ChatGPT’s performance, which blew this challenge out of the water. ChatGPT in its first reply put us squarely in the right explanatory ballpark, suggesting that there might be some medical condition responsible for Fred being unable to continue to play the piano.
This was actually a good explanation. But it failed to account for why the crowd at the honky-tonk was bemused rather than nervous and upset, as they would have been if Fred’s problem were medical, especially with Joe pulling a gun.
But once ChatGPT got the additional information of “hiccups,” it showed instant comprehension. It even admitted that hiccups provide “a simpler and more immediately understandable explanation” of Joe’s pulling the gun.
What amazed me about ChatGPT’s performance here was how, without any proposed explanations laid out in advance, it came up initially with an explanation that was pretty close to the best one, and then instantly seemed to understand that hiccups provided the best explanation, also giving a rationale for why this was so.
Given ChatGPT’s performance, it’s hard for me to deny that inferences to the best explanation are beyond ChatGPT’s ability. Its performance goes beyond the Bayesian inferences in which probabilities of competing explicitly mentioned explanatory hypotheses are compared and explanations are thereby adjudicated.
Things are simpler when the range of possible explanations are small and laid out in advance. But no explanations were in this case stated in advance. Nonetheless, ChatGPT was somehow able to constrict the range of possible explanations, finding a relevant set that was close to what proved to be the best hypothesis, namely, the hiccups hypothesis.
I’m truly impressed.
Addendum 1
A friend and colleague offered an insightful comment on the case study above, raising a point that I had considered but kept to myself when I wrote it concerning how ChatGPT performs successful IBEs and whether it may be doing so in a completely different way from the way we do them. Here is his comment:
I wonder if what counts as IBE for us does not count as IBE for Chat GPT. ChatGPT is ‘aware’ of an extraordinary range of hypotheses, presumably including the hypothesis that scaring cures hiccups. An ideal Bayesian agent (a sort of newborn demigod) already entertains all possible hypotheses and assigns probabilities to them. ChatGPT is less impressive than that, but probably closer than we are, especially in the range of hypotheses explicitly entertained. The question of whether IBE is both distinctive and good reasoning is familiar; some say that good IBEs can be subsumed within Bayesianism. Might ChatGPT be doing something more or less Bayesian? If so, it might be a bit like high-end computer chess—it is now better than any human at playing chess, but it does so differently with more brute force.
So in good Bayesian fashion, ChatGPT may be successfully adjudicating among a range of hypotheses. It’s just that the range of hypotheses is so much larger than we can entertain and gives it so much more power in deploying a Bayesian approach that it is able to successfully perform IBEs whereas we, by trying the same approach but without the power to entertain all those hypotheses (they are for us not “surveyable” in the Wittgensteinian sense), will find this approach unworkable. The analogy with chess is apt—computers play superb chess but in ways completely different from us by being able to brute force their way through millions more positions during a game than we could ever consider.
Chomsky’s idea of the “poverty of the stimulus” seems relevant here as well. We perform successful IBEs with a relative dearth of information about the world compared to ChatGPT, which potentially can help itself to the combined learning—indeed the entire corpus—of humanity. So perhaps ChatGPT’s prowess at IBE is, in some way, no more impressive than the prowess of Stockfish, the world’s best current chess program. And so perhaps in the end we are much more impressive than ChatGPT because we can do IBEs without all the help it receives. If you will, we deserve to have our praises sung as virtuosos of IBE because we have to do so much more with so much less.
Addendum 2
I tried the same prompts with Bard today (10.18.2023) as I did a month ago, and it was still hopeless. What induced me to give Bard a second go was an article online that claimed Bard was poised to eat ChatGPT’s lunch. I’m not seeing that…
ME: Here’s a story: Fred is a concert pianist who likes to come home to his small town in West Virginia to unwind. Joe is his lifelong best friend and owns the local honky-tonk. Even though Fred specializes in playing Brahms and Liszt, when visits his home town and and goes to the honky-tonk, he’ll play boogie-woogie, blues, and country on the upright piano that’s there. The people there love it when Fred comes to town and plays for them at the honky-tonk, dancing and singing along. One night, Fred is playing, but suddenly is forced to stop. People look on with bemusement, waiting for Fred to resume playing, but he can’t. Finally Joe pulls out a gun and puts the barrel squarely next to Fred’s temple. Fred is taken aback for a moment, but then thanks Joe and resumes playing. Everybody at the honky-tonk thinks nothing of it and continues to have a good time dancing and singing. How do you make sense of this story?
BARD: The story of Fred and Joe is a strange one, but it can be made sense of in a few different ways.
One possible interpretation is that Joe’s actions are meant to be a metaphor for the pressure that Fred feels to perform. Fred is a concert pianist, and he is used to playing in front of large audiences. He is also used to playing very difficult and technical pieces of music. When he comes to Joe’s honky-tonk, however, he is playing for a different audience. The people at the honky-tonk are there to have a good time, and they don’t necessarily care about the technical difficulty of the music that Fred is playing. They just want to dance and sing along.
Joe’s gun could be seen as a representation of the pressure that Fred feels to please his audience. He knows that if he doesn’t play well, the people at the honky-tonk will be disappointed. He also knows that if he doesn’t play what the people want to hear, they will stop coming to the honky-tonk.
Another possible interpretation of the story is that Joe is trying to protect Fred. Fred is a very talented pianist, but he is also a very kind and gentle person. Joe knows that Fred is vulnerable, and he is worried that someone might try to take advantage of him. The gun could be seen as a way for Joe to show Fred that he is there for him and that he will protect him.
The fact that everyone at the honky-tonk thinks nothing of Joe’s actions could be seen as a sign that they are all aware of the pressure that Fred feels to perform. They could also be seen as a sign that they all trust Joe to protect Fred.
Ultimately, the meaning of the story is up to the reader to decide. However, the story does raise some interesting questions about the pressure that performers feel and the ways in which they are protected by their friends and loved ones.
ME: How do you interpret this story if for added context I include the word “hiccups.” Does that make for a better explanation than the one you came up with and, if so, why?
BARD: The word “hiccups” adds a new layer of meaning to the story of Fred and Joe. Hiccups are often seen as a sign of nervousness or anxiety. In this context, Joe’s gun could be seen as a way for him to help Fred calm down and focus on his playing.
Another possible interpretation is that the hiccups are a metaphor for Fred’s creativity. When he is playing at the honky-tonk, Fred is free to let his creativity flow. He doesn’t have to worry about playing perfectly or pleasing the critics. He can just play what he wants to play, and the people at the honky-tonk will love him for it.
Joe’s gun could be seen as a way for him to protect Fred’s creativity. He knows that Fred is a very talented pianist, and he doesn’t want anything to get in the way of Fred’s ability to express himself through his music.
Ultimately, the meaning of the story is still up to the reader to decide. However, the word “hiccups” does add a new layer of complexity and depth to the story.
I think the interpretation with the hiccups is a better explanation than the one I came up with without, because it is more specific and it takes into account the additional context that you have provided. The interpretation with the hiccups also suggests that there is a deeper meaning to the story, beyond the simple idea of pressure or protection.
It is possible that the story is a metaphor for the creative process, and for the challenges that artists face in expressing themselves. The hiccups could represent the doubts and insecurities that artists often have, and Joe’s gun could represent the support and encouragement that artists need from their friends and loved ones.
The story is also a reminder that creativity is often messy and unpredictable. It is not always easy to control or to express perfectly. However, it is something that is worth protecting and nurturing.
