Blowfish12: Bot with boyish personality wins biggest Turing test

Tuesday, June 26, 2012

Bot with boyish personality wins biggest Turing test

Eugene Goostman, a chatbot imbued with the personality of a 13 year old boy, won the biggest Turing test ever staged, on 23 June, the 100th anniversary of the birth of Alan Turing.
Held at Bletchley Park near Milton Keynes, UK, where Turing cracked the Nazi Enigma code during World War 2, the test involved over 150 separate conversations, 30 judges (including myself), 25 hidden humans and five elite, chattering software programs.
By contrast, the most famous Turing test - the annual Loebner prize, also held at Bletchley Park this year to honor Turing - typically involves just four human judges and four machines.
"With 150 Turing tests conducted, this is the biggest Turing test contest ever," says Huma Shah, a researcher at the University of Reading, UK, who organized the mammoth test.
That makes the result more statistically significant than any other previous Turing test, says Eugene's creator Vladimir Veselov based in Raritan, New Jersey. "It was a pretty huge number of conversations," he said, shortly after he was awarded first prize: "I am very excited."
First conceived by Turing in the early 1950s, the test is the most famous evaluation of machine intelligence. Human judges converse via a text interface with both hidden bots and humans - and say in each case whether they are chatting to a human or machine.
Turing said that a machine that fooled humans into thinking it was human 30 per cent of the time would have beaten the test. Just short of this, Eugene fooled its judges 29 per cent of the time. In a close second place, came JFred, the brain child of Robby Garner, and in third place Rollo Carpenter's Cleverbot. The other two bots to compete were Ultra Hal and Elbot.
Unlike several of Eugene's rivals, which put together sentences by imitating people they have spoken to before or by searching through Twitter transcripts for conversational ideas, Veselov has given his bot a consistent and specific personality. "He has created very much a person where Cleverbot is everybody," says Carpenter.
Eugene's character is that of a 13 year-old boy living in Odessa, Ukraine. He has a pet guinea pig and a father who is a gynecologist. Is 13 years old about the right age for a chatbot, then? "Thirteen years old is not too old to know everything and not too young to know nothing," explains Veselov.
A veteran of the Loebner prize and the Chatterbox challenge , Eugene was due a win. "We took second place several times but never were we the winners," says Veselov.
Did having a personality give him an advantage? "I think any appearance of a particular personality is likely to have a persuasive effect on judges," says John Barnden, an AI researcher specializing in machine understanding of metaphor at the University of Birmingham, UK, and a fellow judge.
He cautions against concluding that this was Eugene's edge, however - for that you would have to compare two versions of the same bot, but in one case with personality suppressed.
"In my own case it's not so much personality in the abstract that's key as how the system responds to a comment - is the response relevant and non-vacuous?" he adds.
I can sympathize with that: in some cases I knew it was a machine because the entity didn't seem to follow the sense of the conversation. I was however, delighted by how funny, and zany some of the conversations with beings that I labeled as bots (Disclaimer: the best judge award is still to be awarded so I don't actually know how often I was right). They also forced me to consider in a new way, just what it is that makes humans human.