Natural Language and the Winograd Schema Challenge

The Winograd Schema Challenge was proposed by Hector Levesque of the University of Toronto as a test for artificial intelligence, similar to the Turing Turing test. The test is based on a problem of natural language understanding that was presented in an example given by the pioneering AI researcher and computational linguist Terry Winograd. Levesque suggested that this kind of problem can form the basis of a test for human-level understanding of natural language text that is more objective than the Turing Test.

A Winograd schema is a pair of sentences that differ in only one or two words and that contain an ambiguity that is resolved in opposite ways in the two sentences and requires the use of world knowledge and reasoning for its resolution. The schema takes its name from a well-known example by Terry Winograd (1972):

• The city councilmen refused the demonstrators a permit because they [feared/advocated] violence.
  Who [feared/advocated] violence? Answers: The city councilmen/the demonstrators.

Many examples of this kind of inference problem have been collected. Here are a couple more examples:

• The trophy doesn't fit into the brown suitcase because it's too [small/large].
  What is too [small/large]? Answers:The suitcase/the trophy.
• Joan made sure to thank Susan for all the help she had [given/received].
  Who had [given/received] help? Answers: Susan/Joan.

The idea of the chanellenge is to provide some machine implemented reasoning capability, by which one can decide on the referent of the pronoun based on the information given in the sentence. This will typically require the encoding of background knowledge as well as the use of some inference mechanism.

A project on this topic, would most likely confine its attention to a small number of Winograd Schema Challenge problems that are set in a similar domain (e.g. spatial configurations, human interactions). It would be necessary to identify and respresent the background information necessary to solve problems in this domain and show how this can be used for the particular problmems tackled. The required reasoning mechanisms could be implemented by using an existing standard automated reasoning system, such as Prover9 or by a customised program implementation (perhaps using Prolog).

Further information on the Winograd Schema Challenge can be found on Ernie Davis' web page at:
http://www.cs.nyu.edu/davise/papers/WS.html

A collection of papers on the Winograd Schema Challenge can be found here.

Two previous projects on this topic have been carried out. Anyone interested in a project in this area should take a look at these. But bear in mind that there are many different ways that the WSC can be investigated, so many types of project are possible.

• Tackling Domain-Specific Winograd Schemas with Knowledge-Based Reasoning and Machine Learning, Suk Joon Hong (2020)
• Analysis of Existing Approaches to the Winograd Schema Challenge --- Ruiwei Hu (2019)
• Practice, analysis, and evaluation of methods solving Winograd Schema Challenge --- Xiangru Cui (2019)