I recently had an amusing but disconcerting experience when running mid-term speaking exams for undergraduate students. One member of a first-year group was given the instruction, ‘Tell me about one of your childhood dreams’. The boy was, contrary to what he had been told, reading off from a piece of paper, and his answer began as follows: ‘As an artificial intelligence, I don’t have the concept of childhood…’.

What does this story tell us, apart from its comic quality? Firstly, I must mention that the exam questions were, like the boy’s answer, generated by ChatGPT. ChatGPT can thus formulate the sentence ‘Tell me about your childhood dream’ without knowing what it is ‘saying’, given that it admits to having no such concept as ‘childhood’. The more worrying thing is that the student himself had no idea what the words he was saying meant.

John Searle advanced a controversial argument in 1980, when exponents of ‘hard AI’ were claiming that one day there would be computers that could think, known as ‘the Chinese Room Argument.’ Here it is in brief:

‘Suppose that I’m locked in a room and given a large batch of Chinese writing. Suppose furthermore (as is indeed the case) that I know no Chinese, either written or spoken, and that I’m not even confident that I could recognize Chinese writing as Chinese writing distinct from, say, Japanese writing or meaningless squiggles. To me, Chinese writing is just so many meaningless squiggles. Now suppose further that after this first batch of Chinese writing I am given a second batch of Chinese script together with a set of rules for correlating the second batch with the first batch. The rules are in English, and I understand these rules as well as any other native speaker of English. They enable me to correlate one set of formal symbols with another set of formal symbols, and all that ‘formal’ means here is that I can identify the symbols entirely by their shapes. Now suppose also that I am given a third batch of Chinese symbols together with some instructions, again in English, that enable me to correlate elements of this third batch with the first two batches, and these rules instruct me how to give back certain Chinese symbols with certain sorts of shapes in response to certain sorts of shapes given me in the third batch. Unknown to me, the people who are giving me all of these symbols call the first batch “a script,” they call the second batch a “story,” and they call the third batch “questions.” Furthermore, they call the symbols I give them back in response to the third batch “answers to the questions” and the set of rules in English that they gave me, they call “the program”’ (Searle 1980, 417-418).

Searle proceeds to complicate matters by supposing that the people who fed him the Chinese symbols do the same thing in his native language. That is, they give him stories in English and ask him questions about them, which he then answers. To the outside observer, the answers in English and those in Chinese are equally accurate and correct. The man in the room has become so good at manipulating Chinese symbols that his ‘answers to the questions are absolutely indistinguishable from those of native Chinese speakers’. What, then, is the difference between what the man does in English and what he does in Chinese? The element lacking is understanding, unlike English, the man understands neither the Chinese questions nor the answers he gives. ‘As far as the Chinese is concerned, I simply behave like a computer, I perform computational operations on formally specified elements. For the purposes of the Chinese, I am simply an instantiation of the computer program’ (Searle 418).

Without going so far as to say that students are becoming machines, I would at least like to suggest that the growing tendency to view language learning on the analogy of computer programs is obscuring our understanding of the learning process itself. The sinister thing about the anecdote with which I started is that the student taking the exam, like the man in the ‘Chinese room’, was in some sense behaving exactly like a computer. That is, he was manipulating and even uttering formal ‘symbols’ whose meaning was entirely unknown to him. The lesson which we can learn from Searle’s thought experiment is that understanding, whatever it may be, is absolutely distinct from the kind of operations which can be performed by a program. The process of answering the question ‘What was your childhood dream?’ may be represented as searching through a series of possible answers or ‘picture-cards’ in one’s memory, out of which one chooses the corresponding or ‘right’ answer, though I think this would be misleading. Even if we represent things in this way, however, the difference with a computer program is that, supposing the concept ‘childhood dream’ had not been fully acquired prior to the question being asked, the student will actually have learnt the meaning of this expression in the process. The meaning of the question will have become fully clear only once the answer has been given. This process of discovery is possible only because, unlike Searle’s example of the computer, the answer is not given in advance, and the respondent therefore has to spend some time thinking about the question. There is therefore room for error, the student can understand the question incorrectly. I can now give a first indication regarding what AI cannot do: it cannot misunderstand. This is quite logical, seeing as it can not understand in the proper sense either.

What speech acts is a computer incapable of, even supposing that it can ‘speak’ in any sense at all? Slavoj Žižek makes a startling point: a computer cannot swear (Žižek 2025). That is to say, the instinct which prompts us to use a swear word in a moment of great annoyance, emotion or perplexity is foreign to a computer. Žižek explains this as follows: a swear word is often a faute de mieux, an implicit admission that we don’t know the right word to use in the context, hence we have to swear. In other words, swearing shows our relation to something outside of language, something we want to express but cannot. It bears the mark of what Wittgenstein called the ‘unspeakable’ aspects of reality, and hence even in the obscene sense retains the original meaning of swearing an oath. All this seems to go unnoticed by my students, many of whom swear in English frequently during class. Doubtless, they are repeating things they have heard on the Internet, but the fact that they feel no barrier to doing so is revealing of the trivializing effect which computers have on our use of language. Jakobson distinguished six functions of language, of which only one had to do with the conveying of information, which he called the ‘referential’. The originality of his contribution consisted in showing that many of our everyday speech acts have nothing to do with informing our addressee of anything. The ‘phatic’ function, for example, can serve merely ‘to check whether the channel works (‘Hello, do you hear me?’), while the ‘conative’ is best demonstrated by the imperative, as it is directed towards the addressee and not the content (Jakobson 1960, 355). Whether artificial intelligence can fully grasp any of the six functions but for the ‘referential’ is uncertain.

All of the above poses serious ethical problems in relation to language and our use of it, particularly when we are dealing with a foreign language which students access and use via computers more than through other living beings. ‘In most cases, the use of the word in the language is its meaning’, Wittgenstein (2009) said. This means that understanding occurs in the moment of utterance, when the word is used in context. The words we utter can commit us to something, as in the example of swearing an oath used above, and have real consequences. ‘Words are deeds’. The rise of AI provides a chance for teachers to reflect on what it is we wish to impart to our students. Obviously, it is not mere information, which they can obtain in abundance on the Internet. But we must strive to make the classroom a space where their words have a felt weight in the moment of utterance. This is what I fear is being lost in the empty vacuum of ChatGPT, which admitted, in response to a question about responsibility, that it is ‘not responsible for anything’. It should be stressed that ‘responsible’ is here to be understood in the root sense of ‘respons-ible’: ‘able to respond’.

The French word for computer is ordinateur, that is, something that gives and carries out orders. Along with swearing, giving orders to their teacher is something else which students seem to feel no inhibitions about (‘Finish class early’, ‘Give me your pen’, ‘Show us a movie’). After all, an order is a very simple construction and gets the message across as directly as possible. It is not, however, correct usage in the above examples, where ‘correct’ goes beyond grammatical correctness. My fear is that our students have become so used to talking to computers that they proceed to speak English with other people as though they were computers. If we want the young to become capable of performing tasks that are radically different from those of a computer, we must teach them to truly ‘speak English’, and remind ourselves what that means. For the Greeks, after all, speech was what distinguished the human being - or ‘language animal’ (zoon phonanta) - from other animals. It is also what distinguishes us from computers.