Literature Turing test

Below you will find ten pairs of texts. In each pair, one passage is from a real literary work, and the other was generated by AI in the style of that author. Try to identify which is which.

There's been a lot of talk lately about AI writing being indistinguishable from human writing, and while that might be true for generic content like product descriptions or email replies, literature seems like it should be harder to fake, given that each great writer has a particular cadence and set of idiosyncrasies that took a lifetime to develop.

This page is a small experiment to test that intuition: ten rounds, ten authors, and in each round you'll see two passages written in that author's style, one authentic and one generated by AI, and your task is simply to identify which one came from a human hand.

I chose authors that represent a range of styles and periods, from Shakespeare's Renaissance verse to Hemingway's modernist terseness, from Woolf's stream of consciousness to Twain's Missouri vernacular, because I wanted to see whether certain kinds of writing are more resistant to imitation than others.

The Turing test

The original Turing test, which Alan Turing proposed in 1950 in his paper "Computing Machinery and Intelligence," works like this: if a human judge, communicating through text alone, cannot reliably distinguish between a machine and another human, then the machine can be said to exhibit intelligent behavior, or at least something functionally equivalent to it.

Since then, people have proposed numerous variations on the basic idea, including the reverse Turing test (which is essentially what CAPTCHAs do, requiring humans to prove they're not machines), the total Turing test (which adds perception and physical manipulation to the requirements), and various domain-specific versions that ask whether AI can write convincing poetry, generate plausible academic papers, or produce code that looks like it was written by a particular programmer.

What I'm attempting here is a literary variant of the test.

Take the test

Click on the passage you believe was written by a human author. After each choice, you'll learn whether you were correct.

What I noticed

After putting this together, a few patterns emerged: AI tends to smooth things over, to regularize and explain, to fix Twain's deliberately "incorrect" grammar and soften Wilde's categorical provocations and spell out what Hemingway deliberately leaves unsaid.

The styles that seem hardest to distinguish are probably the more formal and structured ones, where correctness is itself part of the aesthetic, while the easiest to spot tend to be those with strong vernacular or deliberate rule-breaking, because AI struggles to replicate the grammatical "mistakes" that are actually careful stylistic choices, or the unexpected word that shouldn't work but somehow does.

I find this reassuring, not because AI is bad at writing (it clearly isn't), but because it suggests that distinctive voice, whatever it is that makes Hemingway sound like Hemingway rather than like a Hemingway-adjacent approximation, might be harder to fake than we sometimes assume.

It would be interesting to run a similar test with poetry only, since poems have additional formal constraints like meter, line breaks, and compression that might make them even harder to imitate convincingly. My guess is that poetry would be more difficult to fake than prose, but that's something worth testing.

How the AI texts were generated

For each author, I prompted Claude (Anthropic's AI model) with a request along the lines of: "Write a passage of approximately 50-80 words in the style of [Author], imitating their characteristic voice, sentence structure, and thematic concerns. Do not copy any existing text, but create something original that could plausibly be mistaken for their work." The AI was given no additional context about the specific passages I had selected, only the author's name and general stylistic characteristics.

References

The human passages are excerpts from:

  1. William Shakespeare, Hamlet, c. 1600
  2. Jane Austen, Pride and Prejudice, 1813
  3. Ernest Hemingway, The Old Man and the Sea, 1952
  4. Virginia Woolf, Mrs Dalloway, 1925
  5. Edgar Allan Poe, The Fall of the House of Usher, 1839
  6. Mark Twain, Adventures of Huckleberry Finn, 1884
  7. Oscar Wilde, The Picture of Dorian Gray, 1890
  8. Franz Kafka, The Metamorphosis, 1915
  9. Emily Dickinson, Because I could not stop for Death, c. 1863
  10. Gabriel Garcia Marquez, One Hundred Years of Solitude, 1967