AI everything! For a month!

That’s one way to describe the talk I gave for Brown’s AI and Humanities Research group, part of the Center for Digital Scholarship. Less clickbaity: How might historians use AI in their research?

I tried a variety of AIs—ChatGPT, CoPilot, JSTOR AI, among other—on projects I’m working on now. I also went back to some earlier projects to see what AI might have done for me. (My work is in nineteenth and twentieth-century American history.) Could it help me find sources? Help me read them? Analyze them? Write about them? The answers, briefly: no, yes, sometimes, no.

Here’s the talk. Summary (not AI-generated!) below.

Four takeaways:

Some of my general thoughts on how to use AIs:

Complementarity. AIs are tools, not machines. You need skills to work with them. You need to bring your expertise. They won’t do your work, but will work with you to help you do your work. Historian Benjamin Breen, the most thoughtful writer on how historians can use AIs, summarizes this as “Augmentation, not automation.”

Rubrics. A good teacher writes rubrics so that students know what’s expected. You need to write rubrics for your AI. What makes for a good answer? The more you tell the AI about what you want, the better the results.

Context. You need to explain the level, style, purpose, and use of your work. AIs will shape their output to the context you give them.

Slow down. Using AI can make you slow down and think about what you’re doing and why. Process first, then product. Slow scholarship—”thoughtful, reflective, and the product of rumination”—might seem the opposite of computer-aided scholarship, but the overlap might be worth exploring.

I approached my AI use as an expert in the field, not as a student. I had particular sources and data sets I wanted to explore, and a set of questions I wanted to answer. I asked questions where I knew enough to evaluate the answers, where I could tell the AI it was wrong, or could do better. This talk is about using AIs to augment your research work, not to learn how to do it. (I did not address the ethics of AI or the question of the sources they are based on; others in the workshop are focusing on those issues.)

Four explorations:

Discovery. AIs are bad at finding data and at finding sources, even the internet-connected one. THey do unsophisticated web searches. You’re better off doing it on your own.

Reading. They’re pretty good at this. An AI will do a decent of reading and summarizing PDFs, and will accurately answer questions about it. They can compare two sources and say something interesting about the differences. They can read small amounts of exotic texts, whether German fraktur or early modern English. They are surprisingly good at reading, summarizing, and simplifying patent documents, a very specialized form of writing—my favorite new discovery. They can read dialects. Specialized tools like Transcribus.eu can even read handwriting.

Analysis. Specialized tools like ChatGPT’s Data AnalystGPT can do a good bit of basic digital humanities work, reading spreadsheets and analyzing them, giving good answers to your questions. I gave it museum metadata, census data, and more, and got useful results. (I didn’t experiment with using AIs to write code; that is clearly useful for some projects, for people with expertise.) They are almost, but not quite, ready to do data cleanup; right now, it seems more aggravation than it’s worth. (The potential is great, though.)

They’re not very good at historiography, in my tests. They don’t seem to have a sense of how documents connect in a way that overlaps both chronologically and thematically. Two specially-trained GPTs that seem better at this: Benjamin Breen’s The Historian’s Friend and my HistoriographyGPT. (These GPTs require a subscription to ChatGPT 4.)

Basic AIs will describe images nicely, and even tell you about similar images. The Historian’s Friend can identify images and provide historical context. The Historian’s Friend does a better job of analyzing images. Breen’s Paleographist GPT offers a more structured way of exploring their cultural meaning.

Writing. GPTs are bad at writing. It’s what they do most easily, but the out-of the-box writing is not very good. The more you provide context and work with it and give it rubrics, the better, of course. What they’re more useful for is editing. Write your document, and ask them to review it. See if the AI’s summary is what you intended to say. An AI can be a convenient reader, ready to give feedback.

AIs are completely useless for creating historically accurate images.

Four Questions

How should we think about acknowledging the work we do using AIs? Should they be included in contributions statements, perhaps in the way that scientific papers list the detailed contributions of each contributor? Historians’s books include detailed acknowledgements; should an AI assistant be included?

How might AIs be used in graduate education? Writing a GPT might be a good assignment. Writing a really good GPT would be an excellent part of a field exam for a PhD student. Figuring out what a GPT needs to know to answer questions in your field would force a student to think hard about what they need to know.

How can we convince students it’s worth learning skills when you can get 90 percent of the way there without knowing them? Translation is the obvious one: if a machine can do a good-enough translation, why spend the years it takes to do a better one? The problem, of course, is that you can’t evaluate the results without knowing more than the machine.

What’s next? Some historians have begun to build custom large language models. Mark Humphries has trained one to talk like an eighteenth-century fur trader. Sphaera is trained on early scientific diagrams. Transcribes.eu trains LLMs to understand handwriting. D’AlemBERT knows Early Modern French. THis is labor-intensive work, but it might be a way to get beyond some of the presentism embedded in the commercial large language models that are easily availble.

Further reading:

Many historians are exploring this topic. Here are few places to start exploring:

Josh Dzeiza, “How AI can make history” Verge, February 15, 2024

Moira Donovan, “How AI is helping historians better understand our past,” Technology Review, April 11, 2023

Benjamin Breen’s blog, Res Obscura

Mark Humphries’s blog, Generative History

ADDED 3-9-2024: American Historical Review special section on AI and History.