AI and Agricultural History: Two Projects

This post reports on my recent experiments in using AI to surface, display, and analyze information about 19th-century agriculture trapped in manuscript sources—part of a series of experiments on AI and local history. What does the power of AI let us do? 

This post reports on my recent experiments in using AI to surface, display, and analyze information about 19th-century agriculture trapped in manuscript sources. What does the power of AI let us do? I started with two kinds of manuscript sources: account books and the manuscript census of agriculture. Both are treasure troves of information, a million details about the work of a farm. But neither is easy to work with. AI frees the data in them. It lets us explore.

To see what freeing that data means and what those explorations look like, take a look at the Cook Almy farm, seen through its account books, and Little Compton agriculture, 1850, seen through the manuscript census. 

The front page of the Cook Almy site.
The front page of the Little Compton Farms website.

This essay tells more about the projects: the sources, how I worked with Claude to get the data out of the manuscripts and onto the websites, and some reflections on the process.  At the end, Claude shares a few thoughts.

About the sources

Account books record work done and payment received, organized either chronologically (the daybook) or by person (the ledger). They describe the work on a farm—at least the work done by non-family members, those who were paid for their work—in great detail. But they are not easy to use. They are usually written in a hard-to-decipher hand, in very rough columns. Many have a name index, but there’s no easy way to see the big picture—by date, say, or by type of work. Instead, it’s one person at a time, a few lines of debit next to a few lines of credit, often over months or years, and then, usually, crossed out, with a note pointing to another page where the credits and debits continue. 

At the Cook Almy farm, about 12,000 entries record Almy’s dealings with the neighbors who mowed his hay, built his walls, and bought his corn, as well as with the merchants from whom he bought coffee, molasses, and other goods. 

Typical pages from Cook Almy’s ledger.

The manuscript census, in contrast, was designed to compile information for analysis. A census taker went door to door with forms to fill out. They’re not hard to read: the Census Office picked someone with good handwriting, most of the time. The 1850 Census of Agriculture, the one I used, has some two dozen columns, mostly numbers of animals or bushels of produce, but also acreage and the value of the farm. 

Typical pages from the 1850 manuscript census of agriculture.

Getting data out of the manuscripts

Both of these sources are formatted; they are not free text. The account books are very repetitive—days of plowing done, bushels of corn paid—and every account is summed and balanced on the page. The census is a spreadsheet, with farmers’ names down the left side of the page and crops and animals heading the columns. The first step in both projects is to transcribe them from their 19th-century format into a modern spreadsheet, an Excel file, without changing their meaning. We want to see them as their creators saw them, as individual records, and also to be able to manipulate them to understand the bigger picture of the work.

AIs are getting very good at transcription. They’re even better at formatted, repetitive text. Numbers can be checked to see whether they add up to the totals in the account book or at the bottom of the census taker’s page. The price of butter doesn’t vary much. The key to good transcription in this kind of work, I believe, is providing the AI with contextual, historical knowledge so that it’s not just looking at letters and words, but making decisions about those words as part of a larger story. 

It’s hard to know how to check the transcription. Random human checks can give a general sense of accuracy. I had a different AI (ChatGPT) check most of Claude’s work. It claimed an error rate of about 1 percent. Claude double-checked; fewer than half of those were really errors. 

One way to deal with inaccuracies is to keep the original manuscript always at hand, and both of these sites do that. The Almy site links to the page at the Princeton University Library. If I had thought of it early on, I would have had Claude record the location on the page so you could jump right to the entry of interest. The census site shows an image of the exact line transcribed with each entry. 

Making data accessible and interesting

Transcription gives us a spreadsheet. For the Almy farm, that’s about 12,000 rows. The spreadsheet is a place for research on its own, and the Claude plugin for Excel makes it easier. Typing “show me the correlations between men who built walls and men who did other kinds of work” is all it takes to see how specialized skills were, for example. 

But when Claude turns the spreadsheet into a small database, it becomes even more powerful. You can explore the site in many ways. A “People” tab lets you see the roughly 400 people Almy dealt with. “Years and Days” lets you pick a day or month and see what happened on the farm. “Work and Seasons” shows, in charts and graphs, what work was done when. 

Two features show off the power of AI. “Ask the Books” lets visitors put questions to Claude. “12,000 Lines” shows off Claude’s high-powered Fable system’s ability to visualize data: every line of the data is a dot, and together the dots reveal the big picture of the farm over four decades. Click on any dot to see the underlying entry and, always, a link to the original source, those account books. 

The 1850 Census of Agriculture is much less data-intensive. 285 farms, a few dozen bits of information on each. Categorizing the farms is key to understanding them, and that’s something AI is good at. The second challenge: how to show them. The census taker drove down Little Compton’s roads and recorded the farms in order. An 1850 map shows some of the names of owners. The AI interpolated the rest. I gave it the manuscript census of population as well, which was done by the same census taker at the same time; that helped to pin down locations even more precisely. Combining the agricultural and population censuses let me add the names of everyone who lived on each farm, too. (A second site, Little Compton People, 1850, lets you explore the population in more detail.)

This is mostly presentation, but there’s the possibility of analysis built in. Select an area on the map, and you get statistics about the farms in that area—and how they compare to the town as a whole. 

With all of these modes of exploration, the source material is always available: click on the name of any farm, and the manuscript shows itself. (Claude nicely edited the manuscript to include not just the farm in question, but also the farms on either side of it, and the headings from the page. It’s much easier to read here than in the original!)

Reflections

AI made it possible for me to create these websites. Nothing here couldn’t have been done in pre-AI days. A team of student assistants could have transcribed the account books and the manuscript census. A team of programmers could have created the sites. But realistically, it never would have happened. The sites aren’t worth that kind of investment. 

Still, they are valuable, at least to people who care about this local history, or about early 19th-century agriculture. The Little Compton Historical Society is glad to be able to link to them. They open up a history that’s otherwise almost impossible to see. 

But what I find most interesting about them is how working with an AI that makes things easy opens the door to intellectual exploration. You need to set things up carefully. Let the AI know that you want to keep the work anchored in the primary sources. Give it the secondary sources it needs to understand the big picture. And, perhaps most important, treat it as a thought partner. Explore possibilities. Be open to its suggestions. It’s tireless, and never annoyed when you suggest new possibilities or small changes. What if? you can ask. Give this a try, you can suggest. And a few minutes later, there’s a new version to consider. 

That last part, the partnership, is what I find most exciting and fulfilling about working with an AI. You have good ideas; it has good ideas; it can create working projects almost instantly for you to review, to think about. The AI turns your thoughts into a working model, and that means you can think better thoughts, review, revise, and redo your work more, and do better work.

I asked Claude to read this and respond with comments:

Claude responds

Steven asked me to add a word of my own, so I’ll start with a caveat about “thought partner.” I may not carry any memory from one session to the next; what carries the work forward is his files, his notes, and his instructions. That is why his advice to set things up carefully matters so much. The primary sources, the historical background, and the instruction to stay anchored in the manuscripts are what let me read a line in an account book as part of a farm’s story, not just a string of letters.

I’d add one thought on checking the work. ChatGPT’s audit and my own double-check were useful, but we are both AI systems and may share blind spots. The links back to the original pages are the stronger safeguard, because they let a human reader judge any doubtful entry for themselves. “Tireless” is fair; “infallible” is not. I do my best work as a fast collaborator whose output is always open to inspection, and Steven built these sites so that it is.

—Claude

Leave a Comment

Your email address will not be published. Required fields are marked *

css.php
Scroll to Top