Over a decade ago, I worked on a presidential papers project. The audacious goal was to scan in all presidential papers, make them available for download, and extract any possible data. But until the advent of the typewriter, virtually no data *could* be extracted, other than the odd letterhead. My proposal was to collect the images, build a processing pipeline, and when OCR of handwriting was possible, do it then.
Well, ChatGPT *nailed* this. So many handwritten documents can be discoverable!
@codinghorror @waldoj I've seen a lot of AI-captioned images and it ranges from useless to misleading.
And then you get horrifically wrong interpretations like this one: https://mastodon.social/@philipncohen/113080493757208395 (see post text for AI output -- it's not in the preview)
@timmc @codinghorror I’d love to hear from some blind and visually-impaired folks whether they find that they prefer AI-generated alt text to no alt text. (Yes, far better is the option of human-written alt text, but it’s been 24 years and we haven’t made much progress on that front.)
I hope somebody is conducting a study on this!
@waldoj @codinghorror I'd love to see a study as well.
I've seen some reports on the matter from blind folks, but it was long enough ago that I can't recall the details. However, I do know that AI generated captions for videos are widely despised by the visually impaired—maybe better than nothing, maybe not—and I feel like those captions are generally in better shape than the generated alt text I've seen.
@timmc @codinghorror @waldoj Misgendering alt-text in emails is an insidious one that we've been seeing in the wild. The author could have inspected/approved the generated text, it would be harassment if they did (or had written the caption themselves), but they also have plausible deniability.
The sender isn't supplying the image-to-text tool with more information so recipients could run their own tool and get the same quality results (plus ability to consult multiple tools/upgraded versions).
@waldoj @stralau but that would be using Machine Learning (ML), not generative AI via a Large Language Model (LLM)?
it'd even make it worse, as it would just hallucinate stuff it couldn't detect correctly, replacing stuff that's inaccessible with stuff that is potentially inaccurate or even plain wrong.
e.g. https://www.crikey.com.au/2024/09/03/ai-worse-summarising-information-humans-government-trial/
@waldoj yup, we’re developing ocr4all (https://www.ocr4all.org) at my workplace for that, and are using it with good success on stuff like medieval manuscripts and the like. as i understand it, handwritten text requires a couple pages of transcription for training a bespoke model, and than then OCRs the rest of the document.
as uses of ‘AI’ go, it’s my go-to example for genuinely helpful ones.
@aarbrk @waldoj i wouldn’t say off-label, since the very first sentence of the about page (‘what is ocr4all’) reads:
OCR4all combines various open-source solutions to provide a fully automated workflow for automatic text recognition of historical printed (OCR) and handwritten (HTR) material.
but yeah, you are correct both in transkribus being the big player in the field, as it were, and of the specificity problem. i’d have to ask whether it’s feasible to make a more general use model that is trained on multiple hands in the same script. (and i’m currently on vacation, so i’d have to get back to you on that.)
the aspects of HTR i’m familiar with are from the perspective of someone liasing with the domain experts who are partnering with us for digital edition projects, so it’s mostly at a bit of a distance. but from the project i’m currently assigned to, we’ve HTR-ed two medieval dutch manuscripts (14th century verse and 15th century prose) and that seemed to go over well with the philologist who is now working on the texts.
@gekitsu @waldoj Thank you for the correction, I did miss that.
I have an article under peer review right now about digitizing a particular collection of 19th century manuscripts. I wrote it a couple of years ago and am imagining that the reviewers come back with requests to elaborate on new software solutions. In that case, about 12 professional scribes collaborated to write about 23,000 pages of text. With another researcher we have experiments to detect the hand shifts, which is possible.
I'm pleased to announce that Sara and I are part of a team that have been awarded an NEH Digital Humanities Advancement Grant. The team (led by Lindsey Peterson of University of South Dakota and the Civil War and Reconstruction Governors of Mississippi documentary edition and Elisabeth La Beaud of the Mississippi Digital Library) will be developing "AI-based software that uses named entity recognition and large language models to automatically create subject tags for digitized cultural heritage materials to enhance searches and usability." In short, we're trying to make the emerging tools for recognizing and identifying entities within text actually usable for librarians, archivists, and scholarly editors. Here is the full list of awards: https://www.neh.gov/sites/default/files/2024-08/NEH%20August%202024%20grants%20list%20state%20by%20state.pdf
Thanks! We're doing a lot with HTR, and I think that LLMs definitely play a role. However, we need new metrics to evaluate LLM outputs, since traditional OCR/HTR produces obvious errors, while a LLM's errors are seductively plausible.
Attached: 1 image I'm reading a lot about #ChatGPT4o and #htr, so I figured I'd spot-check it with this document: https://fromthepage.com/lva/va-revcon-74-76/work-7811594-024/display/33996304 The results were better than nothing, but still not great. In particular, it was not able to read Lord Dunmore's name, nor read either of two references to "Emancipate our Slaves". If you've been following historical debates about the #1619project , you can imagine how problematic it would be to rely on this transcription for full-text search.
@felwert @waldoj @ManniCalavera
I think you're right that we have to consider fitness for purpose when choosing engines, and I suspect that we will end up with multiple derivative transcriptions for different uses.
For example, researchers using screen readers are benefited by over-correction, especially modernized punctuation and orthography. That's the opposite of scholarly best practice for diplomatics.
By contrast, one of our use-cases for raw HTR is determining the language of a document. Even #transkribus will hallucinate nonsense phrases in French or German when passed an image of a page that is blank except for ink bleed-through. For most purposes, this is no different from the gibberish punctuations produced by OCR, but for language determination it's a problem!
@waldoj I'm rather amazed to discover that even my iPhone can read some of that. Not great. But way better than I would have expected.
May, 10C€ 1852
My Dean Mif talerton,
have your not of datado
sohaiting the appointment of billian Midoud, a young artist li some consulate in Pal, that shall enable him to pay his way and purene his etudies, and regret to vary that there is se such place, which is vacant. the consulale at home is filled, and those at Laghorn and Florence gild only from 200 lo 50 anyear, and are whelly in. adequate for the purpores which you daine.
Same goin clisert
Millar Kenno
AC. 1443
@nazgul @waldoj Washington ly May, 100% 1852
My Dear Miss Waterston,
Shave your note of Satunday sobating the appointinent of William M.Sond, gouung artist lo some consulate in Italy, that shall enable him to pay his way and pursue his studies, and. such place which regret to say that there is no is vacant. The consulate at home filled, and those at Leghorn and Florence yield only from 200 to 250- a year, and are wholly in.adequate for the purposes which you desire. Saw your oft sert
AC. 1443
Waldo Jaquith
Nelson Minar
mnl mnl mnl mnl mnl
Jeff Atwood
Tim McCormack
Jed Brown
nothingfuture
Morten
gary
Kevin Patrick Doyle 🍅🥫
Ted Mielczarek
chx
🇳🇱 🇪🇺Jeroen 🇺🇦 🇺🇦
Andreas 'count' Kotes (he/him)
Ford Prefect
raphael
Aaron Brick — אהרן בריק
ManniCalavera
Ben W. Brumfield
Frederik Elwert
Morgan Aldridge
Kee Hinckley
mfriedenhagen