Frederik Elwert
In a methods / #DigitalHumamities class next semester, I want to cover basic corpus creation. Especially, I’ll probably focus on #OCR/#HTR/#ATR and #WebScraping. I find it incredibly hard to find good papers that can serve as a general introduction into these topics. All I find are either practical tutorials, or very specialized papers about specific approaches. Do you have any favorite readings about how to get to a text corpus in DH in the first place? Please share!
Sep 25, 2024 at 10:37amWeb
Show threadFrederik ElwertSep 26, 2024

There seems to be a lot of interest in the question (thanks for the boosts!), but not so many suggestions yet. So I thought I’d share what I have found so far:

Re WebScraping, I think this paper by Black is a really good high-level overview: Black, Michael L. 2016. “The World Wide Web as Complex Data Set: Expanding the Digital Humanities into the Twentieth Century and Beyond through Internet Research.” IJHAC 10 (1): 95–109. https://doi.org/10.3366/ijhac.2016.0162.

Show threadFrederik ElwertSep 26, 2024

Re OCR/ATR, interestingly the #OCR4all paper also offers a very good overview of the different steps and workflows. It has a different purpose, but I think it can still be used in a class context.

Reul, Christian et al. 2019. “OCR4all—An Open-Source Tool Providing a (Semi-)Automatic OCR Workflow for Historical Printings.” Applied Sciences 9 (22): 4853. https://doi.org/10.3390/app9224853.

OCR4all—An Open-Source Tool Providing a (Semi-)Automatic OCR Workflow for Historical Printings

Optical Character Recognition (OCR) on historical printings is a challenging task mainly due to the complexity of the layout and the highly variant typography. Nevertheless, in the last few years, great progress has been made in the area of historical OCR, resulting in several powerful open-source tools for preprocessing, layout analysis and segmentation, character recognition, and post-processing. The drawback of these tools often is their limited applicability by non-technical users like humanist scholars and in particular the combined use of several tools in a workflow. In this paper, we present an open-source OCR software called OCR4all, which combines state-of-the-art OCR components and continuous model training into a comprehensive workflow. While a variety of materials can already be processed fully automatically, books with more complex layouts require manual intervention by the users. This is mostly due to the fact that the required ground truth for training stronger mixed models (for segmentation, as well as text recognition) is not available, yet, neither in the desired quantity nor quality. To deal with this issue in the short run, OCR4all offers a comfortable GUI that allows error corrections not only in the final output, but already in early stages to minimize error propagations. In the long run, this constant manual correction produces large quantities of valuable, high quality training material, which can be used to improve fully automatic approaches. Further on, extensive configuration capabilities are provided to set the degree of automation of the workflow and to make adaptations to the carefully selected default parameters for specific printings, if necessary. During experiments, the fully automated application on 19th Century novels showed that OCR4all can considerably outperform the commercial state-of-the-art tool ABBYY Finereader on moderate layouts if suitably pretrained mixed OCR models are available. Furthermore, on very complex early printed books, even users with minimal or no experience were able to capture the text with manageable effort and great quality, achieving excellent Character Error Rates (CERs) below 0.5%. The architecture of OCR4all allows the easy integration (or substitution) of newly developed tools for its main components by standardized interfaces like PageXML, thus aiming at continual higher automation for historical printings.

MDPI
Show threadAlexander WinklerSep 26, 2024
@felwert vielleicht ist da ja etwas dabei:
Barbaresi, Adrien, und Jens Pohlmann. „A Reproducible IT-Blog Corpus“. Journal of Open Humanities Data 7 (22. Juli 2021): 17. https://doi.org/10.5334/johd.35.
Dumitru, Vlad, Denis Iorga, Stefan Ruseti, und Mihai Dascalu. „Garbage in, garbage out: An analysis of HTML text extractors and their impact on NLP performance“. In 2023 24th International Conference on Control Systems and Computer Science (CSCS), 403–10. IEEE, 2023. https://ieeexplore.ieee.
1/4
A Reproducible IT-Blog Corpus | Journal of Open Humanities Data

The Journal of Open Humanities Data (JOHD) aims to be a key part of a thriving community of scholars sharing humanities data. The journal features peer reviewed publications describing humanities research objects or techniques with high potential for reuse. Humanities subjects of interest to JOHD include, but are not limited to Art History, Classics, History, Library Science, Linguistics, Literature, Media Studies, Modern Languages, Music and musicology, Philosophy, Religious Studies, etc. Submissions that cross one or more of these traditional disciplines are particularly encouraged.  

Journal of Open Humanities Data
Show threadAlexander WinklerSep 26, 2024
@felwert org/abstract/document/10214756/.
Hartmann, Stefan. „Open Corpus Linguistics–or How to overcome common problems in dealing with corpus data by adopting open research practices“, 2023. https://osf.io/preprints/psyarxiv/ywg5v/.
Laippala, Veronika, Samuel Rönnqvist, Saara Hellström, Juhani Luotolahti, Liina Repo, Anna Salmela, Valtteri Skantsi, und Sampo Pyysalo. „From web crawl to clean register-annotated corpora“. In Proceedings of the 12th Web as Corpus Workshop, 14–22, 2020.
2/4
OSF

Show threadAlexander WinklerSep 26, 2024
@felwert https://aclanthology.org/2020.wac-1.3/.
McCarthy, Michael, und Anne O’Keeffe. „Historical perspective: What are corpora and how have they evolved?“ In The Routledge Handbook of Corpus Linguistics. Routledge, 2010.
Reuter, Kevin, und Lucien Baumgartner. „Corpus Analysis: Building and Using Corpora—A Case Study on the Use of “Conspiracy Theory”“. In Experimental Philosophy for Beginners: A Gentle Introduction to Methods and Tools, herausgegeben von Stephan Kornmesser, Alexander
3/4
From Web Crawl to Clean Register-Annotated Corpora

Veronika Laippala, Samuel Rönnqvist, Saara Hellström, Juhani Luotolahti, Liina Repo, Anna Salmela, Valtteri Skantsi, Sampo Pyysalo. Proceedings of the 12th Web as Corpus Workshop. 2020.

ACL Anthology
Show threadAlexander WinklerSep 26, 2024
@felwert Max Bauer, Mark Alfano, Aurélien Allard, Lucien Baumgartner, Florian Cova, Paul Engelhardt, u. a., 275–320. Cham: Springer International Publishing, 2024. https://doi.org/10.1007/978-3-031-58049-9_6.
Xu, Zhipeng, Zhenghao Liu, Yukun Yan, Zhiyuan Liu, Ge Yu, und Chenyan Xiong. „Cleaner Pretraining Corpus Curation with Neural Web Scraping“. arXiv, 14. Juni 2024. http://arxiv.org/abs/2402.14652.
4/4
Show threadJeremy HuggettSep 26, 2024
@felwert
Paige, J. (2024) ‘The Legality and Ethics of Web Scraping in Archaeology’, Advances in Archaeological Practice, 12(2), pp. 98–106. http://doi.org/10.1017/aap.2023.42 may be worth a look? It's intended to act as an introduction to webscraping as a research method.
The Legality and Ethics of Web Scraping in Archaeology | Advances in Archaeological Practice | Cambridge Core

The Legality and Ethics of Web Scraping in Archaeology - Volume 12 Issue 2

Cambridge Core
Show threadFrederik ElwertSep 26, 2024
@BlckheathHopper That looks very interesting, thanks!
Show threadGiulio Cesare SolaroliSep 26, 2024
@felwert @raffaele
Raffaele, do you have any suggestions for Frederik?
Show threadraffaeleSep 27, 2024
@gcsolaroli @felwert someone already suggested ocr4all. tomorrow I’ll try to search my bookmarks, but I think many of things I saved are tech tutorials
Show threadAdrien BarbaresiOct 2, 2024
@felwert I gather information on web texts in #DigitalHumanities contexts on this page, this could be another starting point with references:
https://trafilatura.readthedocs.io/en/latest/compendium.html
Compendium: Web texts in linguistics and humanities — trafilatura 1.12.2 documentation

This page summarizes essential information about building and operation of web text collections. It primarily addresses concerns in linguistics and humanities.

Show threadFrederik ElwertOct 2, 2024
@adbar Thanks, that's helpful!
Show threadPeter BinkleyOct 3, 2024
@felwert would you go as far as looking at RAG applied to a corpus, with tools like this? https://lil.law.harvard.edu/blog/2024/02/12/warc-gpt-an-open-source-tool-for-exploring-web-archives-with-ai/ (which works on a WARC, so it would fit with your web scraping example). I've been meaning to try this (with an open mind and a skeptical eye) but haven't got to it yet
WARC-GPT: An Open-Source Tool for Exploring Web Archives Using AI | Library Innovation Lab

Today we’re releasing WARC-GPT: an open-source, highly-customizable Retrieval Augmented Generation tool the web archiving community can use to explore the in...

Show threadFrederik ElwertOct 3, 2024
@pbinkley Looks very interesting, I'll definitely have a look!
Show threadPeter BinkleyOct 3, 2024
@felwert If you use it, I hope you'll post about the experience!