I think I have a genuine need for an #LLM. Can someone tell me if this is possible?
@openbenches contains ~40k text inscriptions.
Someone wants to know how many are dedicated to men, how many to women.
"To Grandma Sylvia" is obvious.
"To R Smith" is not.
Could an AI give a rough estimate of the gender of a subject?
Could it ignore text relating to who the inscription is from? "To Granny from Dave and Alice".
What would be the most accurate / cheapest / fastest / easiest tool to work with?
Not in any way I'd trust. LLMs tend to have huge gender biases that have, as of yet, gone unaddressed.
They are machines designed to make up plausible sounding responses that are difficult to concretely prove or falsify.
I don't recommend it.
@ajroach42 I guess that's part of the thing I'm trying to understand.
I can see how a biased system might insist that all "Sam"s are default male - or might not understand non-English languages.
But if I can see the assumptions that it makes, that could be very helpful.
Generally speaking, we don't have any insight into the specifics. These machines are essentially black boxes.
What we know about the things they get wrong come from people who are studying the outputs.
But the outputs are inconsistent.
@ajroach42 Yes, it's the outputs I'd like to study.
E.g.
Sylvia - Female 100% confidence
Joe - Male 100%
Jo - Female 70%
Sam - Male 80%
etc.
I appreciate it might be different each run, but would allow me to see how dodgy it was.
I haven't done that kind of testing personally, but I know there are some python utilities designed to do that kind of bias testing.
i haven't done much ML work since OpenAI hit the scene, so the specific analytical stuff I've done is very dated.
I've mostly stepped back and depended on studies others are posting.
The increased interest in analyzing and explaining gender inequalities in tech, media, and academia highlights the need for accurate inference methods to predict a person’s gender from their name. Several such services exist that provide access to large databases of names, often enriched with information from social media profiles, culture-specific rules, and insights from sociolinguistics. We compare and benchmark five name-to-gender inference services by applying them to the classification of a test data set consisting of 7,076 manually labeled names. The compiled names are analyzed and characterized according to their geographical and cultural origin. We define a series of performance metrics to quantify various types of classification errors, and define a parameter tuning procedure to search for optimal values of the services’ free parameters. Finally, we perform benchmarks of all services under study regarding several scenarios where a particular metric is to be optimized.
@benjamingeer thanks - that's useful!
Sadly it doesn't help with the extracting of names from unstructured data.
Others have pointed out too, LLMs are something I would not trust for the job. Since hallucinations are inevitable, that tells me it is the wrong tool for the job, unless you don't care about the accuracy of the results.
Someone else posted some code that will do this. I think regex or some *NIX command line tools could also accomplish this task.
But if accuracy if what you are looking for, then large language models are not the tool for the job.
@sauc3 I think you're misunderstanding me. I want to use the LLM to extract the names. Sure, it might hallucinate "Mr In Loving Memory" - but I consider that unlikely.
I simply can't see how a regex would work on such massively unstructured data.
I am not a regex guru, and the ethical downside of using large language models outweighs any other considerations for me.
But I guess if you don't care about ethics, do whatever you want.
@Edent If the inscriptions are in string form, not image form, this should be trivial, depending on what your acceptable error rate is, and what's your acceptable balance between not providing a response if unsure vs providing an erroneous response.
For 40k text inscriptions, you can even use <1B parameter models, here's a test with facebook's bart zero-shot classifier (which is ancient by today's LLM standards)
EDIT: model link https://huggingface.co/facebook/bart-large-mnli
@Edent definitely, many modern LLMs are trained (and then constrained during inference) to be able to produce structured JSON output, so you can just prompt an LLM like "tell me the gender of this person who the plaque is dedicated to, and, if available, who dedicated it to them", and then provide a JSON schema you want in return.
This webpage is a 10,000ft view of Structured Output for LLMs https://openrouter.ai/docs/guides/features/structured-outputs
@Edent are you sure that you need LLM?
If I were to find a solution I'd probably look at https://github.com/ajdavis/proporti.onl and expanded it with ML model. LLM looks a bit too strong for this.
@xgebi I think I need the LLM for extracting the names from unstructured data.
The classifier looks like it can be done with other things.
@Edent This seems like a reasonably good example of label-based classification: https://huggingface.co/blog/Ihor/refreshing-zero-shot-classification
Here's another example: https://jaketae.github.io/study/zero-shot-classification/
There are a variety of models, usually BERT (and derivative) models are the easiest to play with. Keywords you probably want to look for are NLI, Zero shot or one shot, entailment.
@Edent @openbenches I would give OpenRouter (where you can easily use different models) a try. With $5 credit you can do a lot. Every request costs 0,0001$ or similar (depending on the model). It’s quite easy to integrate with (same OpenAI API). I use it daily to automatically generate alt text, description and hashtags from images (when I automatically post to socials from @bookcorners )
Models I recommend: gpt-5.4-nano or gpt-5.4-mini
@aburka Yes, that's the point.
I want it to classify where there is confidence, and ignore where there is not.
Someone asked me if there were more benches dedicated to men or to women. That's all.
@Edent Well I had a quick go, 20 selected at random broke down as:
both: 4
f: 5
m: 9
n/a: 2
A second batch of 20 to compare broke down as:
both: 4
f: 5
m: 8
n/a: 2
unknown: 1 (ashley)
Surprisingly consistent (and corrected for the missing one). Be interested how those compare with a bigger analysis.
Your All-in-One Learning Portal: GeeksforGeeks is a comprehensive educational platform that empowers learners across domains-spanning computer science and programming, school education, upskilling, commerce, software tools, competitive exams, and more.
@gilesdring @Edent I did something similar with the charity register a while ago: https://dkane.net/2018/names-shared-by-genders/
The list of charity trustees has (or used to at least) some instances with an unambiguous title which I used to extract a list of names .
Terence Eden
Andrew (Television Executive)
Vale
Matthew
Benjamin Geer
sauc3
Bogdan Buduroiu
Liminal witch 🧙♀️ Sarah
Jade
Andrea Grandi 🦕
aburka 🫣
Dahie
Ben C
Giles
David Kane