BrianKrebsJun 23, 2023

Via Slashdot, in the because-of-course-they-are and garbage-in-garbage-out dept.

The people paid to train AI are outsourcing their work… to AI
It’s a practice that could introduce further errors into already error-prone models.

"....a team of researchers from the Swiss Federal Institute of Technology (EPFL) hired 44 people on the gig work platform Amazon Mechanical Turk to summarize 16 extracts from medical research papers. Then they analyzed their responses using an AI model they’d trained themselves that looks for telltale signals of ChatGPT output, such as lack of variety in choice of words. They also extracted the workers’ keystrokes in a bid to work out whether they’d copied and pasted their answers, an indicator that they’d generated their responses elsewhere.

They estimated that somewhere between 33% and 46% of the workers had used AI models like OpenAI’s ChatGPT. It’s a percentage that’s likely to grow even higher as ChatGPT and other AI systems become more powerful and easily accessible, according to the authors of the study, which has been shared on arXiv and is yet to be peer-reviewed."

https://www.technologyreview.com/2023/06/22/1075405/the-people-paid-to-train-ai-are-outsourcing-their-work-to-ai/

The people paid to train AI are outsourcing their work… to AI

It’s a practice that could introduce further errors into already error-prone models.

MIT Technology Review
Squash FollowerJun 23, 2023
Show threadInfoSecBen
@briankrebs as Rodney Brooks said, we keep confusing performance for competence https://spectrum.ieee.org/gpt-4-calm-down
Just Calm Down About GPT-4 Already

<p>And stop confusing performance with competence, says Rodney Brooks</p>

IEEE Spectrum
Jun 23, 2023 at 2:25pmWeb
Show threadRyan MoreyJun 23, 2023
@Infosecben @briankrebs the distinction between performance and competence is not at all made clear in this interview. the key anecdote that’s taken from is this frisbee thing. Current systems (namely, GPT-4 multimodal) absolutely could identify “that's a person playing frisbee” as well as answer the questions he poses. So it's an interesting interview, but the catchy headline is not really substantiated.
Show threadTurduckenJun 23, 2023
@rmorey @Infosecben @briankrebs I think the point isn't so much that we can't build a system that can answer those questions, but rather we still can't build systems that understand the domain in which they operate. Besides we don't actually know how well GPT4 can answer those questions anyways. The only demonstration of its multimodal capabilities was in a paper published by OpenAI months ago at the peak of its hype cycle. The paper was not peer reviewed and had more than a few issues.
Show threadRyan MoreyJun 23, 2023
@Turducken @Infosecben @briankrebs good points. I have yet to find a robust definition of understanding that doesn't apply in some part to current LLM systems, I would be curious to hear your definition. GPT4 multi-modal capability is actually rolling out in Bing search right now, some users have access. Ethan Mollick is one: https://twitter.com/emollick/status/1671692075144806400 still to be generally shown what it's capabilities actually are. But basic image captioning is surely within reach.
Ethan Mollick on Twitter

“OK, I am pretty blown away by what Bing/GPT-4 can do when its ability to process images are turned on. I think this is going to be very useful. I pulled an image from Reddit from 5 months ago (so post training) and asked it for help - and look at its comment on the sticker!”

Twitter
Show threadTurduckenJun 23, 2023
@rmorey @Infosecben @briankrebs Multi modal models are interesting because they can effectively take any form of information and map it to a shared vector/embedding space, which can then be used to convert those representation vectors into a different medium then what they were originally, text->audio. However, while these models can create a map of 'meaning' they are still effectively just mapping functions. They map inputs to most likely outputs without doing much more than that.
Show threadJ. PetersonJun 23, 2023

@Infosecben @briankrebs Can't agree with all of his conclusions. We don't have a shortage of truck drivers because potential drivers thought AI would replace them.

We have a shortage of truck drivers because it devolved from a good union job w/benefits to poorly paying gig work.