This is fineš„š¶āš„Microsoft launches āvibe workingā in Excel and Word
InEnduringGrowStrongMicrosoft says its Agent Mode in Excel has an accuracy rate of 57.2 percent in SpreadsheetBench, a benchmark for evaluating an AI modelās ability to edit real world spreadsheets.
It generates 42.8% bullshit.
jubilationtcornponeThey probably view that as a statistic worth bragging about. Itās not.
If Excel got calculations right 57.2% of the time it would be completely worthless.
Did you read the next sentence? Humans only get like 72% right. Itās not far off at all.
Depending on where you go to school, 70% is passing while 50% is not. While ānot far off,ā one is a C, the other a F.
Thatās not at all what this means. In this instance, 70% is basically āhuman levelā.
I wonder where that āhuman accuracyā statistic is coming from. Plenty of people donāt know how to read and interpret data, much less use excel in the first place. Thereās a difference between 1/4 of people in the workforce not being able to complete a task, and a specialized AI not being able to complete a task. Additionally, this is how you get into the KPI as a goal rather than a proxy issue. AI will never understand context isnāt directly provided in the workbook. If you introduced a new drink at your restaurant in 2020 AI will tell you that the introduction of the drink caused a 100% decrease in foot traffic since thereās no line item for āglobal pandemicā. Iām not saying AI will never be there, but people using this version of AI instead of actual analysis donāt care about the facts and just want an answer and for that answer to be cheap.
As Iāve said many times, though not in this topic - AI is a tool to be used, and using it is a skill that needs to be learned.
For your pandemic example, thatās something that you would need to provide the AI with the context of. The joke of a āprompt engineerā being a job soon actually has merit, in that you want people who know how to use their tools the best. Itās constantly learning through iteration to give the AI a specific instruction set to get the results you want/need.