Mastodawn

Devin Prater :blind:

Okay so there’s been a tad bit of hyping up the Perspective Intelligence app for iOS. It’s accessible, made by a blind person, but don’t rely on it for anything. Here’s the output of asking it how to open a list of links with the JAWS screen reader:

To obtain a list of links from a webpage using the JAWS screen reader, you can follow these steps:

1. **Enter the Page**: Navigate to the webpage you want to check.

2. **Use the "Show Links" Option**:
- JAWS typically has an option to "Show Links" or "Show Links and Form Fields."
- Look for a button or a menu item labeled "Show Links" or "Show Links and Form Fields."
- Click on this option to display the list of links.

3. **Use the "Get Links" Command**:
- If you prefer using JAWS commands, you can use the "Get Links" command.
- On Windows, press `Ctrl + Shift + J`.
- On macOS, press `Cmd + Shift + J`.
- JAWS will then display a list of all the links on the page.

4. **Navigate the List**:
- JAWS will read the URL of each link to you.
- You can click on a link to follow it directly in the browser.

5. **Use the "Get Page Information" Command**:
- Another command you can use is "Get Page Information."
- On Windows, press `Ctrl + Shift + I`.
- On macOS, press `Cmd + Shift + I`.
- JAWS will provide information about the page, including links.

By using these methods, you can easily locate and navigate through the links on a webpage using JAWS.

Any JAWS user knows you hit Insert + F7. Any JAWS user knows there’s no JAWS for Mac. Local AI ain’t gonna be very helpful for anything important.

Ra (Freyja) (it/its)𒀭𒈹𒍠𒊩 Mar 19

@pixelate that's very clearly just Claude

Devin Prater :blind:Mar 19

@freya Nah, Apple Intelligence.

Ra (Freyja) (it/its)𒀭𒈹𒍠𒊩 Mar 19

@pixelate local models or which

Devin Prater :blind:Mar 19

@freya Probably local.

@pixelate You should ask it how to do the same thing with NVDA just for the lolz.

Devin Prater :blind:Mar 19

@cubic My frustration and depression can't take another one of those.

Darrell Bowles Mar 19

@NicksWorld @pixelate um, wow, totally not accurate. It's insert+f7!

Munchkinbear Mar 19

@vol4life8657 @NicksWorld @pixelate again, trust, but verify. holy fuck that was way wrong.

Jakob Rosin Mar 19

@pixelate Finally. Thanks to the intelligence, we are free from VoiceOver?

Devin Prater :blind:Mar 19

@jakobrosin Yep, free just like Siri lets us be.

Lino Morales the no good ham Mar 19

@pixelate Well take that up with Michael Doys or however you spell his last name. I'm sure their is a way to manually edt this INFO.

Devin Prater :blind:Mar 19

@Lino0876 Oh I know there's ways to have the AI grab that info. That app just doesn't do it.

Rich Yamamoto Mar 19

@pixelate @lerven Ahem. Jaws for Mac? Yes, I'll get right on that...not.

Robin Frost Mar 19

@pixelate Oh, my, goodness. I haven't words.

@pixelate Lololol!!! JAWS is on the Mac now? Holy Hallucinations Batman!

@mcourcel @pixelate They're using Parallels or VMWare. <lollollol>

@nycki96 @pixelate Yeah, that's what I was thinking as well. AI is picking up on that sort of info.

@mcourcel @nycki96 @pixelate I hope that in addition to poking fun and criticizing, that this was reported to the developer. If folks don’t want to do this, then perhaps use your coding expertise and build something better.

@pixelate @munchkinbear lol. Funnily enough, I know the person who built it.

Munchkinbear Mar 19

@Millerd16 @pixelate yup, @tayarndt might wanna look at that and fix it. lol.

Devin Prater :blind:Mar 20

@munchkinbear @Millerd16 @tayarndt I hope so. Would be nice to recommend if that RAG is free.

James Dean Mar 19

@pixelate Yes, this uses local models. Which is great for sensative app interactions which is apparently what it's good at. Asking it these kinds of questions is never going to work as well because it really needs access to the cloud to even have a high hope of giving the correct answer.

Devin Prater :blind:Mar 20

@GamingWithEars Nah, they can use RAG or tool calls to web search.

James Dean Mar 20

@pixelate Edit: Just read Michael's posts in response to your posts, so I understand that he is thinking about doing this for free users and all-access already does, which is good, so he's willing to budge on that a bit it seems. Right, but isn't that the entire point, that it doesn't go outside of your device? If you want something to answer questions like that, then use one of the Cloud models. Doesn't this fall under use the right tool for the job? I'm genuinely trying to understand. I'm still trying to figure this stuff out like we all are.

Devin Prater :blind:Mar 20

@GamingWithEars A blind person is going to try to use this for tech support. In fact I asked the model what it could do, and tech support was one of the things. And we all know how people will believe AI. But they can't just say their product sucks and people should use a cloud model.

@pixelate @GamingWithEars Heh. I've had too many failures with AI to just simply believe it at face value.

James Dean Mar 20

@Orinks @pixelate Yes as have I but Devin isn't wrong, some people do trust without verifying its output.

James Dean Mar 20

@pixelate But saying it sucks because it won't do something you think it should even though it's not specifically designed to do that isn't fare to Michael and Taylor.

Devin Prater :blind:Mar 20

@GamingWithEars It's not just me who will think it should answer a rather simple blindness question. Especially since it was written by a blind and low vision team of developers.

Pratik Patel Mar 19

@pixelate That is funny.

Michael Doise Mar 20

@pixelate Hi everyone,

I wanted to comment on what has been reported here since this was not sent to us through the app's reporting tool.

The app does use on device models, which means they know very little about assistive technology, or many topics in general. This is the case for many on device LLMs. I tested some in other apps that think that the latest version of iOS is iOS 8. (1/3)

Michael Doise Mar 20

Currently, the only way to get the app to give accurate information is to use web search with the all access membership. I am also looking at adding fine tuned knowledge through adapters and RAG which Devin has posted about here as well.

The thing to remember is that AI is not a source of truth. No model has correct information. They all have incorrect information or out of date information. It is just how they can get the correct information. (2/3)
@pixelate

Michael Doise Mar 20

Another thing you can do, is create your own custom user agents on the all access plan. This can give your chats different personalities and knowledge based on instructions.

I hope this gives some insight in to how the app works, and please reach out and let me know if there is something that doesn't work, or that can work better. I am always looking for ideas to improve the experience for everyone. (3/3)
@pixelate

Ryan Mann Mar 20

@mikedoise @pixelate if you create your own agent, can you share it with other people who use the app?

Michael Doise Mar 20

@progressivecat @pixelate Not as of yet, but thank you for the idea! I will be adding that soon.

Kaveinthran Mar 20

@mikedoise @pixelate I really appreciate your civility in responding, although it's tough. Something I'd suggest is that maybe you can steer the model to be more different, like saying "don't know" if you didn't know and ask further questions to clarify, or maybe allowing basic web search for free users, like limit them to three sources or two sources. But I guess devoting some attention to the engineering of response and Q&A could improve things. I'm not sure whether you can do those things, as it's better to do all these when we're training our model through reinforcement learning or other post training techniques. I'm not sure whether you can do this only with prompt engineering though, but you can try.
It's also very important to look at any eval work if it has been done for Apple Intelligence, why I'm saying so is this is a small model that probably use a small corpora of data that would surely not include the whole Internet and it's worrying when the model confident level is high on niche things. if the evaluation score is very low on factuality, then I guess you need to reassess your design principle on how you want the model to be used by the users.

Michael Doise Mar 20

@kaveinthran @pixelate So the important thing to remember is that AI is mostly wrong even ChatGPT is wrong if it couldn't search the web. The app is not supposed to be a source of truth but a way to brainstorm and come up with ideas. I'm adding a new feature for all access where the app can do a web search if the model produces bad content. (1/2)

Michael Doise Mar 20

We would love to have web search be free, but there are API costs that go along with this. I think we may look at a certain number of web sarches per day on the free version but I am not sure what that will look like yet. (2/2)
@kaveinthran @pixelate

Kaveinthran Mar 20

@mikedoise @pixelate You're right. You're doing great work. I don't know people that are playing with the foundational Apple Intelligence model to create stuff. So it's a good thing that you're doing here. Thanks for all your contributions and hard work.

Kaveinthran Mar 20

@pixelate I really love the imagination of this model. Simply Wow

The Witchy Bitches Mar 20

@pixelate OK, I read this and I just laughed because I don't work with jaws a lot because most of my students can't afford it so I don't actually know a lot of jaws screen reader commands off the top of my head but the links one is definitely one that I do know so this made me laugh extra hard plus that it thinks Jaws runs on a Mac lol

victor tsaran 5d ago

@pixelate @alexhall Yep, without some serious finetuning that model isn't useful for much. Though, to be fair, its intended use is not to be a knowledge base, more like a tool for summarization, proofreading, etc.

Lino Morales the no good ham 3d ago

@RaineyDay @pixelate Dev. Like I said last week to hell I can't remember who it was. Any get up with Michael. I'm sure he can correct any errors the AI product he's using. Just B/C you say its unreliable doesn't mean it is for other stuff you ask it. #justsayin