Devin Prater :blind:Mar 19

Okay so there’s been a tad bit of hyping up the Perspective Intelligence app for iOS. It’s accessible, made by a blind person, but don’t rely on it for anything. Here’s the output of asking it how to open a list of links with the JAWS screen reader:

To obtain a list of links from a webpage using the JAWS screen reader, you can follow these steps:

1. **Enter the Page**: Navigate to the webpage you want to check.

2. **Use the "Show Links" Option**:
- JAWS typically has an option to "Show Links" or "Show Links and Form Fields."
- Look for a button or a menu item labeled "Show Links" or "Show Links and Form Fields."
- Click on this option to display the list of links.

3. **Use the "Get Links" Command**:
- If you prefer using JAWS commands, you can use the "Get Links" command.
- On Windows, press `Ctrl + Shift + J`.
- On macOS, press `Cmd + Shift + J`.
- JAWS will then display a list of all the links on the page.

4. **Navigate the List**:
- JAWS will read the URL of each link to you.
- You can click on a link to follow it directly in the browser.

5. **Use the "Get Page Information" Command**:
- Another command you can use is "Get Page Information."
- On Windows, press `Ctrl + Shift + I`.
- On macOS, press `Cmd + Shift + I`.
- JAWS will provide information about the page, including links.

By using these methods, you can easily locate and navigate through the links on a webpage using JAWS.

Any JAWS user knows you hit Insert + F7. Any JAWS user knows there’s no JAWS for Mac. Local AI ain’t gonna be very helpful for anything important.

#AI

Show threadMichael Doise

@pixelate Hi everyone,

I wanted to comment on what has been reported here since this was not sent to us through the app's reporting tool.

The app does use on device models, which means they know very little about assistive technology, or many topics in general. This is the case for many on device LLMs. I tested some in other apps that think that the latest version of iOS is iOS 8. (1/3)

Mar 20 at 1:29amWeb
Show threadMichael DoiseMar 20

Currently, the only way to get the app to give accurate information is to use web search with the all access membership. I am also looking at adding fine tuned knowledge through adapters and RAG which Devin has posted about here as well.

The thing to remember is that AI is not a source of truth. No model has correct information. They all have incorrect information or out of date information. It is just how they can get the correct information. (2/3)
@pixelate

Show threadMichael DoiseMar 20

Another thing you can do, is create your own custom user agents on the all access plan. This can give your chats different personalities and knowledge based on instructions.

I hope this gives some insight in to how the app works, and please reach out and let me know if there is something that doesn't work, or that can work better. I am always looking for ideas to improve the experience for everyone. (3/3)
@pixelate

Show threadRyan MannMar 20
@mikedoise @pixelate if you create your own agent, can you share it with other people who use the app?
Show threadMichael DoiseMar 20
@progressivecat @pixelate Not as of yet, but thank you for the idea! I will be adding that soon.
Show threadKaveinthranMar 20
@mikedoise @pixelate I really appreciate your civility in responding, although it's tough. Something I'd suggest is that maybe you can steer the model to be more different, like saying "don't know" if you didn't know and ask further questions to clarify, or maybe allowing basic web search for free users, like limit them to three sources or two sources. But I guess devoting some attention to the engineering of response and Q&A could improve things. I'm not sure whether you can do those things, as it's better to do all these when we're training our model through reinforcement learning or other post training techniques. I'm not sure whether you can do this only with prompt engineering though, but you can try.
It's also very important to look at any eval work if it has been done for Apple Intelligence, why I'm saying so is this is a small model that probably use a small corpora of data that would surely not include the whole Internet and it's worrying when the model confident level is high on niche things. if the evaluation score is very low on factuality, then I guess you need to reassess your design principle on how you want the model to be used by the users.
Show threadMichael DoiseMar 20
@kaveinthran @pixelate So the important thing to remember is that AI is mostly wrong even ChatGPT is wrong if it couldn't search the web. The app is not supposed to be a source of truth but a way to brainstorm and come up with ideas. I'm adding a new feature for all access where the app can do a web search if the model produces bad content. (1/2)
Show threadMichael DoiseMar 20
We would love to have web search be free, but there are API costs that go along with this. I think we may look at a certain number of web sarches per day on the free version but I am not sure what that will look like yet. (2/2)
@kaveinthran @pixelate
Show threadKaveinthranMar 20
@mikedoise @pixelate You're right. You're doing great work. I don't know people that are playing with the foundational Apple Intelligence model to create stuff. So it's a good thing that you're doing here. Thanks for all your contributions and hard work.