Mastodawn

maegul Jul 29, 2023

I've been pondering the posts by @kissane and @siderea about 'not finding your people' #OnHere and am wondering if sentiments toward fediverse-wide #FullTextSearch have shifted at all. I know this has been implemented several times, and then been graciously shut down by developers who listened to community feedback.
#Mastodon #Fulltext #Search

From US - want opt-out fulltext search

37.3%

From US - DO NOT want opt-out fulltext search

13.6%

From Europe - want opt-out fulltext search

33.1%

From Europe - DO NOT want opt-out fulltext search

16.1%

Poll ended at Aug 5, 2023 at 2:15pm.

Brian Krohne Jul 29, 2023

More options:

Not US or EU - want opt-out fulltext search

77.8%

Not US or EU - DO NOT want opt-out fulltext search

22.2%

Poll ended at Aug 5, 2023 at 2:16pm.

Brian Krohne Jul 29, 2023

This is @kissane's deeply empathetic post where she listens to the concerns of people who have left Mastodon: https://mas.to/@kissane/110793942888550843

And @siderea's thoughtful thread about the different things folks mean by "finding your people": https://universeodon.com/@siderea/110794572555824059

Erin Kissane (@[email protected])

This week, I went over to Bluesky and asked people who'd left Mastodon why they left, and lots of people told me. I grabbed the replies and crunched them and wrote up a summary. I think it's really interesting and often kind of wrenching. https://erinkissane.com/mastodon-is-easy-and-fun-except-when-it-isnt #meta

mas.to

Vyr Cossont 🏳️‍🌈🏳️‍⚧️Aug 8, 2023

@krohne @kissane @siderea so heads up, full-text search is finally happening on vanilla Mastodon. it looks like they're going to be using the same combination of public post visibility and the discoverable account flag for opt-in as my Mastodon extended search patch (the one Universeodon runs)

https://github.com/mastodon/mastodon/pull/26344

Add new public status index by jsgoldstein · Pull Request #26344 · mastodon/mastodon

This is a follow up to #25951 The idea here is to allow public posts by discoverable users to show up in search results. This is done by adding a new Public Status Index to Elastic Search. A status...

GitHub

Erin Kissane Aug 9, 2023

@vyr I predict a whole lot of stress response incoming in 5…4…3 😬

(My non-existent kingdom for a community relations/comms team.)

@krohne @siderea

Vyr Cossont 🏳️‍🌈🏳️‍⚧️Aug 9, 2023

@krohne @siderea @kissane yeppppp

Siderea, Sibylla Bostoniensis Aug 9, 2023

@vyr Thanks for the alert!

@krohne @kissane

Jesse Jul 29, 2023

@krohne @kissane @siderea Opt-out should be on an instance basis. As in, by joining this instance you are opting-in to searchability, unless you explicitly opt-out.

Brian Krohne Jul 29, 2023

I do like the idea of people moving to instances where search happens, AND, and think we could let everyone opt-in by including this choice during the on-boarding process. All about what we want to encourage with UX patterns.

Vyr Cossont 🏳️‍🌈🏳️‍⚧️Jul 29, 2023

@siderea @kissane @krohne what the hell's wrong with opt-*in* full-text search

Brian Krohne Jul 29, 2023

I think opt-in search is great. I have signed up for @buercher's #TootFinder, but few people have. What if something like that was the default search in the UI, and people were encouraged opt-in when they signed up?

Vyr Cossont 🏳️‍🌈🏳️‍⚧️Jul 29, 2023

@krohne enshrining some third-party rando in the signup flow is not the way to do this.

the way to do this is to have a standard way to explicitly mark posts as searchable, the same way that Mastodon and several other Fedi servers already have a `discoverable` flag that explicitly marks accounts as searchable. that way, *any* search engine has a way of knowing what people have volunteered for.

https://github.com/mastodon/mastodon/pull/23808 unfortunately, the effort to patch this into Mastodon has stalled out.

Add setting to explicitly consent to full-text search indexing of public posts by ClearlyClaire · Pull Request #23808 · mastodon/mastodon

NOTE: This PR does not implement the search functionality itself, just a framework to express explicit consent for posts to be indexed Mastodon has historically refused to provide full-text search ...

GitHub

Brian Krohne Jul 29, 2023

A per-toot search opt is really interesting. I think that could work well with setting your search preferences during the on-boarding process (e.g. by default my toots should be [ ] Searchable [ ] Not Searchable)

Erin Kissane Jul 29, 2023

@krohne This is 100% my preferred implementation fwiw. Set an account default and have per-post controls. Onboarding needs a really nice visual wizardy walkthrough as well, god help us.

@kissane @krohne the onboarding for #FireFish is so much better - really much more user friendly.

Vyr Cossont 🏳️‍🌈🏳️‍⚧️Jul 29, 2023

@krohne if we don't build something like this, "is this post set to public visibility" is the only intent signal to work with, and searching all public posts surprises Mastodon users.

(and only Mastodon users. "public = searchable" is the rule for full-text search implementations on most other Fedi servers that have it, except for GotoSocial.)

Vyr Cossont 🏳️‍🌈🏳️‍⚧️Jul 29, 2023

@krohne UX-wise, agreed that there should be an opt-in button in the signup flow. you should be able to switch this on easily. but it should never be switched on for you.

Siderea, Sibylla Bostoniensis Aug 3, 2023

@vyr
> the way to do this is to have a standard way to explicitly mark posts as searchable

Man, the problem I have with this is I don't want to mark my posts either universally searchable or not universally searchable. I want to be able to pick and choose which search engines I allow to index my content!

As a side note, part of why I would like to have that option is that I can imagine a little social movement where people in the Fediverse decide to make a point of not letting Google Search index their posts, gravitating some other search engine, thereby giving that search engine a commercial advantage over Google Search, at least for searching the Fediverse.

Vyr Cossont 🏳️‍🌈🏳️‍⚧️Aug 3, 2023

@siderea @krohne hi, not a man

Siderea, Sibylla Bostoniensis Aug 3, 2023

@vyr Oh, sorry, I wasn't aiming that at you, I meant in the other rhetorical sense of "oh man".

Vyr Cossont 🏳️‍🌈🏳️‍⚧️Aug 3, 2023

@siderea @krohne yeah well don't get any of your rhetoric on my gender, thanks, i just got it the way i like it 😝

anyway, controlling how posts get exposed to web scrapers like Google is a related but different issue. specifically, you first have to somehow have enough leverage with Google to get them to agree to whatever scheme you propose, because they don't care about ActivityPub as such. they only see web pages, and those web pages are rendered by a dozen kinds of Fediverse server, and right now, even clicking the "hide me from search engines box" doesn't work 100% on your local instance, let alone once your posts leave it.

Vyr Cossont 🏳️‍🌈🏳️‍⚧️Aug 3, 2023

@krohne @siderea meanwhile, the most commonly deployed opt-in scheme across the Fediverse right now is "if you post with public visibility, you are opting into everything", and if nobody comes up with anything better, that's going to crystallize.

Matthias Bürcher Jul 29, 2023

@krohne An instance using this approach on sign up would probably instance opt-in.

Matt Jul 29, 2023

@vyr @krohne The thing is, I'd gather that most people that the "full text search" arguments are trying to attract are the type of people who will never configure an option, so opt-out makes a lot more sense for the usability of the feature.

Opt-in and opt-out are very, very different kettles of fish, and opt-out is always more effective for the usability of a feature that relies on other people having it activated.

Another approach would be asking people during onboarding if they want their posts indexed (or removed from the index), but even then the default is going to matter as many people skip this sort of stuff as they just want to post.

Defaults will always be what is used by 90%+ of people - which is why there's always so much pressure on Mastodon and not other platforms to implement features. A lot of other platforms have these features, but most people will only experience Mastodon default.

Vyr Cossont 🏳️‍🌈🏳️‍⚧️Jul 29, 2023

@mattswift @krohne there's always the possibility of an "opt-either" onboarding UX. consider the following:

"i want to be visible" on the left half of the screen vs. "i want to be quiet" on the right. one click defaults.

left one turns on discoverable and searchable flags, sets your default visibility to public, and disables the opt-out for web crawlers.

right one leaves the flags off, sets your default visibility to unlisted, tells web crawlers to ignore you, hides your social graph, and requires follow request approval.

anyone who wants further customization can go digging for it.

Matt Jul 29, 2023

@vyr @krohne This is a pretty good idea, people are most likely going to follow patterns in all their options, so a couple set of "defaults" makes a lot of sense!

ben Jul 29, 2023

@krohne @kissane @siderea It's the opt-out part I'd oppose.

If opt-out full-text search had existed from the beginning, I'd have no objection. People would effectively opt in when they registered their account, and could disable it immediately if they wished. But enabling it after the fact for hundreds of thousands of users, including those who are expressly against it, seems insupportable.

At the very least it should be opt-in for existing accounts.

Brian Krohne Jul 29, 2023

That's fair. I have no interest in searching people who don't want to be searched. What if one day (after a good opt-in search was fully-baked) the mastodon software showed everyone one of the (Do you want to accept Cookies Y/N) banners, except it would be (Do you want your posts to be indexed in fediverse-wide search?)

ben Jul 29, 2023

@krohne Prompting someone to opt in or out seems fine by me, yeah. It's the consent that's important.

ScottMGS Jul 29, 2023

@krohne @kissane @siderea I think this needs to be *opt-in* at the server level by having the server register with an external service. I'm sure I don't understand the technical details but I think my single-user Mastodon instance data limits would be swamped just by the external *requests* for full-text searches.

Also, this can't just be a Mastodon thing. It would have to be a modification to the ActivityPub standard, right?

Brian Krohne Jul 29, 2023

Perhaps we have an instance admin-level option that says "No one on my server can opt-in to search." I know there are people who would want that.
The type of fediverse-wide search I am thinking of would not query every server every time there was a search - there would have to be a central index. Because of the way activitypub promulgates toots between instances, the indexer wouldn't necessarily have to connect to every instance either.

ScottMGS Jul 29, 2023

@krohne Yeah, it probably would require a centralized index but that would somehow need to take into account which servers are federated and which are not. Doing a full text search and being presented with harmful posts (racist, anti-trans, pro-nazi, etc.) from Truth Social or Gab or whatever isn't going to fly. As well, having people from defederated instances reach posts through a central repository is also going to be problematic.

Centralization itself may be anathema to federation.

ScottMGS Jul 29, 2023

@krohne This may require stepping back and assessing the use cases for search and addressing them individually. What are all the reasons that people want full search. Assess each case for technical feasibility as well as benefit and harm reduction within the federation paradigm,

Brian Krohne Jul 29, 2023

Surfacing unsavory content has been an issue with some of the fedi search implementations that have come and gone. We would want to see any search using the standard @oliphant blocklists at a minimum.

As for having people from blocked instances (or just anyone on the internet) being able to find your posts, that is a legitmate tradeoff. I would hope being able to block search at the post/user/instance level would help with this.

Brian Krohne Jul 29, 2023

Once we had some proper standards for this, I don't think there would be just one search provider, rather I would expect there would be different options that instance admins could connect to.

THOMnottom Aug 1, 2023

@krohne @kissane @siderea Provisionally - I'd recommend opt-in as the standard for most options like this.

Marius Aug 1, 2023

@krohne @kissane @siderea Why not have "no search" and "opt-in search only" as options?

Zaya the Jew Aug 10, 2023

@krohne What is your thinking behind the geographical categorisation? (I don't mind it, just curious.)