Renaud ChaputAug 24, 2023

Full text search has been merged in #Mastodon `main` branch, and will be in the next (and final?) 4.2.0 beta 🎉

It is opt-in, so it will take some time to be filled with people content as they enable their profile to be indexed, but this was one of the most wanted Mastodon features for some time.

We plan to deploy it to mastodon.social and mastodon.online in the coming days to have a bit more feedback on it and see how it behaves in the wild.

Show threadRenaud Chaput

To opt into your content being searchable, once your instance is upgraded to support this feature, head to the your profile, and the new “Privacy and reach" tag, then tick the "include public posts in search”.

We will try to ensure users know this setting exists, nothing 100% defined yet, but we may add a dismissible banner asking to review your privacy settings page.

Aug 24, 2023 at 7:17pmWeb
Show threadRenaud ChaputAug 24, 2023

Full-text search allows you to search for words in the content of statuses (if your instance has ElasticSearch configured), compared to the current version which only allows searching for hashtags.

In addition, you get a whole lot of search modifiers for advanced search (no UI for those yet):

has:image / has:video / has:audio / has:media
has:poll / has:link / has:embed
is:sensitive / is:reply
language:en
from:account
before:2023-08-08
after:2023-08-08
during:2023-08-08

Show threadJason Miller 🦊⚛️Aug 24, 2023
@renchap will the new search modifiers like `has:image` work on content that is not full-text searchable?
Show threadjrmAug 24, 2023
@renchap Sorry to ask but can we do a search for our own faved toots that has a specific text ?
Show threadCrazyMyraAug 24, 2023
@renchap How do we opt out of this nonsense?
Show threadSander van KasteelAug 24, 2023
@OutOnTheMoors @renchap It is specifically opt-in. So it's not enabled by default.
Show threadRenaud ChaputAug 24, 2023
@OutOnTheMoors This new feature is opt-in, so you do not need to opt out.
Your content will not be indexed unless you tick the new privacy setting to allow it.
Show threadCrazyMyraAug 24, 2023
@renchap I've heard this a few times before about Mastodon features 😂
Show threadCrazyMyraAug 24, 2023
@renchap When I search my @ on Google, my own posts don't immediately appear. But all searchable replies do, so it takes a flick of the finger to scroll up and find my posts.
This is exactly the same "protection" you're promising here, n'est-ce pas?
Show threadMark HughesAug 24, 2023

@OutOnTheMoors @renchap

And regardless, when being opt-in causes it to fail and people start clamouring for opt-out what then?

Search shouldn't be supported because it is being demanded by people who want this to be like Twitter, when it's strength comes from not being Twitter. 🤦‍♂️

Mastodon should move away from not towards Twitter.

Show threadCrazyMyraAug 24, 2023
@markhughes @renchap There aren't even the guardrails here that are still available on Twitter.
Show threadLiz Ellis She/Her🌳Aug 24, 2023
@OutOnTheMoors @markhughes @renchap They need to give users the option of posts being hidden to followers only without having to toggle each post.
Show threadCrazyMyraAug 24, 2023
@LizEllisPhD @markhughes @renchap Or reply-limiting. At the moment, it's wingnuts flying in from everywhere
Show threadLiz Ellis She/Her🌳Aug 25, 2023
@OutOnTheMoors @markhughes @renchap I'm lucky, I fly under the radar! I guess just like on twitter women and minority groups just have to be completely locked down or block like a motherfucker
Show threadLovesTha🥧Aug 25, 2023
@LizEllisPhD @OutOnTheMoors @markhughes @renchap You mean this setting? Does it not work?
Show threadMark HughesAug 25, 2023

@LizEllisPhD @OutOnTheMoors @renchap I wouldn't use that because it is too restrictive.

I *want* people to discover me, but through organic connections such as boosts by those they follow. That makes for organic, growth that is more about quality and connection.

Search is a different thing and will change the nature of connections, communities and Mastodon.

Show threadMackajAug 25, 2023

@markhughes

Other people want to use Mastodon differently though and their needs are just as valid.

The developers are trying to meet the requirements for a broad spectrum of people and find the right balance. That's not an easy thing to do but I think they're making a solid attempt.

@LizEllisPhD @OutOnTheMoors @renchap

Show threadMark HughesAug 25, 2023

@mackaj
Compromise isn't appropriate in all situations, such as when it moves the project towards features that were defined not for users benefit, but for profit

Because users want, or expect features they are used to elsewhere, perhaps without knowing or thinking about their impact, is not enough reason to accede to their requests. Let them choose a service that suits them rather than degrade one that works well and was built w different goals/values.

@LizEllisPhD @OutOnTheMoors @renchap

Show threadMackajAug 25, 2023

@markhughes

Who is making profit out of this?

Discoverability has been Mastodon's Achilles heal and these improvements will go some way to addressing that.

Personally I welcome these changes. I know you won't agree, and that's fine.

@LizEllisPhD @OutOnTheMoors @renchap

Show threadMark HughesAug 25, 2023

@mackaj

Twitter was built for profit and that's the reason Mastodon is and must stay different, and not try to mimic a system that was built for profit rather than users' benefit.

Perhaps when you consider that, and that there are different use cases, some benefiting from search and some hindered or degraded by it you will see the point I'm making.

Because something was useful on Twitter doesn't make it desirable here unless you want Twitter again.

@LizEllisPhD @OutOnTheMoors @renchap

Show threadMackajAug 25, 2023

@markhughes

I don't get it. There are no adverts here to drive a profit economy off the back of Mastodon. And I fail to see how improving the ability of users to find others who are enjoying the same game they are playing, or reading the same scientific paper is going to change that in any way.

@LizEllisPhD @OutOnTheMoors @renchap

Show threadMark HughesAug 25, 2023

@mackaj

I've tried to explain my point as clearly as I can but you haven't understood it so I'll leave it there.

@LizEllisPhD @OutOnTheMoors @renchap

Show threadWiredfire Aug 25, 2023
@LizEllisPhD @markhughes @mackaj @OutOnTheMoors @renchap it’s an open platform. Fork the code, fire up your own server, set your own rules. You can still connect with the rest of the fediverse but do so on your own terms. It’s rather good.
Show threadCrazyMyraAug 25, 2023
@markhughes @LizEllisPhD @renchap This is the healthy attitude
Show threadThiagoAug 25, 2023
@markhughes @LizEllisPhD @OutOnTheMoors @renchap have you tried looking if glitch-soc, pleroma, misskey, calkey or other software gives you what you want?
Show threadMackajAug 25, 2023

@LizEllisPhD

You have that already in Preferences -> Other -> Posting Privacy

@OutOnTheMoors @markhughes @renchap

Show threadLiz Ellis She/Her🌳Aug 25, 2023
@mackaj @OutOnTheMoors @markhughes @renchap Thanks, I can see it on my browser, but not on my app (which to be fair defaults to followers only). I thought maybe it's a bug depending on version?
Show threadMackajAug 25, 2023

@LizEllisPhD

Oh, I don't know if the app follows the account settings, I just use the web app even on my mobile.

Hope this works out for you as you need 🤞

@OutOnTheMoors @markhughes @renchap

Show threadDaniel Schwarz Aug 24, 2023
@markhughes @OutOnTheMoors @renchap oh well, guess you’ll need to fork the codebase then.
Show threadCharlesAug 24, 2023
@OutOnTheMoors @renchap You want your posts to be public or not? Once you've given people the right to view your posts you've given them he right to index them and link to them, otherwise search engines wouldn't be a thing.
Show threadCrazyMyraAug 24, 2023
@aeftaw Fuck off
Show threadWiredfire Aug 25, 2023

@OutOnTheMoors @renchap we need to all be mindful that we’re posting publicly. Public web content gets indexed and scraped.

If you don’t want what your posting to be see at all by anyone other than the intended recipient then Mastodon is not the right platform to use. Most social media wouldn’t be.

Show threadStarkAug 24, 2023

@OutOnTheMoors @renchap

Most of your posts are polls where people opt-in, so the irony that you don't understand this is phenomenal.

Show threadCrazyMyraAug 24, 2023
@Stark9837 Fuck off, cunt
Show threadJoe ❌👑Aug 24, 2023
@OutOnTheMoors @renchap You don't need to. It is opt in. If you do nothing your toots are not searchable.
Show threadAxel GutmannAug 24, 2023
@OutOnTheMoors @renchap It's opt-in. You should read posts to the end before replying.
Show threadCrazyMyraAug 24, 2023
@virbonus @renchap Get lost
Show threadladyterukiAug 24, 2023
@renchap : if I opt-in and don't like it, then opt-out, how does it work ? For instance are toots posted while I opted in still searchable ?
Show threadRenaud ChaputAug 24, 2023
@ladyteruki This feature works with an "indexable" flag set on your profile.
This flag is federated, so if you change it, Mastodon will broadcast the change to instances known to your profile.
Instances will now also refresh remote profiles every 7 days (if they receive content from you), to ensure the flag propagates correctly.
Show threadladyterukiAug 24, 2023
@renchap : oh that's good to know :) Thanks for taking the time to explain.
Show threadladyterukiAug 24, 2023
@renchap : actually while I'm here, may I ask another question ? If I want to reply to a toot posted by a mutual, that I saw a couple days ago (and assuming this person has opted in), can I search among toots specifically posted by that mutual ? Or will I have to scroll through aaaaaall the existing toots with the word or phrase that I remember ?
Show threadRenaud ChaputAug 24, 2023
@ladyteruki You should be able to use the `from:` query attribute to only search content from this user
Show threadladyterukiAug 24, 2023
@renchap : the way to say this seems to imply that's already an existing query ?
Show threadRenaud ChaputAug 24, 2023
@ladyteruki I am not sure, I did not work on search. But it should work with the new search :)
Show threadladyterukiAug 24, 2023
@renchap : awesome ! Thanks so much for the help !
Show threadBerkubernetusAug 24, 2023

@renchap so this is ES-based, rather than using the already-required Postgres instance, correct?

So "lightweight FTS" without adding ES is still an unimplemented feature.

Show threadRenaud ChaputAug 24, 2023
@fuzzychef Correct, this rely on ElasticSearch
Show threadBerkubernetusAug 24, 2023
@renchap someday I'll get enough round tuits to do the non-ES version.
Show threadBart LouwersAug 24, 2023

@renchap @fuzzychef Have you looked at OpenSearch as a FOSS ElasticSearch alternative? https://opensearch.org/

Edit: Looks like the answer is yes, it is compatible. ❤️ https://github.com/mastodon/mastodon/pull/22422

Home

OpenSearch is a community-driven, Apache 2.0-licensed open source search and analytics suite that makes it easy to ingest, search, visualize, and analyze data.

OpenSearch
Show threadRenaud ChaputAug 24, 2023
@bart @fuzzychef The current search implementation relies on the Chewy Ruby gem, which uses Elastic's Ruby client under the hood. So we "officially" only support ES, but I know some instances are using OpenSearch without any issues.
Show threadBerkubernetusAug 24, 2023

@renchap @bart Oh, right, Elastic has licensing issues too.

I was just thinking of overhead. Postgres has FTS capabilities, so being able to use that would make FTS much more accessible to smaller instances, since they wouldn't need to run a third DB service. Opensearch doesn't really help with that.

Show threaddon Elías (como los buses) 🥨Aug 25, 2023

@fuzzychef the open source project Manticore is a fork of the Sphinx search engine, they are working on being a drop-in replacement for Elastic Search, it already is for some use cases.

It is written is C++ it is faster and uses less RAM than than ES which is written in Java.

It would be interesting to see if the gem used to connect to ES supports Manticore, it would be a game changer for lots of instance owners that cannot afford so much RAM

https://manticoresearch.com/blog/manticore-alternative-to-elasticsearch/

@renchap @bart

Manticore: a faster alternative to Elasticsearch in C++ with a 21-year history

Five years ago Manticore began as a fork of an open source version of the once popular search engine Sphinx Search.

Show threadEssem Aug 24, 2023
@renchap Question, is there a modifier that only returns posts that you've interacted with, like with the current behavior? I know a few people that like that functionality and it would be somewhat disappointing if that was completely lost.
Show threadRenaud ChaputAug 24, 2023
@esm I am not sure, I remember it being discussed but I am not sure if the old behaviour has been kept somehow.
An issue with this feature is that we need to store who interacted with which post in ES, and this might create huge amount of data.
I added the question to my list of things to clarify!
Show threadEssem Aug 24, 2023
@renchap hmmm, that does seem like an issue, hopefully something that's addressable though (if ES doesn't instantly break, of course). thanks for the response!
Show threadRenaud ChaputAug 24, 2023
@esm This is only the first step of the implementation and I know people will definitely continue to work on improving it, so I can only assume it will be implemented at some point.
IIRC one of the issue is that for very popular posts, if you store user ids of every interaction in the document, you can get into trouble and reach the max document size
Show threadEssem Aug 24, 2023

@renchap that's... worrying

maybe it's time to look again at ES alternatives? I've been wanting to ask for a while what exactly the blocker is that ES has that the competition doesn't, considering that some other software I've worked with that has similar scope has moved to stuff like meilisearch without issues (apologies if this is a bit off topic)