Mastodawn

#Lemmy gets a (new) search engine

It already had search (with some rough edges), but people are already making their own to fill specific needs ... chiefly it seems to replace the Google `site: reddit.com` search facility.

See ...

https://lemmy.world/post/963301

https://www.search-lemmy.com/

Interesting to see a platform culture completely embrace being open and public.

Announcing a new Search Engine for Lemmy - Lemmy.world

I shared bits and pieces of this before, but it’s officially up and running now: https://www.search-lemmy.com/ [https://www.search-lemmy.com/] This is an enhanced search engine for Lemmy. With a few primary goals: * You can choose a preferred instance. After choosing what your primary instance is, and performing a search ALL links will open in that instance. * This aims to be a replacement for using site:reddit.com in Google, but just for the fediverse. * You can filter the search results by: * Instance – This will filter the results to only show communities that belong to a particular instance. Just type something like instance:lemmy.wrold or instance:https://lemmy.world/. This is separate from your preferred instance, such that you can search for posts on lemmy.world while still opening them on lemmy.ml [http://lemmy.ml]. * Community – You can refine the search by a specific community. You use the same syntax that you’d use here community:[email protected]. * Author – Similar to the above you can also filter by a specific author such as: author:@[email protected]. * The entire thing is open-source. You can view the code and even host your own instance… See more details here: https://github.com/marsara9/lemmy-search [https://github.com/marsara9/lemmy-search]. NOTE: This only supports Lemmy instances for now. Other fediverse type instances may be in the future depending on how this works out. I’ve been working on this over just the last few weeks, so it hasn’t had a chance to crawl much of the fediverse yet. For now it only supports lemmy.world and lemmy.ml but other preferred-instances will come online as time goes by. If anyone finds any bugs, and I’m sure you will, or if anyone has any suggestions PLEASE raise an issue on GitHub for me to track. Lastly, if anyone wants to help contribute please feel free to reach out. NOTE TO SERVER ADMINS: You can prevent your site from being crawled by adding lemmy-search to your robots.txt for the user-agent.

Laurens Hof Jul 5, 2023

@maegul @[email protected] yeah this is so cool and fascinating. Completely different culture on lemmy with regards to search. And I keep being surprised by how separate the communities are. It barely registers on the masto feeds as news

maegul Jul 5, 2023

@laurenshof @fediversenews

Yea #threadiverse and masto are, IMO, not really connected. The UI impedances aren’t matched at all really and so not much makes it over the protocol.

I would be interesting to see how friendica, kbin and hubzilla people integrate with and use #lemmy compared to masto.

Very tempted to give my kbin or friendica accounts another spin. Except that I like hachyderm governance a lot.

Laurens Hof Jul 5, 2023

@maegul yeah agreed on the not actually really being connected. I think youre correct with the UI impedances, but I think its a culture as well, of people not actually being interested in interacting much. I see some comparisons with misskey in that sense, where I dont think that Western masto is really connected with japanese misskey.

Also a hachyderm-style governance kbin/lemmy server would go so hard, id love that so much

maegul Jul 5, 2023

> hachyderm-style governance

There was mention from the admins that they were considering it to some extent. I haven't checked their discord in a while, so I don't know where that got up to.

But yea it'd be nice to see. I'd also hope that it'd get some of the developers on hachyderm interested in helping out with the platform in some way, perhaps even lead to some new experimentation.

Vonck Jul 5, 2023

@maegul @laurenshof @fediversenews

The threads show up pretty well in calkey. that's a mastodon problem.

Looking for a general calkey instance with >1k users.

totally_a_real_lesbian_flag

@maegul @laurenshof @[email protected] I wanted to follow a lemmy community from my newly setup misskey but I didn't seem to get anything onto my timeline and misskey indicated federation problems so idk if it's a gui thing
maybe I did something wrong though

Laurens Hof Jul 7, 2023

@aliceif @maegul in general is federation between lemmy and microblogging kind of iffy, definitely doesnt work as well as it should. Out of curiosity, what happens if you search for an individual lemmy thread on misskey? can you find that, and comment on it?

totally_a_real_lesbian_flag

@laurenshof @maegul yeah i've managed to pull that off before by opening the link you get when copying from the pentagram

- guess i'll try following again (didn't get any replies in that conversation so idk if federation in that scenario works/ed)

Jupiter Rowland Jul 7, 2023

@aliceif @maegul @Laurens Hof Lemmy has federation issues currently. They're working on it.

Jeena Jul 5, 2023

@maegul @fediversenews i remember when people on Mastodon tried to do it all hell broke loose.

maegul Jul 5, 2023

@jeena @fediversenews

Yep. And here the average reaction seems to be “Awesome, how do we integrate this into my fav app?”

Jeena Jul 6, 2023

@maegul @fediversenews i have the feeling that the average age is older on mastodon compared with the Threadyverse, I don't have numbers but I think Mastodon is 30 to 45 and Threadyverse is 25 to 40

maegul Jul 6, 2023

@jeena @fediversenews
Interesting!

I’ve def seen a couple of polls backing you up in the mastodon side. Masto is clearly X-Gen/Xennial and surprisingly Boomer/Jones gen too.

Lemmy/kbin don’t have polls unfortunately. But a post to asklemmy would definitely get responses.

Jeena Jul 6, 2023

@maegul @fediversenews ah great ides I'll sk there

maegul Jul 6, 2023

@jeena @fediversenews

I tried a poll on lemmy. Not sure it went well but there's a bit of a picture of the demographic: https://lemmy.ml/post/1774488

By my count ... median 30s with probably more younger people (10-30) than here.

How old are we here? - Lemmy

Just got interested in what the age breakdown is on here. So … a kind of poll (even though lemmy doesn’t have polls). Let’s keep it as anonymous/general as possible … Use the categories below and upvotes on existing posts as much as possible? 0. 0-9 1. 10-19 2. 20-29 3. 30-39 4. 40-49 5. 50-59 6. 60+

maegul Jul 6, 2023

@jeena @fediversenews

But otherwise, it seems to be consistent with the generational profile on masto ... basically xennial centric.

Jeena Jul 7, 2023

@maegul @fediversenews I also created one ^^ I have the feeling that people in the Threadyverse https://jemmy.jeena.net/post/26055 but it's not easy to extract any numbers out of it 😅

Stefanie Schulte Jul 5, 2023

@jeena @maegul @fediversenews I think the distinction does make sense, because the Threadiverse relies much more heavily on the power of individual magazine ("subreddit") moderators to curb potential trolling and pile-ons than Mastodon does. These are different cultures.

Fell Jul 5, 2023

@maegul @fediversenews Honestly, I wish the rest of the #Fediverse was like that. What's the point of a PUBLIC network when it's not searchable?

Patrick | sieben-grafik.de Jul 5, 2023

@fell @maegul @fediversenews well one can kinda opt-in : https://tootfinder.ch/

Tootfinder

Jupiter Rowland Jul 7, 2023

@Fell @maegul #Friendica has has full-text search six years before Mastodon even existed. #Hubzilla, technically a fork of Friendica by Friendica's own creator, had full-text search four years before Mastodon existed.

Of course, both can only search what they know. They can't search for everything across the entire Fediverse. That's technologically nigh-impossible.

Netzgemeinde/Hubzilla

maegul Jul 7, 2023

@jupiter_rowland

Yes, sorry to pile on you @fell , but your comment highlights very well the problematic dominance mastodon has over the fediverse. Search, for better/worse, is something where it's mastodon that's the odd one out ... and yet, to the vast majority it seems like the norm for the whole fediverse.

It's significant because it means one small software team is basically dictating what the whole fediverse looks like.

Mike Macgirvin Jul 7, 2023

"That's technologically nigh-impossible."

No, it actually isn't. And I've been building it. But I'm retired from open source now and only have an hour or two every day that I can work on stuff like this. Distributed search will be slower than centralised search. That comes with the territory. But there are a number of ways of removing the cognitive barrier. One of these is to launch 'agents' that go out and find stuff for you - and then stick the results in a special stream you can visit and see more results the longer the agent is out there looking on your behalf.

It's not the "instant" search you see in something like Google, but Google is just agents which are already running on somebody else's computer searching and cataloguing distributed stuff.

Mark Britten Jul 8, 2023

@fell @maegul @fediversenews the argument, which I don't think I 100% agree with, is that full text search facilitates targeted abuse.

Jupiter Rowland Jul 8, 2023

Well, the expectation of many is to be able to search the _entire_ Fediverse as it is _right now_. Every last instance of everything. Not just the most popular Mastodon or Lemmy instances. Everything.

In other words, if someone spins up a brand-new (streams) instance for the very first time, all other Fediverse instances must index it in real-time from that very moment on. Right when the admin drops the first test post, all Mastodon and Lemmy instances must know. That's the expectation. And that's what's nigh-impossible.

Google doesn't know the entire Internet either. But most people think it does, and they expect it.