@koherecoWatchdog this is a trade-off that I'm very happy to discuss.

I've thought of dropping DDG (and maybe Brave) results entirely. I'm a big fan of Mojeek, but there's a trade-off between privacy purism and relevance of the results that I'm still trying to strike here.

As an example: a search of my name on Mojeek won't return any links to my content on LinkedIn, Medium, or CF-powered websites like Hackernoon, dev.to, IoT4All, BetterProgramming. Nor any links to my books (because they are sold on the likes of Amazon, eBay, ebooks.com or springer.com), nor any links to my music (because, outside of Bandcamp, it's on the likes of Spotify, Tidal etc.), nor any links to my past talks (because they are mostly on YouTube).

Yes, it reports the links to my self-hosted websites and my apps on F-Droid, but those are only a small fraction on the content about me on the web. And the same considerations that apply to searching my name apply to any other search that a user may want to perform. Even the most privacy-aware user wouldn't want to use a search engine that shows to them only a fraction of the web. It should probably be their choice if they want to click on a result hosted on Amazon or Medium. The search engine can ensure that their privacy is protected and that it doesn't collect any private data about users (or that could associate users to queries), but IMHO it shouldn't omit results that may be relevant for what the user actually needs to do.

If users can't find what they're looking for, they'll just fall back on another search engine, which defeats the whole purpose of running an alternative engine. And, if users fall back on another search engine too often, they'll eventually just stop using yours - a point that @thelinuxEXP made quite well in a recent video.

@blacklight
The primary job of a good search tool is to filter out (or down rank) the results you don’t want.

Searx instances that show Bing or Google results are a dime a dozen (regardless if they src from a syndicate). There’s nothing special about your instance or any other searx instance if it only proxies the giants. What makes a search svc stand out is
1→ unique sources (yacy, mojeek, etc), or
2→ unique filtering (anti-CF or anti-bloat).
@thelinuxEXP

@koherecoWatchdog @thelinuxEXP in Searx/SearxNG is quite easy, even on the client, to customize which engines you want to query. So if a user is bothered by results returned by Bing or DDG, they can simply turn off those engines and turn on only Mojeek/Gigablast instead. I would also love it if CF filtering was available as a feature that the client can toggle as it likes.

But, as admins, we shouldn't be too opinionated and take those decisions for the user. My priorities for a search engine are:

1. It should not track you, nor collect any data about you, nor show you unsolicited sponsored content.

2. It should be functional for its purpose - which is to return what the user is looking for in most of the cases.

A search engine like the one that you describe probably won't return results from more than half of the Web, and it's hard to tick the "be functional" box if you filter out so many results.

As a developer, a search engine that downranks results from StackOverflow, or doesn't return technical articles from Medium, dev.to or Hackernoon, would be almost useless.

A search engine that doesn't return (or downranks) results from the Guardian, the Economist or the CNN (or even Fox) would provide a very incomplete view of the world when you search for news.

A search engine that doesn't return results from LinkedIn or Glassdoor won't be of much help to someone is looking to hire or be hired.

A search engine that doesn't return results from any major sources for online shopping or reviews won't be very helpful if I want to buy a new product.

My point is that even the most privacy-enthusiast user sometimes needs to access websites that aren't 100% privacy respecting, because some content that they need may be only available on those websites.

And if my search engine doesn't return those results, then people will just go to another search engine that may have those results, but it's likely to be more privacy-invasive than mine. And that's exactly what I want to avoid: the purest search engine is literally of no use if people can't get relevant results out of it for their day-to-day activities.

@koherecoWatchdog @thelinuxEXP "privacy seekers" is an umbrella term for people that fall on a wide spectrum. On one end, you have those who simply click on "Reject cookies", or use the browser's incognito mode, and they're fine with it. On the other end, you have people like you who uncompromisingly shape their whole online experience around the idea of absolute privacy. And you have a lot of shades of gray in between.

When I take decisions on how to shape the public services that I host, I try to aim at the middle point in this spectrum - the guy who wants no ads, no trackers and no bloat, and as little JS as possible, but who doesn't mind reading news articles on a major outlet, or following artists on Spotify, or discovering companies and co-workers on LinkedIn, or searching for programming questions on StackOverflow, or reading blogs on Medium. These users may decide to surf these websites like a guy who wears four condoms, just in case (with a DNS sinkhole, ad/tracker blocker, Tor, NoScript, some alternative client for those services, or all of these solutions), but they may still be interested in content that is only available on these platforms, and leaving out results from these platforms will lead to a bad experience, functionally speaking. The Web is powerful only when it doesn't get too opinionated about what results should reach the end user.

I try to also give freedom of configuration to those in the "privacy seekers" spectrum that are more uncompromising. For example, users can disable search engines that they don't want in their results, and I would also be happy to work on a PR for Searx/SearxNG to implement a user option to disable CF results if there's enough interest. But these settings shouldn't be the default, nor should the service be too opinionated and impose them to the user. If I do so, I may gain the trust of the more uncompromising users, but I will lose all the others in the middle of the spectrum. And those in the middle of the spectrum are likely to just go back to whatever crappy search engine they were using before, so they won't be better off.

If those who are more uncompromising decide that they won't get onboard with a solution just because it doesn't follow exactly their idea of how the Internet should work, then we'll keep getting fragmentation, forks and endless discussions, instead of creating solutions that appeal to the highest possible number of users and can cause a real dent in Big Tech's numbers.

When designing for privacy, I believe in striking reasonable trade-offs between providing a service that appeals to enough users (because function-wise it is similar to what they were using before, so migrating doesn't come with huge costs) and privacy purism. If you're too purist and pretend that any website that is somehow touched by Big Tech doesn't exist, then you lose users. In many cases, those users will just keep using Google, Bing, DDG or whatever, so they won't be any better off. If we push people away with our purism, we may be able to build our little happy bubble where Big Tech is not allowed in any form and shape, but we won't be able to build a convincing case for others to join us. And that really doesn't align with my mission - which is to raise the privacy level for as many people as possible, not to provide a privacy purist solution only for a niche.

@koherecoWatchdog @thelinuxEXP

1. Onion results: they would be there if they were implemented in the source code. Unfortunately, the only engine in SearX/SearXNG that seems to be configured with an onion_url is Ahmia. There is a PR for adding onion URLs to DDG: https://github.com/searxng/searxng/pull/1506 and a related open issue: https://github.com/searxng/searxng/issues/1505.

2. Onion URL for the engine: it would require me to set up the SOCKS proxies and all on the machine, as well as properly secure them. This is something that I'm planning to do, but you should also keep in mind that I'm self-hosting all of this stuff either on my Linode instance or my server at home (where I also run my Mastodon instance and Gitea server btw), and there's a physical limit to how much stuff I can self-host. Contributions for the costs of hosting and maintenance are welcome, if people expect me to run more services.

3. Clearnet URLs: they have been configured now.

4. Cloudflare flagging: again, it's not a feature that has been implemented in both SearX/SearXNG. Since spotting CF results isn't that difficult, I guess that PRs are welcome.

5. Rankings: AFAIK there's no way of tweaking with the rankings via configuration. Ranking are calculated on the basis of the engines' reliability and their own rankings. You can only completely exclude some domains from the results.

6. Red-flagging/downranking archive-hostile websites is another feature that isn't available in the upstream code.

7. The issue tracker links to Github because that's where both SearX and SearXNG are hosted, and that's where developers usually read what users reports. The alternatives would be: 1. either forking SearXNG on my Gitea instance and provide that link instead (something I'm very hesitant at doing, because that would make me the proxy maintainer for everything related to SearXNG), or 2. convince the maintainers to move the project somewhere else.

8. Naming issues ("engines" vs. "syndacates"): again, these should be addressed via PRs and discussions. Just because I'm running an open-source service it doesn't mean that I'm supposed to address/implement alone everything that should belong to the existing lifecycle of that open-source project.