Did someone complain? Or why stop?

No idea honestly. If anyone knows, let us know! I dont think its necessarily a bad thing, If their crawler was being too aggressive, then it can accidentally DDOS smaller servers. Im hoping that is what they are doing and respecting the robot.txt that some sites have.

I think it's just one HTTP request to the nodeinfo API endpoint once a day or so. Can't really be an issue regarding load on the instances.

It’s not about the impact it’s about consent.

True. Question here is, if you run a federated service... Is that enough to assume you consent to federation?

Why invent implied consent when complicit consent has been the standard in robots.txt for ages now?

I guess because it's in the specification? Or absent from it? But I'm not sure. Reading the ActivityPub specification is complicated, because you also need to read ActivityStreams and lots of other referen es. And I frequently miss stuff that is somehow in there.

But generally we aren't Reddit where someone just says, no we prohibit third party use and everyone needs to use our app by our standards. The whole point of the Fediverse and ActivityPub is to interconnect. And to connect people across platforms. And it doen't even make lots of assumptions. The developers aren't forced to implement a Facebook clone. Or do something like Mastodon or GoToSocial does. They're relatively free to come up with new ideas and adopt things to their liking and use-cases. That's what makes us great and diverse.

I -personally- see a public API endpoint as an invitation to use it. And that's kind of opposed to the consent thing.

But with that said... We need some consensus in some areas. There are use cases where things arent obvious from the start. I'm just sad that everyone is ao agitated and seems to just escalate. I'm not sure if they tried talking to each other nicely. I suppose it's not a big deal to just implement the robots.txt and everyone can be happy. Without it needing some drama to get there.

Robots.txt started I’m 1994.

It’s been a consensus for decades.

Why throw it out and replace it with imied consent to scrape?

That’s why I said legally there’s nothing they can do. If people want to scrape it they can and will.

This is strictly about consent. Just because you can doesn’t mean you should yes?

It's been a consensus for decades

Let's see about that.

Wikipedia lists http://www.robotstxt.org as the official homepage of robots.txt and the "Robots Exclusion Protocol". In the FAQ at http://www.robotstxt.org/faq.html the first entry is "What is a WWW robot?" http://www.robotstxt.org/faq/what.html. It says:

A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced.

That's not FediDB. That's not even nodeinfo.