Is content moderation a dead end? — Benedict Evans

Can we get content moderation to work, or is it as much of a dead end as virus scanning? Do we need to change the whole model of social instead?

Benedict Evans

On content moderation, the metric that seems to be getting picked up is #Prevalence (Facebook) or #ViewRate (YouTube/Google), which looks not just at the number of items posted, but the times each are viewed. This is beginning to approach a useful metric, but still poses several problems.

  • It's easy to focus on simple numbers or characteristics. These almost always provide an oversimplified and incomplete view. Noting how many violative / disinformational pieces of content are posted without accounting for their presentation within members' streams or search results is incomplete.

  • Computing items times impressions ... is much better. It also provides a basis for determining the moderation load required vs. degree of access granted, given distributions of prevalence which tend strongly to follow a power law distribution. (More below.)

#facebook #google #youtube #moderation #contentModeration

2/

  • Adding in a risk component, a calculation of a potential for harm for a given piece, seems the next logical stage in the process. This prioritises content for moderation.

  • From this, and making a broad incursion into potentially Orwellian territory, is identifying risks by individuals or groups (detected clusters or networks) which might then affect moderation and distribution (prevalence/view rate) scores or targets.

  • Given that social media are, well, social, with groups tending to have a high level of coherence, behaviour towards standards might be determined and behaivours including failure-to-flag or amplifying offending content being further considered. Effectively a trust metric for posting and amplification.

#contentModeration #moderation #youtube #google #ViewRate #Prevalence

3/

Among the interesting suggestions I"ve seen recently (and have encountered / thought of myself previously before) is this "feudal vouching system":

highly engaged community members, vouch for others, who again vouch for others. If people in this "Dharma-tree" accumulate problematic behaviour points, the structure at first, bubbles better behaved members to the top. If the bad behaviour continues, by single members or sub-groups, the higher echelon of members will cut that branch loose, or loose their own social standing.

https://news.ycombinator.com/item?id=26797017

Rather than use high engagement as a basis for vouching, arbitrarily selected communities, perhaps of about 50-150 active participants (posting or moderating) might be better. Think Mastodon, but in order to be federated, good posts must be "voted off the island", in the good sense.

I've been applying a somewhat similar notion to collecting and managing reading material and suggestions, what I call #BOTI, or Best of the Interval. It's a round-robin style system, where I compile a list of references over a period of time (monthly to annually seems to be most appropriate for me, though hour / day / week / month / year / decade / ... could be applied). At the end of an interval, some limited number of items is carried forward.

This is one way of addressing the "firehose of content" nature of information, recognising that in any given time period, you only have so much personal bandwidth to dedicate.

With the federated model, content federating is itself subject to assessment, and is effectively re-vetted. Note that different communities might favour different content: cooking, kittens, Kabbalah, canoodling, cypherpunk, core meltdowns, classic cars, concerthalls. Vetted / re-vetted streams themselves might be of interest.

Both community- and time-based elements of this could get interesting.

4/

Years ago, i suggested a algorithmic approach to moderation to a Open Source pro... | Hacker News

@dredmorbius This is something like what I had been thinking of for Coagitate.

@woozle Miranda's Knitting & Tea Circles were my attempt.

Community formation is a major hurdle.

A major problem of growth-related risks is that low-scale risks / problems and high-scale risks / problems are all but entirely distinct sets.

Initiatives which launch with eyes on large-scale problems tend to fail before attaining that scale. (In large part because virtually all initiatives fail, but also because they're not focused on the initial problem of growing first.)

@dredmorbius There never was the software-infrastructure to support the kind of reputation-management we're talking about, though.

@woozle I'm not sure the problem is, strictly speaking, a technical one.

Or to be more specific, I'm pretty certain it is NOT a technical / software problem.