countersignal:

Right, this is one of the most interesting things to come out of the LW-sphere that no one has picked up on.

Both Handle and Vladimir made the point that… well, I’ll quote Vladimir, because his way of putting it is short.

In a world where people make decisions according to this principle [that “should stop offending as soon as ey realizes that the amount of pain eir actions cause is greater than the amount of annoyance it would take to avoid the offending action, even if ey can’t understand why it would cause any pain at all” (Yvain)], one has the incentive to self-modify into a utility monster who feels enormous suffering at any actions of other people one dislikes for whatever reason.

Utilitarianism has a hard enough problem dealing with pre-existing utility monsters — but now it turns out that it can create them. The existence of an agent who is known to apply this form of utilitarianism incentivizes everyone who knows it to self-modify into feeling as much pain as possible at the slightest divergence from their desires in order to manipulate him.

If there’s only one such agent, that’s not much of an incentive. But, to take this beyond utilitarianism: if it’s generally accepted that it’s a noble goal to “comfort the afflicted and afflict the comfortable”, to “punch up”, etc., then there’s a strong incentive to become as afflicted or as low as possible, in order to gain power, or at least legitimacy.

That Brianna Wu article about how many death threats she got — that’s a claim to legitimacy based on affliction and lowness, and the obvious response is “well, shit, I wish I got that many death threats so I could have some moral legitimacy”.

The problem there isn’t just that it’s a bad idea to incentivize that sort of thing. It’s also that the people who value power the most will be the most morally legitimate or will be able to most easily manipulate utilitarian agents: if you don’t think gaining that level of power/legitimacy is worth the pain required, you won’t try. So it rewards the most power-hungry and best at unconscious self-modification to fit the demands / follow the rules of the power structure they’re manipulating.