"}Right, this is one of the most interesting things to come out of the LW-sphere that no one has picked up on.
Both Handle and Vladimir made the point that\u2026 well, I\u2019ll quote Vladimir, because his way of putting it is short.
In a world where people make decisions according to this principle [that \u201cshould stop offending as soon as ey realizes that the amount of pain eir actions cause is greater than the amount of annoyance it would take to avoid the offending action, even if ey can\u2019t understand why it would \ncause any pain at all\u201d (Yvain)], one has the incentive to self-modify into a utility monster who feels \nenormous suffering at any actions of other people one dislikes for \nwhatever reason.
Utilitarianism has a hard enough problem dealing with pre-existing utility monsters \u2014 but now it turns out that it can create them. The existence of an agent who is known to apply this form of utilitarianism incentivizes everyone who knows it to self-modify into feeling as much pain as possible at the slightest divergence from their desires in order to manipulate him.
If there\u2019s only one such agent, that\u2019s not much of an incentive. But, to take this beyond utilitarianism: if it\u2019s generally accepted that it\u2019s a noble goal to \u201ccomfort the afflicted and afflict the comfortable\u201d, to \u201cpunch up\u201d, etc., then there\u2019s a strong incentive to become as afflicted or as low as possible, in order to gain power, or at least legitimacy.
That Brianna Wu article about how many death threats she got \u2014 that\u2019s a claim to legitimacy based on affliction and lowness, and the obvious response is \u201cwell, shit, I wish I got that many death threats so I could have some moral legitimacy\u201d.
The problem there isn\u2019t just that it\u2019s a bad idea to incentivize that sort of thing. It\u2019s also that the people who value power the most will be the most morally legitimate or will be able to most easily manipulate utilitarian agents: if you don\u2019t think gaining that level of power/legitimacy is worth the pain required, you won\u2019t try. So it rewards the most power-hungry and best at unconscious self-modification to fit the demands / follow the rules of the power structure they\u2019re manipulating.