2020 BRN Discussion, page-28468

"

Goodhart's Law states that "when a measure becomes a target, it ceases to be a good measure." The reason is that when something becomes measured, people try to influence it, so the data that were going into the measure are no longer as pure as they were before. I've started to wonder if this will happen with AI data sets. We see this behavior a lot in humans. When we know something is measured, or when we know we are being watched, we act differently. It's why surveys aren't always accurate. It's why review systems are gamed. But it makes me wonder - will we adjust our behavior when we know our decisions are data inputs into AI systems? This gets particularly complicated when AI systems are doing more complicated decision making. It could particularly be bad if humans aren't monitoring the data. For example, imagine an AI system that decides who gets a promotion at work. It monitors communication skills through emails and meeting recordings. It looks at work assigned in project management tools. All independent data, until someone finds out it is going into an AI tool for promotion analysis. Would that change how you might portray yourself, or those you like, to the system? After all, we do this to some extent when it is humans monitoring the systems. We call it corporate politics. My point here is that, one of the reasons we like AI for some tasks is that it is perceived to be "objective." But they won't be objective if people start shading the data in the directions they want the algorithms to go. Humans game systems. I expect we will game AI as well. Thanks for reading. @RobMay

"