Company X is a global Fortune 500 retailer with thousands of employees developing technology solutions and AI-powered apps and products.

When Company X trains employees, a core goal is to help them understand the culture and values of the company. How an employee anticipates a customer need, delights the customer, and creates a human connection is key to the brand's ongoing success. The brand is centered around customer connection in a physical store. Technology has to amplify that human connection, not get in the way of it.

Artificial intelligence offers brands like Company X a powerful personalization tool. Personalization is one of the four pillars of Company X’s digital strategy. Data is used to create the most personal view of each individual, which can then be used to provide new experiences.

When Sonder Studio was asked to speak to Company X’s technology employees about AI, we wanted to leave them with one key idea; the paradox of personalization. The paradox of personalization says that the best way to personalize a person’s future is to make that future more predictable, which, over time, reduces a person’s self-determination—their ability to control their own life. This has the effect of turning personalization into a kind of anti-personalization by limiting their choices to those predictable by the AI based on historical data of the behavior of people like them.

Anyone who watches Netflix, listens to Spotify, spends time on Instagram, or buys stuff on Amazon has probably experienced this paradox. On one level, you want to find something new and have a different experience, but on another level, you are comfortable with what you already know. Some platforms are more likely to send you to an adjacent and unexpected place (TikTok is famous for this), but they all operate on the same basic principle: the owners of the platform make more money if they can predict what you want.‍

To increase the efficiency of personalization, designers need to put individual consumers in a box that will fit the prediction of who they will be tomorrow. When this is done so accurately that it becomes over-personalized, it can feel creepy. This is more common than you might think. According to one survey, 75% of consumers find most forms of personalization somewhat creepy, and 22% said they’d leave for another brand because of creepy experiences. But personalization is here to stay: according to another report, 52% of customers say that their satisfaction improves as digital experiences with brands become more personalized.

How did the AI know that I would want that? Or perhaps the question is, would I have wanted it if the AI hadn’t said I would? Am I making choices for myself, or is the AI making choices for me?

AI changes how people work together, whether part of a customer/employee interaction or an employee/employee interaction. Sonder Studio worked with Company X to help their human employees design “machine employees” that act in accordance with brand values of supporting human connection and agency. To do this requires resolving the paradox of personalization by ensuring that people retain human agency and choice as they interact with AI.

‍You can find out more about how humans and machines can work together better by signing up for our newsletter. Sign up for our newsletter here. 

‍You can also learn more about Company X by reading how its leaders use data and AI for good, better solve complex problems, built a data-driven decision making culture, and built more responsible AI.