“Sai woke to the rousing first movement of Vivaldi’s violin concerto in C minor, ‘Il Sospetto.’ He lay still for a minute, letting the music wash over him like a gentle Pacific breeze. The room brightened as the blinds gradually opened to the sunlight. Tilly had woken him right at the end of a sleep cycle, just as he liked. He smiled. ‘Good morning, Tilly,’ he said. ‘Good morning, Sai!’ Tilly’s voice was cheerful and energetic. ‘You have forty-five minutes until your meeting with Mr. Tilapia. Shall I play the news for you while you get ready?’ ‘Yes, please.’ The wall display lit up with a carefully curated selection of news stories. As he brushed his teeth, Tilly’s smooth, pleasant voice read the headlines to him, highlighting the stories she knew would interest him the most”.
Ken Liu – The perfect match
The above exert of “The perfect match” by Ken Liu (Liu, 2016) might seem familiar. In my eyes, this story exert encapsulates a view of the future of intelligent systems that has come to be the standard for many companies and designers alike. In the story, Tilly – Sai’s personal assistant system – is created to support Sai in all activities in his everyday life (from cooking and reading the news to choosing conversation topics and possible partners for dates). Tilly is omnipresent, being both integrated into the home and present as a wearable entity. But why would we, as designers, create such systems and what are the consequences? In this post, I will try to take you through some of the points I found interesting about this topic.
Futures like the one described by Ken Liu have been suggested as probable scenarios have been suggested before. See also the book “Prototyping 2040” (Skalska & Kołodziej, 2022), where a scenario is discussed where everything is smart; the objects recognize and understand human emotions and psychology (the scenario is called “Kind capitalism”). Previously, this would not have been feasible. But the quick advancement of ChatGPT can even make the strongest believers that machines can never truly be intelligent doubt. For now and the close future, however, the systems will remain oblivious to the work they do. What the systems of the next few years will do is gather as much information as possible. The large amounts of data gathered are concerned with the factual occurrence of actions, rather than the reasoning behind it (Anderson, 2008).
But if systems cannot understand us (and up to a certain extent we don’t understand the systems either), why would designers spend any time creating them? I think this can be found in the nature of societal progress. In his co-emerging future of immortalia, Reon Brand (2019) suggests that anthropocentrism is an, if not the most, important underlying value of that future. It is precisely this anthropocentrism that makes intelligent products so appealing to designers; what if we can provide users with experiences and functionalities that have never been seen before? The systems relieve us of our difficult, back-breaking, or meaningless tasks. Systems that gather data about us allow added functionalities, providing us with more freedom.
But is this true? As these new products and systems (such as Tilly) gather more data they could also increasingly function as a digital panopticon, as suggested by Byung-Chul Han (2017). For the perceived freedom gained, users willingly and constantly expose themselves. The gains are easy to spot, but the costs in the form of the data that is given away is harder to see. In a way, users willingly fill their homes with spies and onlookers to all aspects of their lives. Companies use the data to produce more and more products and services to support users, at the same time forcing users to constantly improve and adapt with their products (increasing pressure on ourselves (Han, 2015)). So how free are we truly if pressure to constantly change increases (supported by that convenient new assistant)?
Designers play a crucial role in this, as they design the systems into the background. Especially in the stream of ubiquitous computing devices are placed in the background to an extent that they are almost indistinguishable from “dumb” products.
I belief that we, as designers, should keep this in mind as we are working on intelligent systems. Do we genuinely belief that the product or service that we put out in the world provides so much benefit that it justifies the increased pressure on the individuals that form the consumers? Is it always appropriate to make our designs blend in or should they stand out, showing the world the actual agenda that they pursue? Let Ken Liu’s work be the cautionary tale it is, where Sai must also be shown how little he sees of what is being stolen from him.
Jenny challenges Sai’s complacency, highlighting how Tilly and the corporation behind it, Centillion, control and manipulate users by collecting exhaustive data on them. She tells Sai, “Look at you. You’ve agreed to have cameras observe your every move, to have every thought, word, interaction recorded in some distant data center so that algorithms could be run over them, mining them for data that marketers pay for. Now you’ve got nothing left that’s private, nothing that’s yours and yours alone. Centillion owns all of you. You don’t even know who you are anymore”.
Ken Liu – The perfect match
References
Anderson, C. (2008, June 23). The End of Theory: The Data Deluge Makes the Scientific Method Obsolete . Wired. https://www.wired.com/2008/06/pb-theory/
Brand, R. (2019). Co-Emerging Futures A model for reflecting on streams of future change. https://www.researchgate.net/publication/333972702
Han, B.-C. (2015). The burnout society. Stanford University Press .
Han, B.-C. (2017). Psychopolitics : neoliberalism and new technologies of power. Verso.
Liu, K. (2016). The perfect match. In The paper menagerie and other stories. Saga Press.
Skalska, Z., & Kołodziej, R. (2022). Prototyping 2040: The Futures Thinking Book. Greenhat, BLue Media.
The pictures in this post have been created by Adobe Firefly on May 21st, 2025 using the following prompts:
Man lying in bed, awoken by a television, wall, morning light
Man , brushing teeth by a television, wall, morning light
man, sitting in a room, looking on a phone, window, looking at a panopticon
a spy robot, taking pictures of a person eating breakfast, taking over shoulder of robot
The style of the images is chosen as to fit with the aesthetic often used for AI systems.
