Exploring future scenarios to combat bias in AI-driven experiences

A member of our team setting up our speculative design workshop, FutureShift, in a large classroom.

How might we empower people to discuss, critique, and reflect on how personalization algorithms might find their way into unexpected areas of their lives? FutureShift is a workshop for making alternative future scenarios and values more tangible during the design process.

For my Master’s capstone project in Human Centered Design & Engineering, I created a speculative design workshop to explore the implications of machine learning algorithms in the near future. Participants explored Black Mirror-esque scenarios and created boundary-pushing systems and prototypes that might exist in those scenarios.

Speculative design gives us an opportunity to debate and critically reflect on what we want and don’t want to see in our future as a society. To understand how we want to shape and be shaped by algorithms, we need to create space to challenge presumed virtues and dominant values inscribed in them. (Since this workshop, other companies have used a similar process.)

Skip to our Process Book where we documented our workshop and learnings!

Collaborators

Project Team:

  • Paul Roberts: Design Strategist & Project Manager
  • Ariel Duncan: UX Researcher

My role: Visual Design Lead. Responsible for creative direction, and writing & editing for the workshop and process book. Led literature review and partnered with Research Lead on expert interviews. Collaborated on all other aspects of this project.

Timeline: 10 weeks

Concept

Is frictionless design the best design we can create?

At the outset, we wanted to question the ideal that frictionless, seamless design is good. From fake news on Facebook to predictive policing software, we realized we were blind to how algorithms dictate many aspects of our lives. As algorithms increasingly make decisions for us, they gain agency without us noticing it.

We wanted to shift the agency back towards people, and let them decide how much power algorithms should have over their day-to-day decisions.

To do this, we shifted our thinking from creating incremental solutions to using design to provoke, critique, and reflect on our own unquestioned biases.

Project Goals

  1. Enable participants to explore the social and ethical implications of technological predictions through making a “speculative object” that expressed their (one of many) desired relationships with algorithms.
  2. Display these objects at a “yard sale of the future” where the fictitious products would be sold alongside items one might find at a real yard sale. The yard sale provided a real context to start conversations with visitors around the values embedded in fictitious products.

Final Design

We created a workshop called FutureShift.

We conducted our workshop with 4 participants who came from the University of Washington. Diversity of participants was critical, so the university environment allowed us to recruit people from different majors, from Junior to Ph.D level. At the end of the workshop, we surveyed participants on their experience.

Workshop Materials:

A member of our team facilitating our speculative design workshop, FutureShift.
Participants explored future implications of technological predictions through structured activities. They picked a future “signal” and explored it by considering its social, technological, economic, environmental, and political dimensions. Then they created a narrative using those considerations and their desired relationship with that signal. Finally, they created a “object from the future” using materials we pre-selected for the making activity.
Parts of the FutureShift Workshop which included a warm-up activity, facilitation prompt cards, and prototyping near future objects.
We took an iterative approach to developing the workshop. We created two warm-up activities to make an abstract topic more concrete and relatable. We also developed cards with questions relating to the STEEP (social, technological, economic, environmental, political) dimensions to help people imagine more diverse scenarios. To aid people in expressing their product’s narrative, we participated in the making process.
An example of a speculative object that a participant made during the workshop.
An example of a “speculative” object a participant created. She wanted to help teenagers disguise their faces when facial recognition technology was everywhere.
Examples of objects that participants prototyped during the speculative design workshop.
Other examples of speculative objects participants created to express their point of view about a future prediction.
A member of our team selling items at the FutureShift Yard Sale.
A yard sale provided a real context in which people from our community can evaluate participants’ objects from the future.

Outcome

Workshop

While we didn’t expect to cultivate agency in 3 hours, we saw hints of agency emerge through both form and narrative, and during our discussions and post-workshop evaluation.

A participant holding his prototype of an idea based on "social reputation coach" to undo the effects of race and class biases.
A participant holding her prototype of an idea based on tracking nutritional needs of children.

Yard Sale

We had a total of 19 visitors and engaged in conversations about workshop participants’ speculative objects with 10 visitors. We filmed some of the visitors’ responses.

A neighbor who shared her perspectives on personalization algorithms that might inhabit digital products of the near future.

Design Process

Understanding the issue at scale

To conserve time, we conducted 6 interviews with experts in the field and read over 50 pieces of academic literature and industry reports.

High-level research goals:

  • Unpack the interplay between algorithms and their impact on culture & society, and uncover methods for revealing how algorithms work (i.e. reverse-engineering algorithm’s outputs).
  • Explore strategies from the field to inform how we could inspire awareness, discussion, and reflection around algorithms and people’s desired relationship with them.
A screenshot of a Skype call with Pamela Pavliscak to discuss her work with emotional design, artificial intelligence, and practical ethnography.

Findings from discovery research

From talking to experts, we learned that we can repurpose our design process into conceptual tools to challenge our assumptions, preconceptions, and givens about the roles algorithms play in people’s everyday lives.

Slide with a quote from Audrey Desjardins, an Assistant Professor of Interaction Design at the University of Washington.

1) People have conflicting relationships with algorithms

People were conflicted about how algorithms “see” them because they didn’t trust the intentions behind the companies (i.e. Facebook, Google). But at the same time, they were frustrated when these systems didn’t know enough about them to personalize the experience.

2) People make up stories about how algorithmic systems work

Then, they behave according to what they believe–sometimes trying to train the system to better align with their preferences. People see the data and content that are served up but not how they’re being pulled together or weighted.

3) Opaque decision-making processes contribute to people’s uneasy relationship

  • It’s hard to challenge a decision made by algorithms when the process is “blackboxed.” Usually we only see the outputs, and the lack of transparency makes it difficult to trace a particular decision and much less hold the system accountable.
  • Algorithms are perceived to be objective when they’re not. They encode human biases in the form of rules, categories, and criteria. It’s easy to accept a result as a given, and this sense of certainty discourages people to challenge an algorithmic output.
  • We risk creating algorithms that exclude and discriminate if we don’t actively challenge our narrow view of the world. Right now, the capabilities to change how algorithms work rest with developers, designers, and researchers. By including more diverse segments of our population, we can avoid blindspots and anticipate emerging desires around the kind of relationship people want with algorithms.

Bottom line

If machine learning algorithms already walk a fine line between creepy and annoying, there was an opportunity to work backwards from people’s preferred relationship with them.

Reframing the Challenge

Without challenging personal assumptions that go into designing algorithms, we recreate current inequalities in algorithmic systems. But that’s difficult to change if we don’t shift the agency back to “non-experts” – in a process to highlight people, perspectives, and values that might be forgotten during product development.

New challenge:

Empowering non-experts to voice their concerns and tensions about how algorithms are currently designed.

The opportunity:

How might we empower people to discuss, critique, and reflect on how algorithms might find their way into unexpected areas of their lives?

Two immediate audiences:

  1. Non-experts: These are people who don’t work at McKinsey (for example), contribute to industry trend reports, or have regular access to design/development teams on algorithmic products. We want to hear from people who aren’t usually included in developing algorithms or thinking about long-term future.
  2. Design, development, and research teams: These people have direct impact on creating algorithmic systems. We want to empower them to challenge their own perspectives about “what’s desirable.”

Why it matters:

By working backwards from preferred visions, we can enable tangible actions today while including alternative values and considering the ethical implications of rules, categories, and criteria we embed into algorithms.

Arriving at Our Concept

Broadening our view of “the future”

To challenge our conceptions of a “desirable future,” we needed to push the extremes into the “ridiculous.” We took inspiration from Dunne & Raby’s ideas on speculative and critical design and Montgomery & Woebken’s hands-on, participatory approach to challenging conventional values. To see how we might apply these ideas in practice, we borrowed from Dr. Dan Lockton’s “designing for agency” and A.B. Kocaballi’s agency-sensitive design qualities.

Questions to inform our design strategy

  1. How might we empower people to challenge the status quo of how the future is currently being imagined and to add their own personal critiques and desires into the consideration?
  2. How might we make algorithms and their impact more visible and tangible?
  3. How might we include a more diverse audience when we discuss future implications of algorithms?

Deciding on a participatory workshop framework

After exploring digital “interventions” and social experiments, we decided to modify a workshop framework from the Extrapolation Factory because it was an approachable way to engage everyday people in thinking about the future. The workshop walked people through making speculative, future objects and then placed these objects in everyday contexts to engage the public in spontaneous discussions about possible futures.

The set up for the speculative design workshop that the Extrapolation Factory created.
The Extrapolation Factory sold participant-made objects in a real convenience store in New York City after the speculative design workshop.

Iterating the Workshop Concept

We iterated on the Extrapolation Factory’s workshop concept to support our design strategy.

1) From present solutions to future implications

We modified the Future Cone and the Futures Wheel to stretch participants’ imaginations beyond present-day scenarios.

Brainstorming visual frameworks for the speculative design workshop.
Version 1 of the visual framework for the speculative design workshop that we set up in a classroom wall.

But we eventually took out the STEEP lenses (Social, Technological, Environmental, Economic, Political) because participants were getting stuck at categorizing the future signals instead of articulating their desired scenarios. We also simplified the Futures Wheel to keep the momentum of coming up with varied implications from participants’ chosen signals.

Visual framework of the Future Cone and Storymaking portions of the speculative design workshop we set up in a large classroom wall.

We curated a list of future signals from research firms like Gartner and Intel to provoke discussions and probe for people’s desired relationships with algorithms.

Prototyping the speculative design workshop where participants pick out a few near future scenarios that pique their interest.
Examples of near future scenarios.

To embed their own personal critiques and desires, participants created narratives to articulate their point of view about the future implications of their chosen signals. Even through we removed the STEEP lenses, we brainstormed provocative questions about each lens and created a deck of “STEEP” cards to help participants articulate their thoughts.

FutureShift card deck to help facilitate conversations about the near future scenarios.

2) From barely noticeable to visceral

Participants made tangible objects to playfully materialize their hopes and concerns about algorithms, and interact with the future on a visceral level. By using tangible materials, we were able to talk about other possible versions of future in a more concrete way.

A prototype session where a participant tests out the object-making portion of the speculative design workshop.

We iterated on the kinds of materials that would be conducive to expressing abstract ideas. The challenge was to overcome messages already encoded by professional industrial designers about what the finished products “should” be. In the end, we opted towards modular materials, breaking apart finished products if we needed to. We selected items that more loosely communicated about how they should be used.

A participant investigates materials for object-making during the speculative design workshop.

We also tested around a context to display participants’s objects. An appropriate context encourages people to consider near-future products side by side with the everyday and the “outdated” products. This sort of comparison brought out visceral and emotion connections to potential scenarios embodied in future products, and we used them to generate discussions with the public.

Selling ordinary items alongside participant-made objects at the yard-sale-of-the-future event in a residential neighborhood.
Brainstorming public engagement ideas on a whiteboard.

We tested our assumptions around how “familiar” a context would be to ensure that it grounded both the public and workshop participants. Using guerilla testing, we asked people to list as many objects as they could think of in a potential context and then interviewed them about the personal stories attached to those objects.

Researcher conducting concept testing to prioritize public engagement ideas.

We landed on yard sale of the future concept based on accessibility, familiarity, natural evaluative-ness, and feasibility.

3) From deductive thinking to inductive thinking

Future predictions are often dictated by research firms and academic think tanks. We opened up space for non-experts to voice their own versions of “what’s a desirable future?” To make it more relatable to everyday life, we tested two precursor activities: Meet My Algorithms and Algorithmic Self Drawing Exercise.

Several participants placing popular consumer digital products and services on a 2x2 matrix during a warm up exercise for the speculative design workshop.
Close up shot of a warm-up exercise during the speculative design workshop.

We leaned heavily on facilitation, working 1-on-1 with participants on articulating their hopes and concerns about a future prediction. Facilitation was a key element in building awareness and reflection through spontaneous conversations.

Facilitator guiding a workshop participant during the scenario selection step.

We also worked 1-on-1 with participants during the making activity to help them “think with” the materials. In the post-workshop survey, participants said that our facilitation made the workshop more enjoyable and helped them express the narrative they wrote.

Facilitator leading a workshop participant in the object making exercise to prototype personalized experiences based on a future scenario they chose.

Walkthrough of FutureShift Workshop

Our workshop, which we named FutureShift, consisted of several phases.

High-level overview of the speculative design workshop.

Let’s Use an Actual Example

Example of how we moved a participant through the design workshop, took their speculative object or prototype, and displayed it in the yard sale. The purpose of the yard sale is to engage the public in spontaneous conversations about the implications of future digital products that increasingly rely on sophisticated learning algorithms to provide personalized experiences based on users' data.

Experiencing the Workshop

Here are the steps through the FutureShift workshop.

1) Meet My Algorithms

As they entered, participants were invited to chart their relationship with common algorithms (e.g. Facebook, Amazon, Google, Netflix) on a wall. They mapped these algorithms on a 2×2 diagram using two scales: “trustworthy” to “creepy” and “gets me” to “doesn’t know me at all.”

2) Algorithmic Self Drawing Exercise

To reveal the invisible work of algorithmic systems, participants sketched and shared versions of themselves from the perspective of three popular algorithms: Facebook, Netflix, and Amazon.

3) Choosing a signal

Participants were invited to review 35 predictions about the impact of algorithms on culture and select one or two that they’d like to explore further. These “signals” challenge stereotypical ideas about what counts as “the future” by exposing participants to a variety of scenarios about how algorithms might influence unexpected areas of their life.

4) Sorting futures

Participants chose whether their signal about the future belongs in “probable”, “plausible”, or “possible” future. Using a pink thread, they mapped a route through the appropriate section of the Future Cone and into the first arrow of the next phase of the diagram. The Future Cone helped participants understand “futures” in the plural.

5) Storymaking

Participants fleshed out the “world” of their signal, creating multiple narratives that explored the implications on other aspects of life in that future scenario. A deck of provocative questions helped encourage participants to consider different perspectives. We created a two-level chevron as a visual cue to push them to explore secondary and tertiary consequences of their signal’s scenario.

6) Making speculative products

Participants were invited to choose materials that were of interest to them and physically render any aspects of the narratives they created. We placed the materials close to the Future Cone area to help them make connections between the materials’ qualities and the scenarios they were considering.

7) Putting the products up for sale

Participants were invited to imagine their object existing in the context of a yard sale of the future. They created narratives about how their object ended up for sale and the price they would ask for it. Doing this helped them think about how their object might be used, become obsolete, or be discarded.

8) Yard Sale of the Future

We hosted an actual yard sale where participants’ objects were displayed amongst “normal” items one would find at a typical yard sale. Participants’ objects were used to inspire spontaneous discussions about the possible futures they represented as visitors browsed items for sale. We were able to engage passersby in brief but emotionally-charged conversations about the role algorithms might play in the near future.

Reflection

The future is something that people expect to imagine with others.

Imagining the future is hard, precisely because we focus on the now, the tangible experience. But that highlights a larger issue in user-centered design. As designers, we focus around the narrow goals of the user and the business as it exists today and underestimate the long-term impact of things we design. Our work processes are set up this way: agile, lean methodologies, design sprints, etc. Without changing our work processes, we won’t be able to change our outcome. We need to shift away from our myopic tendencies and ask different questions of ourselves and our design parameters. When we haven’t been asking “what if?” and “is this a good idea?”, it’s harder to parse out which possibilities are preferable and which are terrifying, as Black Mirror invites us to consider.

We can accomplish this through critical reflection. But as facilitators, we need to think of ourselves as both individuals and researchers, rather than imagine that we can be outside of the processes we’ve designed. To effectively generate constructive debates, participants need to trust us–and trust each other–to bounce perspectives off of one another. We realized that creating the story (storymaking step) was critical in helping one another reflect and consider the implications of several possible futures. Writing a story about a desired future builds shared understanding of the values and preferences built into it and “backcast” to the present to address upcoming shifts.

Is a “disruptive innovation” a good one? We won’t know if we don’t talk about it.

More Projects