Designing backwards: fighting biases and avoiding unintended consequences in AI-driven experiences

A member of our team setting up our speculative design workshop, FutureShift, in a large classroom.

How might we empower people to discuss, critique, and reflect on ways in which AI programs find their way into unexpected areas of our lives? FutureShift is a prototyping workshop for exploring alternative future scenarios and values in emerging technologies.

For my Master’s capstone project in Human Centered Design & Engineering, I created a speculative design workshop to explore the implications of machine learning algorithms in the near future. Participants explored Black Mirror-esque scenarios and created boundary-pushing prototypes that might exist in those scenarios.

Skip to our Process Book where we documented our workshop and learnings.

Collaborators

Project Team:

  • Paul Roberts: Design Strategist & Project Manager
  • Ariel Duncan: UX Researcher

Role: Visual Design Lead

  • Responsible for creative direction, and writing & editing for the workshop and process book.
  • Led literature review and partnered with Research Lead on expert interviews.
  • Collaborated on all other aspects of this project.

Timeline: 10 weeks

Is frictionless, seamless design good for society?

From fake news on Facebook to predictive policing software, we realized we were blind to how machine learning algorithms were dictating many aspects of our lives. As algorithms increasingly make decisions for us, they gain agency without us noticing it.

We wanted to shift the agency back towards people, and let them decide how much power AI programs should have over their day-to-day decisions.

To understand how we want to shape and be shaped by AI programs, we need to create space to challenge presumed virtues and dominant values inscribed in them.

Speculative design gives us an opportunity to challenge our own biases (our “givens”) and critically reflect on what we want and don’t want to see in our future as a society.

We created a workshop called FutureShift

We led participants through an interactive, hands-on workshop to explore the social and ethical implications of technological predictions. They made “speculative objects” that expressed their (one of many) desired relationships with AI-enabled products and services.

We staged a “yard sale of the future” where we sold these fictitious products alongside real items. (Yes, people actually bought stuff and we made a tiny amount of money.) We started conversations with visitors about the fictitious products and tried to understand how they thought about the unexpected ways that technology might personalize and predict in the near future.

Workshop Materials:

A member of our team facilitating our speculative design workshop, FutureShift.
Participants explored future implications of technological predictions through structured activities. They picked a future “signal” and explored it by considering its social, technological, economic, environmental, and political dimensions. Then they created a narrative using those considerations and their desired relationship with that signal. Finally, they created a “object from the future” using materials we pre-selected for the making activity.
Parts of the FutureShift Workshop which included a warm-up activity, facilitation prompt cards, and prototyping near future objects.
We took an iterative approach to developing the workshop. We created two warm-up activities to make an abstract topic more concrete and relatable. We also developed cards with questions relating to the STEEP (social, technological, economic, environmental, political) dimensions to help people imagine more diverse scenarios. To aid people in expressing their product’s narrative, we participated in the making process.
An example of a speculative object that a participant made during the workshop.
An example of a “speculative” object a participant created. She wanted to help teenagers disguise their faces when facial recognition technology was everywhere.
Examples of objects that participants prototyped during the speculative design workshop.
Other examples of speculative objects participants created to express their point of view about a future prediction.
A member of our team selling items at the FutureShift Yard Sale.
A yard sale provided a real context in which people from our community can evaluate participants’ objects from the future.

Some reactions we gathered

1) Workshop

While we didn’t expect to cultivate agency in 3 hours, we saw hints of agency emerge during our discussions and post-workshop evaluation.

A participant holding his prototype of an idea based on "social reputation coach" to undo the effects of race and class biases.
A participant holding her prototype of an idea based on tracking nutritional needs of children.

2) Yard Sale

We had a total of 19 visitors and engaged in conversations about workshop participants’ speculative objects with 10 visitors. We filmed some of the visitors’ responses.

A neighbor who shared her perspectives on personalization algorithms that might inhabit digital products of the near future.

Since conducting this workshop, we discovered other companies have used a similar process.

Design Process

Understanding the issue at scale

We conducted 6 interviews with experts in the field and read over 50 pieces of academic literature and industry reports.

A screenshot of a Skype call with Pamela Pavliscak to discuss her work with emotional design, artificial intelligence, and practical ethnography.

Highlights from discovery research

Slide with a quote from Audrey Desjardins, an Assistant Professor of Interaction Design at the University of Washington.

1) People felt conflicted about how machine learning algorithms “see” them because while they didn’t trust the intentions of some companies (i.e. Facebook, Google), they became frustrated when digital services didn’t “know” them.

2) People see the data and content that are served up but not how they’re being pulled together or weighted. Some people tried to manipulate algorithms through guesswork.

3) It’s hard to challenge a decision made by algorithms when we only see the output – much less hold the system accountable. They encode human biases in the form of rules, categories, and criteria.

Bottom line:

The ability to change how algorithms work rest with developers, designers, and researchers. Without challenging our own personal assumptions that go into designing algorithms, we recreate current inequalities in algorithmic systems.

Reframing the challenge

We have an opportunity to shift the agency back to “non-experts” – in a process to highlight people, perspectives, and values that might have been forgotten during product development.

New challenge:

Empowering non-experts to voice their concerns and tensions about how machine learning algorithms are currently designed.

Opportunity:

How might we empower people to discuss, critique, and reflect on ways in which AI programs might find their way into unexpected areas of our lives?

By working backwards from preferred visions, we can enable tangible actions today while including alternative values and considering the ethical implications of rules, categories, and criteria we embed into algorithms.

Arriving at our concept

Push into the “ridiculous” to challenge what we think is a “desirable future.”

We took inspiration from Dunne & Raby’s ideas on speculative and critical design and Montgomery & Woebken’s hands-on, participatory approach to challenging conventional values.

To see how we might apply these ideas in practice, we borrowed from Dr. Dan Lockton’s “designing for agency” and A.B. Kocaballi’s agency-sensitive design qualities.

How can we empower everyday people touch, see, smell, and imagine “the future” today?

We want to…

  1. Empower people to add their own personal critiques and desires into how the future is currently being imagined.
  2. Make algorithms and their impact more visible and tangible.
  3. Include a more diverse voices when we discuss future implications of algorithms.

We decided to modify a participatory workshop framework from the Extrapolation Factory. Participants made speculative, future-facing objects and then placed them in everyday contexts to engage the public in spontaneous discussions about possible futures.

The set up for the speculative design workshop that the Extrapolation Factory created.
Workshop setup from the Extrapolation Factory.
The Extrapolation Factory sold participant-made objects in a real convenience store in New York City after the speculative design workshop.
Speculative objects from the workshop were displayed in a bodega in NYC where unassuming customers can see them next to real merchandise.

Iterating the workshop concept

1) From present solutions to future implications

We modified the Future Cone and the Futures Wheel to stretch participants’ imaginations beyond present-day scenarios.

Visual framework of the Future Cone and Storymaking portions of the speculative design workshop we set up in a large classroom wall.

We curated a list of future signals from research firms like Gartner and Intel to provoke discussions and probe for people’s desired relationships with AI programs.

Examples of near future scenarios.

After choosing a future signal, participants wrote a narrative to articulate their critiques and desires related to the prediction. I created a deck of cards with provocative questions to help participants articulate their thoughts.

FutureShift card deck to help facilitate conversations about the near future scenarios.

2) From barely noticeable to visceral

Participants made speculative objects to playfully materialize their hopes and concerns about AI programs. By using materials that people can see, feel, and smell, we were able to talk about other possible versions of future in a more concrete way.

A prototype session where a participant tests out the object-making portion of the speculative design workshop.

One challenge was to overcome professional designers’ and engineers’ point-of-view already encoded into finished products. So we broke apart finished products to make them more modular and “playable.” We also carefully selected items that loosely communicated how they should be used.

A participant investigates materials for object-making during the speculative design workshop.

An appropriate context encourages people to consider near-future products side-by-side with the everyday and the “outdated” products. This sort of comparison brought out visceral and emotional connections to potential scenarios embodied in future products, and we used them to generate discussions with the public.

Selling ordinary items alongside participant-made objects at the yard-sale-of-the-future event in a residential neighborhood.

We tested our assumptions around how “familiar” a context should be to ensure it grounded both the public and workshop participants.

Brainstorming public engagement ideas on a whiteboard.

We landed on the yard sale of the future concept because it was accessible and familiar to people and we could set it up within our time constraint.

3) From deductive thinking to inductive thinking

Future predictions are often dictated by research firms and academic think tanks. To make it more relatable to everyday life, we tested two warm-up activities: Meet My Algorithms and Algorithmic Self Drawing Exercise.

Several participants placing popular consumer digital products and services on a 2x2 matrix during a warm up exercise for the speculative design workshop.
Meet My Algorithms activity helped participants get into the right mindset before the workshop.

We sparked spontaneous 1-on-1 conversations with participants to help them articulate their hopes and concerns about a future prediction.

Facilitator guiding a workshop participant during the scenario selection step.

We also worked 1-on-1 with participants during the prototyping activity to help them “think with” the materials.

Facilitator leading a workshop participant in the object making exercise to prototype personalized experiences based on a future scenario they chose.

In the post-workshop survey, participants said that our facilitation made the workshop more enjoyable and helped them materially express the narrative they wrote.

Reflections

Coming soon!

More Projects