The course schedule is below. Please be aware that specifics are subject to change. Readings are chosen to be interesting, informative, and, sometimes, controversial. Opinions expressed in the reading s may or may not represent the opinions of the instructors.
Table of Contents
Week 1: Introduction to Human-AI Interaction
Week 2: Practical Machine Learning
Week 3: Agents, Direct Manipulation, and Interactive ML
Week 4: Machine Learning + UX
Week 5: Where Does Data Come From?
Week 6: Data, Bias, and Trust
Week 7: AI (and Technology) Ethics
Week 8: Active Learning (Putting Humans In-the-Loop)
Week 9: ChatBots and Conversational Agents
Week 10: Facilitating Human-Machine Partnerships
Week 11: Recommender Systems
Week 12: Visualization
Week 13: Explainable AI
Week 14: Content Moderation
Week 15: AI, Humans, and the World
Week 1: Introduction to Human-AI Interaction
Goal: Overview of the course and motivation for deeply considering humans in AI systems. Why do humans matter in AI? Where do humans fit into AI systems? What are the opportunities for doing better with better consideration of people?
August 27, 2018 (class 0) - Introduction
In Class:
Readings (read before class):
Assignments:
- Assignment 0 is out today, due for presentation in class 3 (Wednesday, 8/5).
- Sign up for Slack: (link). Please make your username your Andrew ID.
- Read through the schedule, think about which week you’d like to do a panel.
August 29, 2018 (class 1) - Intelligent Augmentation vs. Artificial Intelligence
In Class:
Readings (read before class):
Assignments:
- As you work on assignment 0, think about whether the technology you’re describing is designed to augment human intelligence or designed to replace it. Does it come from the traditional “AI” stance, from the “HCI” stance, or some combination?
Week 2: Practical Machine Learning
Goal: Gain some experience applying machine learning to real problems focusing on classification problems, and understand the supervised machine learning process (data collection, data curation, model building, testing, iteration).
September 5, 2018 - Introduction to Machine Learning Classification
Readings:
In-Class Activities
- [Brief mock panel, Jeff and Joseph]
Assignments
- Assignment 0 due, be ready to present in class
- Assignment 1 is out. Take what you learned in lecture today about setting up your machine learning environment and start early.
Week 3: Agents, Direct Manipulation, and Interactive ML
Goal: The great interaction debate, including tradeoffs inherent in different approaches, what has changed (or not) since the original debates, and introduction to design patterns that reduce negative impacts of uncertainty in AI systems.
September 10, 2018 - Direct Manipulation vs. Agents
Readings:
In-Class:
Assignments:
- As you work on Assignment 1, think about how errors and bias in ML relate to the concepts of direct manipulation and agents that we are talking about today.
September 12, 2018 - Interactive Machine Learning
In Class:
Readings:
Week 4: Machine Learning + UX
Goal: Introduction to machine learning, how it surfaces in user interfaces, and how to think about machine learning performance (errors, recall, precision, etc.)
September 17, 2018 - How Does ML Surface in UX?
In Class:
Readings:
- Kay, Patel, and Kientz. How Good is 85%? A Survey Tool to Connect Classifier Evaluation to Acceptability of Accuracy. CHI 2015.
- Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., ... & Teevan, J. (2019). Guidelines for Human-AI Interaction.
- Krzysztof Z. Gajos, Dan Weld, and Jacob Wobbrock. Decision-Theoretic User Interface Generation. AAAI 2008.
September 19, 2018 - Design + ML
In Class:
Readings:
- Qian Yang. The Role of Design in Creating Machine-Learning-Enhanced User Experience. AAAI Spring Symposium, 2017.
- Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017, May). UX Design Innovation: Challenges for Working with Machine Learning as a Design Material. In Proceedings of the 2017 CHI conference on human factors in computing systems (pp. 278-288). ACM.
- Yang, Q., Scuito, A., Zimmerman, J., Forlizzi, J., & Steinfeld, A. (2018, June). Investigating How Experienced UX Designers Effectively Work with Machine Learning. In Proceedings of the 2018 Designing Interactive Systems Conference (pp. 585-596). ACM.
- (optional) Cris Cristina. The Designer’s Guide to Machine Learning.
Assignments:
Week 5: Where Does Data Come From?
Goal: In the real world, data for the problems you care about won’t usually be prepackaged into nice, already-existing datasets -- you’ll have to create new datasets, and most of the data you’ll be interested in comes from people. This week is about where data comes from, and how you can go about finding, collecting, and managing data. We’ll pay special attention to the humans involved.
September 24, 2018 - Crowdsourcing Data Collection
In Class:
Readings:
September 26, 2018 - Who Is The Crowd?
In Class:
Readings:
Week 6: Data, Bias, and Trust
Goal: Despite our best efforts, all datasets have bias. In some especially bad cases, data might be biased against a marginal subpopulation (e.g., racial bias, gender bias, etc.), but sometimes bias can be harder to spot. Once we recognize that our data will have bias, we can work toward datasets that limit undesirable bias and seek to mitigate potential negative effects of remaining bias in the models we create. As we build models with data that make it into user interfaces, we will work toward understanding how people might appropriately decide when and how to trust the underlying models that result.
October 1, 2018 - Data, Bias, and Targeting
In Class:
Readings:
October 3, 2018 - Trustworthy AI
In Class:
Readings:
Assignments:
Assignment 2 due (grading in-class) now due 10/8
Assignment 3 out (image labeling + biases) Due Monday the 22nd BY CLASS, 2:59PM
Week 7: AI (and Technology) Ethics
Goal: Ethics are the moral principles that govern our behavior in our lives. The ethics by which we choose to develop technologies (including AI technologies) can determine in very real ways how those technologies will benefit (or not) the humans who provide the data to train models, who use the systems we develop directly, or who may face the consequences of AI systems deployed for them or around them.
October 8, 2018 - Ethics of User Data
In Class:
Readings:
Assignments:
- Assignment 3 actually out, Due Monday the 22nd BY CLASS, 2:59PM
October 10, 2018 - Accuracy Problems or Technology Problems?
In Class:
Readings:
Week 8: Active Learning (Putting Humans In-the-Loop)
Goal: In machine learning classes, we sometimes treat datasets as fixed. In the real world, data continues to grow, and a big challenge is thus how to continue to incorporate new data into models as they grow. This is challenging because we need to have continued access to ground truth labels, don’t want to accidentally make performance worse, and most machine learning algorithms don’t allow new examples to be easily added without retraining the whole model.
October 15, 2018 - Interactive Machine Learning
In-Class:
Readings:
- Kapoor, A., Lee, B., Tan, D., and Horvitz, E. 2010. Interactive optimization for steering machine classification. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pg. 1343-1352.
- Fiebrink, R., D. Trueman, and P. R. Cook. A meta-instrument for interactive, on-the-fly machine learning. NIME 2009.
October 17, 2018 - Humans in the Loop
In-Class:
Readings:
Week 9: ChatBots and Conversational Agents
Goal: Chatbots, dialog systems, and conversational agents are starting to be used and interacted with by people at a huge scale. Twitter bots are interacting with unsuspecting people, companies continue to rely on increasingly sophisticated dialog systems to reduce costs, and listening devices are entering our homes at large scale in the form of speech-controlled devices. How are these systems built, what are their limitations, and what might they look like in the future?
October 22, 2018 - Issues with “Smart” Conversational Agents
Readings:
Assignments:
October 24, 2018 - Agent-User Feedback Loops in Chatbots
In Class:
Readings:
- Chatbots: changing user needs and motivations - Petter Bae Brandtzaeg, Asbjørn Følstad
- Luger, E., & Sellen, A. (2016, May). Like having a really bad PA: the gulf between user expectation and experience of conversational agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (pp. 5286-5297). ACM.
- Druga, S., Williams, R., Breazeal, C., & Resnick, M. (2017, June). “Hey Google is it OK if I eat you?”: Initial Explorations in Child-Agent Interaction. In Proceedings of the 2017 Conference on Interaction Design and Children (pp. 595-600). ACM.
Week 10: Facilitating Human-Machine Partnerships
Goal: Both humans and machines struggle to complete many types of tasks well, and oftentimes they have complementary strengths. Constructing fruitful human-machine partnerships is thus promising in a lot of hard domains, but non-trivial in most of them. Humans can struggle to understand and fix the sometimes nonsensical errors made by AI, and rigid AI systems can struggle to incorporate the decontextualized and slow input of people.
October 29, 2018 - Crowds + Machine Learning + Users
Readings:
October 31, 2018 👻 -- Human Robot Interaction
In Class:
Readings:
Week 11: Recommender Systems
November 5, 2018 - Introduction to Recommender Systems
Readings:
- Jack Nicas. How YouTube Drives People to the Internet’s Darkest Corners
- Paul Covington, Jay Adams, Emre Sargin. “Deep Neural Networks for YouTube Recommendations.”
- Zeynep Tufekci. YouTube, the Great Radicalizer.
- Martijn C. Willemsen, Mark P. Graus, and Bart P. Knijnenburg. Understanding the role of latent feature diversification on choice difficulty and satisfaction. User Modeling and User-Adapted Interaction (UMUAI 2016).
- (Grad) Ricci, F., Rokach, L., & Shapira, B. (2015). Recommender systems: introduction and challenges. In Recommender systems handbook (pp. 1-35). Springer, Boston, MA.
Assignments:
November 7, 2018 - Effects of Recommender Systems on People
In-Class:
Readings:
- “Extremism pays. That’s why Silicon Valley isn’t shutting it down” - | The Guardian
- Nguyen, T. T., Hui, P. M., Harper, F. M., Terveen, L., & Konstan, J. A. (2014, April). Exploring the filter bubble: the effect of using recommender systems on content diversity. In Proceedings of the 23rd international conference on World wide web (pp. 677-686). ACM.
Week 12: Visualization
November 12, 2018 - An AI-Centered Introduction to Visualization
Readings:
November 14, 2018 - Visualizing AI to Understand It Better
In-Class:
Readings:
- Jain, M., Kota, R., Kumar, P., & Patel, S. N. (2018, April). Convey: Exploring the Use of a Context View for Chatbots. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (p. 468). ACM.
- Samek, W., Wiegand, T., & Müller, K. R. (2017). Explainable artificial intelligence: Understanding, visualizing and interpreting deep learning models.
Week 13: Explainable AI
Goal: AI systems are being deployed in increasingly diverse and complicated situations, and the machine learning models underlying these systems are often incredibly difficult to understand, having been trained
November 19, 2018
In-Class:
Readings:
Assignments:
November 21, 2018 🦃
No Class due to Thanksgiving.
Week 14: Content Moderation
Goal: The rise of the social web was once heralded as liberating, democratizing force that would only lead to positive changes in the world. In the past few years, we’ve learned that social web technologies are amplifiers, and they can amplify both good and bad aspects of humanity. What is the role of AI in helping users deal with content? In this week, we’ll cover how people are working with AI to moderate content, how content algorithms drive engagement and seek to influence human behavior, and overview technology tasked with identifying hateful, fake, or otherwise harmful online content (and the very real difficulties of figuring out these problems automatically).
November 26, 2018 - The Role of AI in Content Moderation
Readings:
November 28, 2018 - Content Moderation by Amplifying Human Moderators
In-Class:
Readings:
Week 15: AI, Humans, and the World
December 3, 2018 - Jobs?
In Class:
Readings:
- President Barack Obama on How Artificial Intelligence Will Affect Jobs | WIRED
- Nilsson, N. J. (1984). Artificial intelligence, employment, and income. AI magazine, 5(2), 5.
- (Grads) Brynjolfsson, E., Rock, D., & Syverson, C. (2017). Artificial intelligence and the modern productivity paradox: A clash of expectations and statistics. In Economics of Artificial Intelligence. University of Chicago Press.
December 5, 2018 - How to make AI better for humans, and how not to break the world
In Class:
- Jeff and Joseph Discussion/Response
Assignment 6 due Dec 12 by 11:59PM
Week None: Other resources
As we find other resources that you might find interesting, we’ll post them here.
Practical Data Science class: If you wanted to learn more about collecting and managing large collections of data, see Zico’s course on Practical Data Science