ECE598SG: Special Topics in Learning-based Robotics

Fall 2019

Overview

This course will introduce students to recent developments in the area of learning-based robotics. The course will start with the instructor providing an overview of background material from relevant sub-fields: computer vision, machine learning, reinforcement learning, control theory and robotics. This will be followed by discussion of advanced techniques for arriving at policies for robots, such as model learning, model-based RL with learned models, imitation learning, inverse reinforcement learning, self-supervised learning, exploration, and hierarchical reinforcement learning, and application of these concepts to robot navigation and manipulation. This part of the course will be covered via student-led discussion of recent research papers that develop and validate these techniques. Course also includes open-ended project work that will provide students a flavor of how to conduct research in this emerging area.

Quick Info

• Instructor: Saurabh Gupta (saurabhg@illinois.edu)
• Lecture time: Tuesday and Thursday, 2.00pm to 3.20pm
• Lecture Venue: 2015 ECE Building
• TA: Xinke Deng (xdeng12@illinois.edu)
• Credits: 4
• Office Hours:
  • Saurabh: Tuesday 3.30pm to 4.30pm, Friday 10am to 11am at 319 CSL.
  • Xinke: Friday 11am to Noon at 139 CSL.

Announcements

• Dec 03: Friday Office hour for Saurabh on Dec. 05 are from 10.30 to 11.30, instead of the usual 10 to 11.
• Nov 19: Friday Office hour for Xinke on Nov. 22 is canceled. Make up Office hour on Dec. 2, 4pm to 5pm.
• Nov 12: Friday Office hour for Saurabh on Nov 15 is canceled. Make up Office hour on November 18, 2.30pm to 3.25pm.
• Nov 12: Nov 14 class moved to Dec 03. No class on Nov 14.
• Sep 27: Friday Office hour for Saurabh on October 11 is canceled.
• Sep 27: Starting Oct 4, Friday office hour for Saurabh will be from 10am to 11am.
• Sep 17: Office hour location for Saurabh has changed to 319 CSL.
• Aug 27: Additional office hour for Saurabh on Wednesday Aug 28 from 9am to 10am at CSL 330.
• Aug 26: Course webpage goes live.

Learning Outcomes

After taking this course you will be able to:

• Understand basic techniques for learning policies for robots.
• Understand and appreciate recent literature in robot learning.
• Design approaches to learn policies for solving various robotic tasks.
• Design experiments or conduct analysis to validate approaches.

Prerequisites

This is an advanced gradate course aimed at graduate students conducting research in relevant research areas. The course will largely cover relevant papers published within the last few years in computer vision, robotics, and machine learning. Students should be familiar with reading and critiquing research papers, and should have a basic understanding of concepts in artificial intelligence, and machine learning. Students must have taken at least one of the following (or equivalent) courses: ECE 448 / CS 440 (Introduction to Artificial Intelligence), ECE 544NA (Pattern Recognition), ECE 549 / CS 543 (Computer Vision). If you are not sure whether you meet the prerequisites, talk to the instructor after the first class or in office hours.

Syllabus

Here is a tentative syllabus for the course. TBD readings will be filled in over time, but at least 2 weeks before class.

Date	Topic	Material / Readings
Aug 27	Introduction and Course Overview
	Part 1: Background
Aug 29	Computer Vision Review	3D Reconstruction, Recognition, CNNs for Recognition. See also Szeliski Chapters 4, 7, 14.
Sep 03	Robotics Review	Configuration Space, Forward Kinematics, Inverse Kinematics, Motion Planning, Optimal Control. See also Modern Robotics Chapters 2, 4, 6, 10.
Sep 05	MDP Review	Terminology, Policy Evaluation, Policy Improvement, Policy Iteration, Value Iteration. Class slides. See also: David Silver’s slides here and here, and Sutton and Barto Chapters 3, and 4.
Sep 10	MDP Review	Model Free Reinforcement Learning: Monte-Carlo and Temporal Difference Learning and Control, Off-policy learning. Class Slides. See also: David Silver’s slides here and here, and Sutton and Barto Chapters 5, 6, and 7, Playing Atari with Deep Reinforcement Learning.
Sep 12	MDP Review	Model Free Reinforcement Learning: Deep Q-learning, Policy gradients. See also: David Silver’s slides here, and Sutton and Barto Chapters 13
Sep 17	Deep RL	Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor Mastering the game of Go without human knowledge
	Part 2: Alternatives to Solving Unknown MDPs
Sep 19	Model Building	PILCO: A Model-Based and Data-Efficient Approach to Policy Search Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
Sep 24	Model Building	Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control SPNets: Differentiable Fluid Dynamics for Deep Neural Networks
Sep 26	Imitation Learning	A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning End-to-End Training of Deep Visuomotor Policies
Oct 01	Inverse Reinforcement Learning	Maximum Entropy Inverse Reinforcement Learning Apprenticeship Learning via Inverse Reinforcement Learning
Oct 03	Self-Supervised and Unsupervised Learning in Computer Vision	Learning Features by Watching Objects Move Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks
Oct 08	Self-Supervision in Robotics	Visual Reinforcement Learning with Imagined Goals Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours
Oct 10	Exploration	Curiosity-driven Exploration by Self-supervised Prediction Diversity is All You Need: Learning Skills without a Reward Function
Oct 15	Exploration	Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play Intrinsic Motivation for Encouraging Synergistic Behavior
Oct 17	Hierarchies	Feudal Reinforcement Learning Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
Oct 22	Social Learning	Time-Contrastive Networks: Self-Supervised Learning from Video Learning Navigation Subroutines by Watching Videos
	Part 3: Case Studies
Oct 24	Navigation	Cognitive Mapping and Planning for Visual Navigation Beauty and the Beast: Optimal Methods Meet Learning for Drone Racing
Oct 29	Navigation	Semi-parametric Topological Memory for Navigation Bayesian Relational Memory for Semantic Visual Navigation
Oct 31	Manipulation	Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
Nov 05	Manipulation	Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost Learning Task-Oriented Grasping for Tool Manipulation with Simulated Self-Supervision
Nov 07	Hardware and Sensors	A Soft Robot that Navigates its Environment through Growth TBD
Nov 12	Multi-task Learning	Task2Vec: Task Embedding for Meta-Learning Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
Nov 14	Modern Deep RL vs Classical Control No class!	A Tour of Reinforcement Learning: The View from Continuous Control Moved to Dec 03 Towards Generalization and Simplicity in Continuous Control
Nov 19	Lessons from Cognitive Science and Psychology	The Development of Embodied Cognition: Six Lessons from Babies TBD
Nov 21	Big Data vs Clever Algorithms	(a) The Bitter Lesson, (b) Re: A Bitter Lesson Intelligence without Representation
Dec 03	Modern Deep RL vs Classical Control	A Tour of Reinforcement Learning: The View from Continuous Control Towards Generalization and Simplicity in Continuous Control
Dec 05	Project Presentations
Dec 10	Project Presentations

Evaluation

Tentative, certain details may be adjusted based on how the class size evolves.

• Class Presentation (20%): Parts II and III of the course involve student-led presentations of recent research papers. Every student is expected to present at least one paper. You should plan for a 20 minute presentation, plus 10 minutes for discussion. In your presentation, focus on why the paper is interesting. Different papers can be interesting for different reasons: some papers can be conceptually novel, some can be presenting new tasks or techniques or capabilities, and yet others can be presenting detailed empirical evaluation; focus your presentation on these interesting bits. Also prepare some points of discussion. The presenter should meet with the instructor with a presentation draft in the week before the presentation is scheduled, sign up for a meeting slot here. Students will be graded based on their presentation.
• Paper Reviews (20%): Students will write reviews for papers covered in the class. Students are required to read all papers, however, they are required to only submit reviews for one paper per class. These reviews should have the following structure: a short summary of the paper, strengths of the paper, constructive criticism of what the paper lacks or how the paper can be improved, and possible future directions arising from the paper. Paper reviews are due before noon on the day the paper is presented. Submit your paper reviews in this Google form (you will need to sign-in into GoogleApps with your Illinois NetID). More specifics on the form of review are provided on the Google form. Lowest two paper review scores will be dropped.
• Class Participation (10%): In addition to presenting and leading discussions, students will also be graded on their general participation in the class.
• Course Project (50%): Students will engage in projects in groups of 1-3. Projects should involve investigation of relevant research questions in computer vision, robotics and machine learning. Projects could be done in simulation, or on real-robot platforms. Evaluation will be based on the following deliverables. Projects are expected to be research projects, they should create new knowledge that doesn’t exist yet. Project evaluation will be based upon the following elements:
  • Project proposal (5%, Due Sept 30). Project proposals will need to be approved by the instructor. After you have submitted the project proposal, please sign up to meet with the instructor and go over your proposal. If your project proposal needs substantial revisions, you will need to submit a revised project proposal. This revised project proposal will be due on Oct 10. Meeting slots are available starting next week, so you are encouraged to submit your project proposal ahead of time, and get early feedback.
  • Mid-term Project Report (5%, Due Nov 4). Think of this report as a paper sketch with proposed formulation, details about experimental setup, preliminary results, and mock final results.
  • Project presentations (20%, in class on Dec 5 and Dec 10). More details coming.
  • Final paper (20%, Due very soon after Dec 10, final date coming soon). More details coming.
  • Submit your project proposals, revised project proposal (if necessary), mid-term project reports, and final papers, by emailing them to Xinke Deng (xdeng12@illinois.edu). Submit by due date, anytime on earth.

Related Courses

• Sergey Levine’s Deep Reinforcement Learning class at UC Berkeley.
• David Silver’s Reinforcement Learning class at UCL.
• Katerina Fragkiadaki, Tom Mitchell’s Deep Reinforcement Learning and Control class at CMU.
• Emma Brunskill’s Reinforcement Learning class at Stanford.

References

• Reinforcement Learning: An Introduction. Richard S. Sutton and Andrew G. Barto. Second Edition, MIT Press, Cambridge, MA, 2018 Available online.
• Computer Vision: Algorithms and Applications. Richard Szeliski, Microsoft Research. Available online.
• Modern Robotics: Mechanics, Planning, and Control. Frank C. Park, Kevin M. Lynch. Cambridge University Press. Available Online.