My Review of RL (CS7642) Reinforcement Learning
Grade: A
Difficulty: 7/10
Rating: 10/10
Time Commitment: 12 hours/week
-------------------------------
Overall
This is a well organized course with an instructor who has been teaching this course for many years. It's essentially a sequel to ML (CS7641).Lecture quality
The lecture format is exactly the same as ML (CS7641): two professors (Charles Isbell & Michael Littman) having conversations. Usually Littman is the "teacher" who explains and Isbell plays the "student" who asks questions.I know it comes down to personal preference, but I really enjoyed the interactive style of lecture. It was engaging and fun. But I understand many students hate it. Indeed, the lecture is not concise. They intentionally take a detour. For example, for what could be a 5 minute short video of a powerpoint slide with 4 bullet points with some definitions and formula, this course instead presents a 25 minute video where they start with a toy problem and come up with naive solutions and show why they fail, and discuss their limitations, and then eventually derive an algorithm that works. It gives you a whole intellectual journey, with some jokes along the way.
Considering the amount of math they go through in each lesson, this style of lecture really helped me build intuitive understanding.
Assignments
- 6 homework assignments (5% each): It's concise & fun python programming exercise to get hands-on with specific RL topics. Every homework has autograder for students to verify their implementation. Although each homework is small in scope, it's not trivial. Many students often get stuck on some parts of implementation and spend hours/days before finally coming to an 'aha' moment. This is the only class where I regularly went to the office hour to ask questions because I often got stuck.
- 3 projects (15% each): Projects are hard, and requires you to thoroughly read papers, understand and implement algorithms, replicate experiments and analyze.
- Final exam (25%): Closed book, multiple choice format. It's hard but not unreasonable. I took detailed notes of the lecture throughout the semester, so I was able to use them for a quick review and do decently on the exam.
Grading
HW & exam are auto graded. The only human (TA) grading is for the project reports. TA grading standard is not so lenient, but reasonable enough. I got 95 ~ 100% for homework and projects, then got 75% for the exam, barely bringing the total score to ~90%. There was a grading curve so anything above 83% became an A.Thoughts
The instructor (Miguel Morales) was fantastic. I once joined his office hour to ask a homework-related question. He explained really well. He is truly knowledgeable and passionate about RL. I liked his positive energy. I learned so much by listening to him.RL is a hard subject. But unlike the ML course (CS7641) with its notoriously open-ended assignments that tried to cover so many ML topics, this course has a much narrower & specialized scope and each assignment is very specific as to what they want you to do.
I found the overall time commitment for RL was less than other courses like ML, DL, GA, GIOS.
FAQ
-
Can I take this course without taking cs7641 ML first ?
- Yes, it's not ideal but it's doable. The course content is self contained.