LAST WEEK!! It is officially the end of my DREU 2024 experience. I had an amazing time learning about reinforcement learning, policy shaping, and working with a real-life robot. Learning how to implement my own algorithms and analysing its results taught me a lot about human-robot interaction. I spent this week polishing my final report - I learnt how to use LATEX to create multi-columns for the formatting for the report. I have learned a lot from this internship - new software, living in Medford, exploring Boston, and working on PhD and fellowship applications. I am excited to go home after this internship and relax for a month and then apply for graduate school.

The second last week was a blur of editing my final report and presenting for the lab. I gave a 10 minute presentation to the lab about my entire project including my findings. It was a good experience as I could give the lab an overview of my entire summer and could answer their questions about it. Some of the questions were difficult because the lab members were more informative about my topic than I was - which makes sense because they have been studying these topics for a lot longer than I have. My final report is a work in progress. Thankfully, I had the original paper to start off with as a sample but it is difficult to write algorithms and background about a topic that is not that familiar to me but I am excited to complete it and finish my internship.

I cannot believe week 8 is over and I only have 2 more weeks left in the program. Time has flown by!! This week I spent time conducting tests on the algorithms - I increased the scope from a 3x3 grid to a 7x7 grid and conducted lots of statistical tests. My tests got significant results for the new algorithms - they showed that the 2 algorithms are significantly (p < 0.05) different from the original algorithm but not very different from each other. These results are great!! I started writing my report with the help of a template given by Elaine and the original paper’s format. I am going to conduct more tests to collect more data for the report in the coming weeks. My course is basically over - they need me to submit the final proposal next week. In the weekend I plan to mostly relax and spend some time in the neighborhood as I will have to leave very soon.

This week was very exciting!! Elaine was happy with how I was able to implement the algorithm and suggested I get to work on developing ways to improve it. I spent a while trying to understand the best approach and soon was able to come up with an algorithm - Best Actions algorithm. The algorithm worked well and was able to perform better than the paper’s original algorithm. The new algorithm was able to pick the best action based on the human’s prior feedback on previous actions. I was happy with the results. Elaine was quite happy with my approach and suggested I try another approach by adding “features” to the states which will help me identify similar states. I was able to implement that as well and could see that both algorithms perform very well. I am excited to see how they compare to each other as well as the original algorithm. I will conduct tests on all the algorithms to gather information about how well they work. I also had to do work for the online course - the research proposal required revisions. I also had a great time seeing the 4th of July fireworks in the MIT area with my housemate.

This week went by really fast. I worked on developing ideas for my individual project. I knew I wanted to do policy shaping so I was reading related papers and trying to come up with ideas that I could implement for the next 5 weeks. I came up with 9 ideas and presented them to Elaine. We went through each idea and picked one that she said is novel and also doable for the rest of 5 weeks. My work involved the extension of this paper which basically allows a robot to learn using policy shaping even when the human is not paying attention to it all the time. Therefore, this is modified policy shaping. I had to first implement the paper’s algorithm and then come up with ideas on improving their work. By the end of the week, with the help of the graduate students in the lab, I was able to successfully implement the paper. I also had the other summer course classes to attend and was doing their homeworks as well. Their main assignment is the research proposal whose first draft I had to do this week as well. I am excited about improving the paper’s work and playing around with how I can implement my ideas on their algorithm.

HALF WAY DONE!! This week officially marks my half way point into the REU. Anisha (the other DREU student) and I were able to finish debugging our code, restructuring it so that the algorithm is now using a standard Markov Decision Process (MDP) and implement a basic Q-Learning algorithm and uses the same environment to run multiple tasks. We were able to make the robot arm stack two objects on top of one another and also find the shortest path for a 3x3 grid to push a cup from start to end state. This was a very rewarding experience and also challenging as neither of us has any experience doing this. We did run into lots of difficulties and bugs and sought help from the lab members and the postdoc researcher. Anisha and I also started a research course by CRA - [UR2PHD] (https://cra.org/ur2phd/) which is going to be conducted over the course of 4 weeks with 3 two hour sessions in a week where we will learn crucial research methods and also give a research proposal at the end of the course. There are daily assignments for the course which I was also doing for the week including a Mentor-Mentee contract for which Elaine (my mentor) and I sat down together and discussed our goals for the research experience; I found this very helpful because it helped us come on the same page regarding the next 5 weeks.

Week 4 has been interesting. We finally were able to implement the robot to learn how to move the cup from the start to end state using QLearning Reinforcement Learning algorithm on Tuesday. It was really satisfying to watch it work! Elaine was very happy with our progress! Our next steps are to convert that code to a generalizable Markov Decision Process code which has been a tough process. For the rest of the week, we spent the entire time restructuring the code. There is still work to be done on that. For the weekend, I had family visiting Boston, so I spent the weekend roaming around the city and going to good places to eat.

This week we finally made progress with using the robot arm. Elaine was not here for the lab meeting as she had too many meetings so Reuben (the postdoc) helped Anisha and I make progress with using the robot arm. With his help and a few other graduate students in the lab, we were able to implement a Q-Learning algorithm for a 3x3 grid.There is still work to be done to make the robot arm as we were having complications with that. Next steps would be to fix the robot arm issue and then make it pick up a cup and make it drop the cup in the terminal grid. For the weekend I was mostly just cleaning my room, getting groceries, and exploring the neighborhoods in Somerville. I am looking forward to fixing the bugs in our code in the coming weeks.

The week was mostly spent on understanding how to start using ROS and reading different papers on Q-Learning Algorithms, Reinforcement Learning tutorials, and Policy Shaping algorithms. I also met Elaine (my mentor) in-person this week in lab meeting. It was a 4-day week because it was Memorial Day. I spent my weekend going out with Anisha (the other DREU student) and we spent a day exploring the Cambridge area and got ice cream. In the coming week, we are planning on getting started with working with the robot arm and finding out our tasks.

FIRST WEEK!! I had a great first week! It was really exciting and everyone at my lab was really welcoming. My mentor, Elaine was sick so I only met her once during lab meeting via Zoom but most of my work was simply going through the onboarding document which has tutorials on basic Python, numpy, linux, as well as ROS - the software needed to run the robots. I love my off-campus house, it is a 20 minute walk away from my lab. I am excited to venture more into the work.