Marc G. Bellemare
is TalkRL podcast. All reinforcement learning all the time. Interviews with brilliant folks from across the world of RL. I'm your host, Robin Chauhan. So I am super excited to introduce our guest today.
Robin:Professor Marc G. Bellemare is a research scientist at Google Research, an adjunct professor at McGill University, and a Canada CIFAR AI chair. Thanks so much for joining us today, Professor Bellemare.
Marc:Thank you. It's a real pleasure to be here today.
Robin:So how do you describe your area of focus?
Marc:Right. So that's a that's a great question to start. I'm a reinforcement learning researcher, so really I care about all things reinforcement learning. But if I want to narrow it down a little bit, I'd say I care about two things primarily. The first one, I would say, is the problem of representation learning or learning representations, and the other problem is the problem of exploration.
Marc:And the way I think of these two problems is basically how do we think or understand how any intelligent agent, describes in their brain or in their machines what they know? And then, how do they behave or how do they act on the basis of that knowledge? And, in some sense, you know, to me this is really the core of, artificial intelligence, especially when we think about agents and reinforcement learning. As humans, we're incredibly good at this. So let me give you an example.
Marc:When I've moved to a new city in the past, at first none of the streets, none of the signs, none of the landmarks were known to me. And so what do we do? Well, know, at first we start exploring maybe the neighborhood, looking for a grocery store, looking for a pub, looking for a park, and we're very good in general at sort of making this mental map very quickly from very few experiences of samples and knowing where to go next. Know, I found a grocery store, I'm not going to go look for more grocery stores, even though there might be a better one just around the corner.
Robin:So can you tell us a bit about your path in coming to RL? How did you end up in RL?
Marc:I've been excited about reinforcement learning, since my early days as an undergraduate student And actually before then, I can sort of date this back to my teenagehood, I was really interested in AI. Actually, there's unfortunately, I can't find it anymore, there used to be this GeoCities webpage where very naively had laid out a plan for doing AI research. I think I'm 11 or 12 years old at that point. But I was lucky at McGill University to, to be introduced to RL specifically by working with Professor, Professor Joanna Prekup, who is in Montreal, who was teaching the AI class and then, I really loved the idea of reinforcement learning, it made sense to me that AI should be about learning and so from that point on we worked together, I joined her lab to be an undergraduate research assistant, did my Masters with Professor Perkop, and then after that, I went to the University of Alberta where, Richard Sutton, was and is and many other phenomenal researchers in RL. And so, really, it was accidental that I ran into RL, but I loved it.
Marc:You know, was love at first sight and since then it's been just following, where RL is happening and what's exciting in the field.
Robin:So as many of our listeners will know, you've been involved in many of the important advances in RL research, and including, I'll just list a few here, co authoring the DQN Nature paper that arguably started the deep RL revolution, introducing the ALE, the Arcade Learning Environment, which has been a central RL benchmark and still is, and of course, distributional RL. So how much of this path would you say was kind of planned in advance, or did it involve a lot of exploration or luck?
Marc:It's hard to separate luck from plan. Definitely, I didn't come into this thinking, you know, I'm going to start my research in reinforcement learning as an undergrad and then eleven years down the road or what is it, fifteen years down the road, I'll have this distributional reinforcement learning idea. I'm very much someone who likes to think of opportunities as something to be taken and not taken. I think in many cases it's a mix of being at the right place at the right moment but also challenging myself to be in those places. One way to think about it may be to avoid early local minima and try to push the boundaries of what we know and challenge what other people think about the field.
Marc:So specifically, if I think about the distributional reinforcement learning, it's a great example for me of a project that that simmered really for a very long time before before we eventually put out that paper in 2017. There's still something ongoing now. The project actually started very early in my time at DeepMind. I was working at a time with Joel Vanessa, had been my PhD advisor actually at the University of Alberta, and he had this idea of predicting the probability distributions of random returns. And at the time, this seemed very strange and esoteric and we worked on this and we actually, to this day, think this is phenomenal work.
Marc:We we actually use a compression algorithm to do reinforcement learning. It's it's a bit wild. And when we were done, there were actually a few open questions. We looked at this and we said, have no idea how to deal with these problems, but it feels like we should work on them. And it took about three years to eventually get to distribution around.
Marc:And so there was no plan to get there, but the question was there and and when it felt like we had the right pieces in place, then we then with other coauthors, we actually took on this problem. And so I really like to think of it as, you know, the the the work I'm most proud of is is a buildup of experience rather than a single sort of idea.
Robin:So we've seen an explosion, in work building on the seminal DQN letter that you co authored, in Nature in 2015. And, more variants seem to show up all the time on archive almost every week. And Google Scholar says there's over 14,000 citations for that paper. So when you did that work, did you have a sense that, that you were creating this whole new field?
Marc:Right. It's pretty amazing the, the amount of interest and the the revolution that the DQN algorithm created. It's worth pointing out actually that people had used neural networks before with reinforcement learning, right? So dating back all the way to Jerry Tesoro's TD gammon was explicitly using a network of sigmoid units to learn to play backgammon. And when Andrew Ng and Peter Beale flew helicopters in early and mid two thousands, they were also using neural networks as part of their project.
Marc:I think the big revolution with DQN was sort of taking this to the next level. And in fact, Martin Reedmiller was a member of the team who worked on this. He'd also been using neural networks, in similar context. But with Atari, we we had an extra piece, which is we wanted the system to be a general purpose, and this was really a game changer, right, to say you have 60 games and you need to play all of these 60 games. And we'd struggled to come up with the right solution during my PhD to this problem.
Marc:We could only think of heuristics. DQN was revolutionary because it said here's one way you do it, in a very, very clean and simple manner. Now you asked, did we know we were gonna create this whole field? At the time, the paper was incredibly controversial because it it went against, I think, a lot of the things we thought were true or important in reinforcement learning. So, let me say it actually took me about two years to even think that this was an important result, in the sense that this is how we should do things from there on.
Marc:And as a great anecdote for me, what changed my mind was the we had, it happened that actually the the Dikkin work we had done on, roughly 55 games, from the Atari 2,600, but the the paper that I had written during my PhD had a different set of games. There was three games that hadn't been included in the DQN paper for various, engineering reasons. So I I saw these games and I thought, here's my chance to prove to people that they're wrong about deep neural networks, and I'll I'll run DQN in these three games. It will fail miserably. I will have, you know, I'll I'll collect my own human scores, and then, you know, I'm exaggerating a little bit here, but effectively, this is a real test set.
Marc:And lo and behold, I trained DQN in these three games and it beat me single handedly on all three games and I thought that's it, there's just no going around this evidence.
Robin:So I think when David Silver introduced you at NeurIPS twenty twenty, he described your work on distributional RL as one of the most important innovations in RL to date. I'm paraphrasing there. So I wonder besides distributional RL, would you describe any other innovations since DQN as being very important to the theory and practice of RL on that level? Like what types of things might we consider fundamental advances versus more incremental improvements?
Marc:I think there's been a lot of, both sometimes, you know, even things we might think of as incremental still have, still have long term value because they get developed over multiple papers. And sometimes they're important even though we haven't finished exploring them in some sense. If I think about prioritized replay, which is something actually we're revisiting right now, I think prioritized replay has been an important piece in the puzzle. I don't think we fully understand it just yet. Certainly, think a major technical achievement or almost like a paradigm shift is the idea of doing distributed computing and distributed reinforcement learning.
Marc:And the idea that if we have a simulator, can now go and train or run hundreds, if not thousands of agents in parallel to collect this data and also distribute, of course, the computation, the learning part on multiple accelerators. That's been fundamental, in all projects where right now the only way we know how to solve these problems is by throwing a massive amount of compute at them, right? So, it might bring down the training time down from years to a matter of weeks, and that's day and night for any kind of practical application.
Robin:So, DQN itself was relatively simple. And since then, complexity has gone up by quite a bit. If we look, for example, at agent 57, which is also targeting ALE, You'd have to read a lot of papers to understand all the different components in Agent 57. So I wonder how you feel about where things are going in terms of figuring out what the key components are that are needed to do RL well, and is that process just kind of getting started or are we nearly there in terms of figuring out what that is and what is there, how do we know when we arrive there?
Marc:I think a challenge in knowing what we need and what we don't need is that we, as a research community, I think we need more tests, of the methods in new settings. An example I like when I think about this question in particular is the UCT algorithm that, that was effectively designed for search in large environments, and really has had its heyday in the game of computer Go. UCT is effectively a very fast search technique based on a very few simple principles. UCT is often thought of as it's a very simple idea that's very difficult to break and cope. And I think in many ways, the state of reinforcement learning right now is almost the opposite of this, which is we have a lot of bells and whistles, and it's not clear that they're reliable or robust.
Marc:But this said, I think the pieces are there. We just need to to figure out through more trial and error, maybe through more experimentation, which parts matter. So I think I think we have most of the parts. If I look at our experience working with Lune on flying balloons with deep reinforcement learning, there we made the choice of keeping it simple. It was a choice we had to make just because we were building everything from the ground up and when you're building everything from the ground up, any sort of thing that you leave in the system that you haven't tried out might cause you trouble down the road.
Robin:So we see, agents and algorithms getting more complex. Do you think that that diversity and complexity will continue in specialization? Or do you see things kind of unifying, at some point? Like it seems like there's so many different, almost like a family tree of RL agents and algorithms. Do you see that continuing to split or some kind of unification happening?
Marc:I do think I don't think we'll see some kind of unification, but I think it's always been a challenge in reinforcement learning given the vast diversity of problems that we want to bring, that we want to use our algorithms on. Really, each of these problems maybe needs to be handled a bit differently. An analogy here is that if we think about computer vision and say natural language processing, these two things are fairly different perceptual spaces but also the kind of problems that people look at are also pretty different. Continuous control and let's say Atari are not as distinct as these two things, but they're still pretty distinct, right? And it might just not be possible to unify the two if really what we care about is top performance on one of these benchmarks.
Marc:So this said, I think if we let go a little bit of the state of the art, the desire to have state of the art performance in a benchmark and we focus more on will this do the job, then we would start unifying algorithms a bit more.
Robin:So maybe related, like we see games like StarCraft needing a lot more domain specific structure in their agents, like we see in AlphaStar. And then with DQN, we had very simple monolithic agents, that might not do well, in without that structure. Should we expect monolithic agents to be useful going forward or are they just maybe a phase? Or is it maybe because our function approximators aren't that good yet that we need these more complex agent designs and then in the future could maybe better function approximators allow us to fall back to monolithic designs again?
Marc:I think it's interesting to also ask the question, do we expect an agent to be monolithic? And so let me try to unpack this a little bit here. If we look at DQN, DQN was already an agent, what I would call an agent architecture where one piece is the network, one piece is the learning rule, one piece is the replay buffer, one piece is the target network, one piece is how you select actions. So already I think DQN, I would actually call it an architecture more than a monolithic design, and I agree with you that this seems to be a trend that's continued. I think it's actually very natural that if we have a complex system with a lot of moving parts, we might want to build specialized modules to deal with each of these parts.
Marc:And it might not be possible to write down, if you will, a unifying equation that would unify all these parts in a very nice elegant mathematical or algorithmic formulation. Think about an operating system, right? Nobody would expect an operating system to be monolithic, I think, in that respect.
Robin:So speaking of function approximators, like neural networks have come a long way since the original DQN. Do you think that we need, we'll get more progress from just from like tagging along with supervised learning and the improvements in neural networks and function approximators?
Marc:I think we've seen, pretty impressive gains in performance from using transformers, but it's not clear to me that the problems that supervised learning is addressing are the problems that RL needs to address. So in that sense, I would love to see is more transformer like things designed for reinforcement learning. And maybe in fact, we saw a bit of a flurry of this early in the days of DQN and Atari and we see a bit less of it now. I would say where deep learning or supervised learning as as assisted reinforcement learning is when the modalities look the same. Right?
Marc:If you have images as inputs, then you use a convolutional network or something like it to process these images. But if your images, if your inputs are, you know, a vector of atmospheric data, then maybe the convolutional network doesn't make sense anymore.
Robin:So people talk about three types of unsupervised, supervised, and reinforcement learning. And then it seems clear that reinforcement learning can subsume supervised learning just by treating action labels or actions as label predictions. I wonder if, do you think that anything could ever subsume RL or would we ever always think of it as like the cherry on top as Jan Lakun says? Or is that even a question that make any sense?
Marc:That that, is RL the cherry on top? I think there's many models we we aren't even considering right now. A different way to think about this is why are we why do we first of all, you know, RL is is Rich often would say RL is is a is a a problem setting, not necessarily a solution. And I think in that respect, if we just think of this the class of problems we can can describe with RL, it's a pretty wide class. There are problems that don't fit in the paradigm of say a Markov decision process, right, which assumes that effectively given a state it doesn't matter what happened in the past.
Marc:There's actually this framework called AIXI by Marcus Witter who is an incredibly general framework, so it's good and interesting to ask the question, why aren't we all using AIXI? That's a model that could subsume RL in the way that you're asking. And we don't use it, I think, because it's so general that it's very difficult to make progress. So maybe a different way to answer your question is I think we will need a new model, and we'll get to a new model once we understand the failings of the current model. The same way that for a while now we've understood that it's very difficult to learn from trial and error to make decisions in a purely machine learning or supervised learning context.
Robin:So, what would you say are the main bottlenecks to progress in RL right now?
Marc:I would say that benchmarking is an incredible bottleneck and actually let me elaborate a little bit on this. It's not so much the availability or unavailability of benchmarks but rather we don't really have problems that I feel are fundamentally challenging us in new ways. I actually want to hark back to the previous question. We don't really have problems that are challenging, our use of reinforcement learning as a model for, how an agent interacts with its environment. And, you know, why is that?
Marc:I think in part it's because ever since deep reinforcement has come around, we've had a lot of interesting follow-up questions, and they're all incredibly important. But it's also that it might be a question of hardware or it might be a question of computation that we're we're simply lacking the the inspiration, if you will, to to go to the next step. You know, Ray Kurzweil often talks about s shaped progress, right? We're the flat part of the progress curve and for a while everything looks the same and then there's this paradigm shift and everything changes and then we are a new flat part. I definitely feel like we're in a flat part right now and when somebody comes up with that next paradigm, then we'll see a massive upheaval and they'll unblock everything.
Robin:Can you say anything about the relationship between empirical RL, theoretical RL and RL in neuroscience? Like, do they all inform each other or is there maybe more structure than that? How do you see those three things?
Marc:They certainly inform each other. And for me personally, the way that, this plays out is that I I love reading papers across the field and trying to understand the perspectives that these, these different subfields will take on the problem. Now now your question is, you know, do they inform each other or do more than that? I think one challenge when you cross fields is to be able to speak the same language and then to understand the problems or the challenges faced by one of the subfields. As somebody who's actually straddling theoretical reinforcement learning and empirical reinforcement learning, I'm first of all, I'm very grateful that my colleagues on both sides sort of seem to be very happy with with with this this straddling.
Marc:But it does make it really difficult because we have to play catch up understanding both why do people working on the theory care about this specific question and how does it translate into a practical concern, does it? And and vice versa, right? People might work on a practical concern, would be fairly easily addressed from a theoretical perspective. But at end of the day, I think this is how we make progress is by bringing new perspectives into our own problems.
Robin:So you work on exploration. How do you explain why exploration in RL is such a hard problem? Why is it so hard?
Marc:I think there's a number of answers to this question. Maybe the most important one is that we don't know I don't think we actually know what we're looking for. Actually, from a theoretical perspective, exploration is really well understood. There's still phenomenal work coming out in this space but I think we've identified the major pieces of the puzzle and we can derive algorithms and have sample complexity bounds that say if you collect as much information, then then you're done. And so why why is that not enough?
Marc:Why aren't we done from a practical perspective? Well, first of all, it's very difficult for theoretical results to go beyond a certain point or certain level of of of precision. So they tend, for example, to have a worst case analysis of a problem. And maybe the worst case problems are just really, really hard, and we we don't encounter them in practical terms. The other aspect of this is it goes back to this this modeling perspective, which is I don't think we really know what it means to explore in most scenarios that we actually care about.
Marc:Right? So so our our notion of what exploration means is is really grounded in theory, which is collect enough data that you have the right you can make the right decisions. But what if it's never possible to have enough data? You know, I think it's Jeff Bezos who likes to say that we should make decisions when we have about 70% of the available information. That suggests that, you know, and this is more about business, of course, but that suggests that there's a lot of situations where we'll never have enough information.
Marc:There'll always be this uncertainty and maybe we need to think about exploration differently in that respect.
Robin:Do you think humans are good at solving the exploration problem?
Marc:Not in the sense that theoretical exploration would have it. I think humans are very good at having some heuristics. Actually, I'm gonna say no. Humans are terrible at exploration. There's there's this great example of, this is this is, this is taken from the I'll see if I can remember correctly now.
Marc:This is a famous athlete, at the Olympics that discovered the what's called the fosbury flop. And the fosbury flop, in a nutshell, was, this was this was for, this was for jumping over bars. And it was a completely different way of jumping. So running and then jumping and actually flopping over backwards to go over the high bar. And it's a very simple mechanism and once that athlete discovered that jump, everybody started doing it because it made sense.
Marc:And before that, nobody had thought about it. And so what does that tell us that you had generations of athletes, doing these high jumps and not discovering the fuzzy flop? And what did it take to get there. And so I think in many situations when we find a good enough solution, we stick with it and often exploration occurs because we see somebody do it better than we have been doing it or it occurs because we're sort of forced to explore, right? And, it's taken out of our hands.
Marc:But when we have a chance, I think we actually we're very poor at it.
Robin:And we saw on Twitter, an agent coming up with that with that jump in some kind of simulation.
Marc:That's right. That's right. That was just a few days ago. That was actually a really fun moment to see this, to see this online and exactly. I I haven't had a chance to peruse the whole details of this work, but we would exactly imagine that how do you incentivize an agent and what are the conditions in which it's going to be incentivized to say there's something you need to be looking for here that's better.
Robin:On the other hand, the agent couldn't get injured as it tried thousands of variations on jump styles.
Marc:Exactly. I think that's actually a very important point that I I wonder if some of the, some of our biases towards not exploring, it could be injury, it could be it could be time, it could be also that, you know, we have other things in our mind that day and so we're not in a mindset to try things out. We just want, you know, when I order from the restaurant, it's been a while now since I ordered the restaurant, but when I ordered the restaurant, I might stick with something I know just because, you know, this is I don't go out very often and there's a risk to making the wrong choice and so I might as well maybe we're being myopic in some sense, right? That we are, we're making choices that are immediately useful as opposed to optimizing for the long term.
Robin:So going back to ALE, when you first did the ALE paper, how did you think about when it might be solved or was early DQN already effective on ALE at that point?
Marc:So by early DQN, I don't know if you mean the very early, the work I did during my PhD. Is that right?
Robin:Well, there was the 2013 variant.
Marc:I see. So, the maybe I can actually, digress very quickly here. So, the ALE actually was, designed all the way back to 2008, although it took a few years to get it off the ground. So for me, the earliest experiments with, playing Atari games go back even before DQN to the work, I did during my PhD and also a master's student did during their masters, Jvarna Duff. And so these methods were fairly primitive and what they did is they did actually what we knew how to do before.
Marc:So speaking of exploration, we were stuck in a certain way of doing things. We would write down a program that would extract a large number of features from the image and we call these domain independent features because they had to be a program that could work for all 60 games and then the agent would learn from these features. And these actually, the the learner would learn to some of these games, but it also performed quite poorly in other games and was quite slow at times. So to answer your question, how long did we think it would take to to get to where we got? Let's say in 2013 when we published the Arcade Learning Environment paper, we thought it was five to ten years before we would make significant progress, on the basis that we didn't know how to make these features.
Marc:And it turns out we were completely wrong and the answer was, well, you know, throw a convolutional neural network at it and let it do its magic.
Robin:So, ALE has continued, to be used to this day. And so, would you consider ALE still unsolved, by today's generations of agents? Or how will we know when ALE is really outgrown? Any guess on when that might happen? How long will it stay relevant?
Marc:Right. So, term unsolved is always tricky because I've used this, the label solved in the context of, for example, checkers. So, when Jonathan Schaeffer, at the University of Alberta and his team, found, basically solved checkers and said, if you play optimally from the first position in checkers, it's a draw, That is what I would call solved. When we look at Atari, and it's the same thing for gold really, and the game is so large that we're not at that point where we can say this is the optimal play in the sense that it will give you the optimal rate of reward. But this said, think, you know, in a different sense, we're very close to saying, well, we have superhuman players, so aren't we done?
Marc:So I think in that sense, the ALE is solved. We have algorithms for creating policies that do achieve things that, you know, in many of these games they're beyond humans. There's actually a few games for various reasons that are really hard to play on the keyboard and humans do pretty poorly. Breakout is one of them. And and, you know, you see these agents actually, in this case, they are finishing the level because there's a bug where the Atari cartridge will crash after level two.
Marc:So these, I suppose, we would call almost solved. But I think the real value of a benchmark is not so much in being solved as much as how it inspires us and challenges us. And, one place where it's very clear we haven't done this is we haven't really demonstrated something we thought we would be able to do much more quickly when we started working on this, demonstrated the the ability of an agent to learn quickly and from few experiments. And there's been actually a lot of interesting work in model based RL in trying to get there, but I don't think we're quite there. Brandon Lake had a great paper in 2016 where they made that point much better than I'm making it right now, where they said, let's actually look at human agents and ask the question, how long does it take humans to learn to play a game like Breakout or Frostbite?
Marc:And we see a learning curve within episodes, right? It's the third playthrough and you're already twice as good as you were in the first playthrough. We're nowhere near this with RL today.
Robin:So, the original Atari games themselves came out in, I think, 1977, that's forty four years ago. And the RL community is is kind of still working on them with ALE. Do you think that games will always be, like, far ahead of our ability to to learn strong agents to play them?
Marc:That's right. So I I'm guessing here you mean by games that there are new games being created that are more challenging.
Robin:Mhmm.
Marc:I don't think so. And, the answer to this, I suppose, is is multifaceted. But, actually, let's look at the games that people are playing today. You know, my partner and I have been playing a lot of Overcooked lately. And and in many ways, Overcooked is a lot simpler game than some games we saw in the nineteen nineties or the early two thousands.
Marc:So I don't think games are necessarily getting more complex, but you're right that maybe there's something special about Atari that we've lost since. Atari games were games that came out of the arcade cabinet, the the arcades, and and and, you know, they had to be implemented in arcade cabinet, and they they were all designed for the most part, actually, all of them, but all most almost all of them were designed to be fast to be fast paced and to to give the player continuous rewards. Right? You're playing Space Invaders and you have to keep getting score and increasing your score, and the game has to end soon so that the arcade cabinet can collect more quarters. That's something very special and even with the NES, we don't see this anymore and even less so with with later platforms and, you know, you have these beautiful open ended games today that that just are completely different.
Marc:Minecraft is a great example of this. So maybe maybe the video games developers will keep coming up with new ways of thinking about games that are more complicated, but I think I think, you know, we have a good handle on certain kinds of games. Maybe more importantly, I think games are just one reflection of our lives, and what's more likely to happen is as we understand better how to apply RL in a real life context, then games will look easier too, right, as opposed to the way we've taken now which is ignore that an entire game is really a depiction, a very crude depiction of a real life. You know, Pong is a game of tennis in some sense. And so once we've cracked out the real life scenarios a bit better, then I think it'll be easier to play these games.
Robin:And then I guess related, is there do you see a natural successor to ALE or is this successor like a larger set of benchmarks? Like would it make sense to collect some games from more recent times into a new benchmark or I guess you partly answered that, that the challenges might be of
Marc:a different kind? Right, exactly. I don't think that a bigger ALE is the solution. We've seen some incredible work in the RL community trying to develop these new benchmarks. If I think about OpenAI for example, OpenAI has been working hard at this for the last almost since their inception.
Marc:They started with G. M, looked at Universe for a little while, you know, there's been ProcGen that is a procedurally generated benchmark that came out recently, and all of these are challenging the field in their own way, but it's related to your question is again, which part of games do we still need to figure out? And I think one thing that Atari did that benchmarks previously hadn't done is to say, you really have to address the perceptual challenge. How do you map images to actions? Right?
Marc:The domains we had before, for the most part, let's say, except for sort of special case applications, were much more, you know, if you think about mountain car, where you have a vector of two, it's two real values and you've got to make a decision based on these two real values or a lot of grid worlds, we have a lot of grid worlds in reinforcement learning where everything is effectively, you could draw it on a piece of paper. So when we look at more games or bigger games, we're just really saying let's keep the perceptual component and crank up the volume to 11, but that's not, to me, that's not really changing the fundamental question. Now, the more interesting question, I think, is when games get more complicated. Again, what does it mean to play Minecraft? It's a very different question than playing Palm.
Marc:But maybe there's other ways that we can ask this question, which is why is Minecraft an interesting problem in the first place and what is the challenge that it's trying to make us face? And I think we don't really know, we don't really have a good answer to this question, which is why we haven't seen a natural successor to the ALE just yet. And maybe in some sense we've seen a fragmentation of the field into multiple benchmarks.
Robin:So, we had your PhD advisee, Doctor. Marlos Machado on recently, and he spoke about the LUN controller, which you both worked on. And we'll have links to his talks on the episode page at talkrl.com. And we will also have a link to your talk, that you gave to the University of Maryland recently where you went into detail on this work. So, I mean, have limited time here, so I'm not going to ask you to repeat, you know, all the very interesting, things that you said in that talk.
Robin:I recommend listeners check out the Machado episode and his talk and your talk. But could you remind us of the overall goal, of the LUN project and of the controller itself?
Marc:For sure. So LUN is a subsidiary of, Alphabet that is now winding down unfortunately, but was tasked with developing basically balloons, giant balloons that could fly in the stratosphere. And one of their missions, is to deliver internet connectivity to regions where that might be difficult, for various, infrastructure reasons. And so, as part of this challenge, the balloons that are being flown are what we would call underactuated. Effectively, one of these balloons is floating in a stratosphere about 20 kilometers high in the air, and, and the only things that it can do is it can go up, down, or maintain its altitude.
Marc:And so if you want to get from point a to point b, what you have to do is you have to catch winds going in the right direction. And that's, as you can imagine, it's really complicated when you're flying in this messy chaotic wind field in the stratosphere. So, what we did is we actually used reinforcement learning to learn a flight controller that could do all of this, in simulation and then deploy that reinforcement link controller to fly the balloons and we actually saw these balloons deployed, for example, over Kenya. This was a massive success for us to maybe in January we could go to Flightradar, the website, and actually on the website see the flight paths of a deep barrel agent over Kenya. This was magical for me.
Robin:I understand it performed really well and did you expect that in the beginning or were there points where you had some doubts that this was going to work out well?
Marc:I don't think I had any doubts in the sense that the way that the CTO Lou and Sal Candido pitched the project to me and also a colleague of my a colleague of ours, James Davidson, who was involved very early in the project, it was very clear that this was a perfect fit for reinforcement learning because of this underactuated nature where really we thought no other controller is really going to be able to do as well here and both a perfect fit in terms of the model, but also the tools that we had available to us. So, you know, discrete, small number of discrete actions. The the analogy is not perfect, but really this look like Atari in the stratosphere. And so to me, that made sense that this is what we should try to do. And and indeed, you know, the the the way we went about this is not quite Atari, but we try to follow the pattern of the AlphaGoal project of, you know, some choices are less important, some design choices are less important, you're helping the network, what kind of training, and focus on the right choices, and I think that paid off.
Marc:So I didn't have a doubt that this would work. I was surprised at how quickly we got there.
Robin:So was it obvious to you right away, it sounds like it was, that you would use model free off policy distributional RL for this? Like, was that very clear from you from the get go or did you ever take any deliberation to to decide to to go with that?
Marc:We did some experiments very early on with actor critic methods, which I suppose would be model free but value based and maybe a bit less off policy. I was actually hoping following this Althago pattern that we could use something like like Althago did, which is tree search with value estimates. And this is actually what what I learned working with Lune, which is you have to understand the problem, and and the problem will dictate some of your solutions. In this case, the simulator is pretty slow and trying to do any kind of search with the simulator is really hard. We can do search, it's with the simulator of the simulator and that's not ideal.
Marc:So model free just emerged as the thing we could do well and it also happened to be the really the only thing we could do. The distributional RL part, I think the project would have worked well without it, but it just made sense because we knew it so well we could control it and guarantee quality of the process.
Robin:Were you pretty confident that the simulator was going to be good enough to get the results you needed and there wouldn't be like a big sim to real gap or was that not a risk?
Marc:I wasn't confident at all and I think, that Marlos would say the same. We were incredibly surprised when the balloon flew its first flight in July 2019 that it flew so well. I used to have Marlos's words on record, about this. Let's let's say that he he was, quite surprised. The thing so you you used the word risk and in some sense we weren't too concerned about risk because when you fly one of these balloons, there's a lot of safety layers, of course, that, you know, this is a real system.
Marc:And I think this is something that sometimes RL researchers forget. When you implement reinforcement learning in a real application, the RL is just one part of a very large system. And so, down to the engineer looking at the balloon and asking the question, that balloon does something fishy, I'll take control. So the risk wasn't there, but the positive results was a surprise, for sure.
Robin:So if I understood the reward, is entirely about staying within that designated circle, is that right? And there was a bit of shaping outside the circle? Did did did that, did did that reward function take some some deliberation or was that pretty obvious to you?
Marc:Exactly. So the reward function maybe to to restate it is if you're within 50 kilometers of the station that you want to station keep at, then you, you receive a reward of plus one. So it's very classic. Right? We we try to keep things simple.
Marc:Zero if you're not in the circle and one otherwise. We found it useful to add a bit of shaping outside the the region. That's an artifact. We really didn't need it to get good performance, but it it helped a little bit. There's an old another component which is that we discourage power usage, and this was done because power when you're flying a balloon is actually, of course, at a premium.
Marc:The balloon is solar powered, and that was necessary. So just to get technical for a second, it's a multiplicative power penalty where we shrink the reward on the basis of, using power. Did it take a lot of tuning? Not really. I think it's funny that early on we actually tuned it quite a bit, but it was sort of, moving the wrong piece.
Marc:The reason why things weren't working is because we didn't have the right distributed training code. Once we fix the distributed training code, we we realized the reward function didn't actually matter that much. The reason for this, the way I like to encapsulate this is when you're formulating a problem as a reinforcement learning problem, you don't want to tell the agent how to do it, you want to tell it what success is. And for us, success is one within the region, so that's what we should go for for really.
Robin:Cool. Okay. So you mentioned how magical it was. I can't imagine what it must have felt like after so much of your work really combined over the years to come up with this result in terms of DQN, the distributional work you've done and the whole field partly driven by ALE as a benchmark. So did you see this as a, as kind of one of the highlights of your career?
Robin:Like what did it mean to you to see that?
Marc:Well, in terms of, you know, is it the highlight of my career? I hope so. It's hard to know. I can't judge of the future just yet. But, was so great for me here is to see, as you say, on a problem that nobody had really considered before that we could bring the tools we knew so well to great success.
Marc:Let me break it down a little bit here. I think the most exciting thing to us was just the fact that Arnold worked in that setting. Forget about good performance, forget about beating state of the art, even though the controller that Loun already had was really, really powerful and tuned for production capabilities, just the fact that it worked was pretty impressive. The reason for this is reinforcement learning in its core is just a handful of equations. And maybe, you know, now with an architecture thrown on top and to say this thing starts its life, if you want to call it that, knowing nothing about balloons and just by trial and error gets to a point where it's now flying a balloon very, very well, that is just amazing, right?
Marc:So, know, maybe making a parenthesis here, this is actually what got me into reinforcement learning. I didn't mention this at the beginning. My very first project was actually applying, redoing Jerry Tazor's work, applying neural networks to backgammon. And, this was an eye opening moment to me. I knew about RL, I'd read a textbook, I'd taken Professor Perkopf's class, and I wrote down this baguette program, and because I love to tinker with things, also wrote an interface so could play against the player.
Marc:And I think one or two months within my internship, I trained a program with a neural network and it beat me. And you know, I'm not a bad back end player and it beat me and I thought, this is amazing, you know, this is a collection of numbers that's beating me at a game that matters to me. So, I think that's generally generally speaking the feeling we've had with the Loon project, in general.
Robin:Awesome. Okay. So, moving to distributional RL. You you've shown, you've shown everyone how effective distributional RL is. Do you think that we should be learning distributions even in supervised learning?
Robin:Or is there something very specific about value functions that makes learning distributions for them especially helpful?
Marc:I think so. Actually have a good colleague of mine, Martha White, the University of Alberta actually has a paper where she looked at this question, specifically, should we should we think of using, the classification loss in in context where we're doing regression, which is in some sense sort of the abstract version of the question you're asking. I do think that in reinforcement learning, there's something a little bit more interesting that happens, and that's because a lot of the distributions that we encounter are a lot more varied. Right? So in both cases, we're mapping inputs, maybe images or or vectors to outputs.
Marc:And in the supervised learning, the outputs are targets, and we don't really expect the distribution to be too complicated. Maybe that's wrong. Right? Maybe we we should change our our our view on this. When we think about return distributions, they're really more elaborate.
Marc:Right? We've seen this when we when we look at return distributions coming out of Atari. For example, we have these visualizations in the 2017 work with space invaders. We've seen this colleagues of mine have seen this in continuous control tasks. The analogy that, that I like to use here is you can think of of RL or classic RL as, you know, taking a photograph of the world.
Marc:There's a real interactions that the agent has with those environments, and the expected value is a bit like a black and white photograph. And and what distributional RL gives you is a color version of that same photograph. It really shows you all the details that you would otherwise be missing out on. I don't think we even know yet what to do with these details, but now we color we have have color photography. What are we going to do with it?
Robin:So in terms of distributional agents, for discrete off policy, I guess model free URL, we've seen, a number of different ways of presenting, the value function distribution. So you've shown C51 and then there's quantile regression and IQN, implicit quantile networks, then FQF came along. I'm wondering is the problem of how to represent these distributions solved now or or are there some open questions there still?
Marc:I guess there's always open questions and it's only a question of how deep do you want to go or how precise do you want to be. In the context of distributional reinforcement, I specifically guess you're asking about the part I would call the representation. How do we describe the, the probability distribution? How do we operate in that probability distribution? I don't think we have a perfect algorithm.
Marc:The, a lot of the field is focused on understanding a little bit better how to make algorithms that are based on on, you know, the right loss functions and the right, coupling the right representation with the right loss function. But maybe the answer to the question, are we done? Depends on what we want to do with the with these algorithms. If, for example, we want to actually, we know that if we want to maximize expectation, we we really need to predict the expectation, except we know that predicting distribution has this funny effect where we we get more stable behavior, and that we don't really understand. But if instead of maximizing expectation now, we want to be to have risk sensitive behavior, you know, maybe I don't want to shave off thirty seconds off my commute if it means that there's a fifty percent chance that I'll miss my turn.
Marc:For those kinds of questions, I think we don't have a good answer on how to do a distributional RL just yet.
Robin:I really enjoyed your kind of talk in November 2020, on distributional RL. That was a tour of distributional reinforcement learning. And again, we'll have a link to that on the episode page. But in this talk, mentioned that there's some evidence for T learning and also distributional value functions in the brain. Is that right?
Robin:And was that part of your inspiration at all for focusing on distributional RL?
Marc:It wasn't actually. The evidence, as far as I understand, came out after we'd done the original work, but it's been a real thrill to the neuroscience community pick up on this and be quite curious to try to understand this. You know, in some sense it makes sense that if you can learn it and it's relevant to behavior that the brain should be learning it.
Robin:So, and then you said in that talk, and I'm paraphrasing, you said something like the methods use, stationary policies for fixed worlds. And then in the real world, it doesn't really make sense to react in a fixed way. Could you help us understand maybe what you meant by that? Were you talking about exploration or ensuring sensible responses to new observations or continual learning or maybe something else?
Marc:I suppose that's a pretty cryptic comment that I made during the talk. My feeling here is that, maybe the way I like to describe this is that most of our interactions are one offs. I'll go to a restaurant once or I'll go on vacation to a remote location once And when we use reinforcement learning, the paradigm we're in is one of repeated trial and error. I've done the same things thousands of times. So to me, there's a bit of a disconnect here between how reality proceeds and the framework that we're operating in.
Marc:So, I see this as something that maybe we need to address going forward.
Robin:So, I want to ask you about something you said on Twitter. I follow your Twitter and I encourage listeners to check out, Professor Bellemare's Twitter account. So you said the RL of the future is not an algorithm in the TCS sense. We think of algorithm as the equations for learning the policy. That's probably too narrow.
Robin:Instead, we need to think agent architectures, understand the relation between equations and non equations. Can you say anything more about that in terms of what do you mean by the non equation part?
Marc:So I guess that's also another cryptic comment. We talked about agent architectures before. The the I think actually some of these these ideas we've talked about today, the fact that reinforcement learning at the end of the day gives us a model. And, if I go back, in fact, and read my colleague, G. G.
Marc:E. Noy's point, he was saying maybe we need the right RL algorithms. Can we start unifying things, for example, or can we come up with the right learning rule? And my response to this is maybe we just need to change the model, and the non equations here are all the things that are not modeled or don't really fit in that neat mathematical framework. I think there's a lot that we don't understand that doesn't fit in the mold of RL that is worth revisiting.
Robin:Cool. And then briefly, I was hoping to ask you about a paper you co authored, Hyperbolic Discounting and Learning Over Multiple Horizons. That's Fetis et al 2019. And, so, so far most work seems to use a fixed gamma discount for a fixed horizon. And this paper looks at, I guess multiple horizons.
Robin:But do you think like ALE led us to focus on, on MDPs with a very certain range of horizons or what type of environments may benefit from from these multiple horizons and maybe hyperbolic discounting?
Marc:Totally. I don't think anybody's have actually made ever made that remark before. But certainly the nature of Atari games means that for the vast majority, take place at a certain, temporal resolution, if you will, right? You're playing at the arcade and you, you have to respond, you know, every half second you have to do something important, and and maybe every three seconds you receive a point for doing something important. And so this is why, in fact, we've been able to use a fairly constant discount factor across the entire ALE.
Marc:Now, this said, there have been domains, for example, DOTA two that OpenAI worked on where they use a much larger discount factor. I think it's in tens of thousands of equivalent steps in terms of the equivalent horizon. This would be a 100 times bigger than Atari or maybe even a thousand times bigger. I think the reason why I suppose, you know, what kind of environments would benefit from hyperbolic discounting or multiple horizons, I think as we move towards more naturalistic environments and less game like environments, we'll see more of this. I think we see plenty of examples of problems where, you know, what matters more is to get a vague idea of the future.
Marc:Maybe what we haven't figured out is why hyperbolic discounting would be better in that context. Is there truly a reason to be hyperbolic? One theory that I love is that that we're bounded agents, bounded rationality agents, and so we have to make some choices on the basis of, you know, finite data and constantly changing worlds and hyperbolic discounting is maybe one solution to this problem.
Robin:Can you say a bit about your, your current research interests and, types of things your your students are working on, these days?
Marc:I think we're looking at a lot of different directions. And in part, that's, that's a consequence of maybe a broad interest on my end and also a feeling that in looking for that next paradigm shift, we do have to keep an open mind that we won't get our next big breakthrough, just by changing the Q learning into a new version of Q learning. So some of the directions we're excited about is, again, the space of representation learning, trying to understand trying to understand how do we describe more complex phenomena. Right now, in some sense, we've tied to the success of DQN that the only real way we know how to build representations is to do deep learning and let the magic happen. And while that's good, a little bit unsatisfying that we can't go deeper and understand this better.
Marc:Another place where we've been doing some work that I'm quite excited, I think is going come out quite soon, is to revisit benchmarking. Again, one way to challenge ourselves is to understand how we design good benchmarks and steady benchmarks and so that what we've been finding in fact is a little bit that a lot of the progress that we're making in the field sometimes is maybe stationary and not much is changing. So this is some of the work that we've been working on lately.
Robin:And then besides your your own work and your group's work, are there things happening in in RL, lately that you're that you're pretty excited about?
Marc:I think from an applications perspective, offline RL is going to be a game changer. You know, it used to be called batch RL, so it's not a new problem, but it we we see it, for example, when we move from video games where we have a simulator to robotics where we we don't. We need to understand how to learn from a fixed set of data. Really, what supervised learning does so well, reinforcement learning really struggles with. So that I'm very excited.
Marc:I think it's going to come in quite soon. I'm very excited to see where we can take things. I also love the work that's going on in model based RL. I think it's addressing an important question of how do we deal with counterfactuals, the fact that many things that arise around us we might not have been exposed to before. I think last but not least, I'm also excited to see what neuroscience can contribute to reinforcement learning because whenever I pick up a neuroscience paper, I see an incredible amount of interesting phenomena that we're completely ignoring.
Robin:So, Professor Bellemare, I can't really explain to you how much this episode meant to me. I've been reading your name in the literature for years. I've been a big fan. And if anyone would have told me that I would have a chance to interview you in this podcast when I started in 2019, I'd not I probably would not have believed them. So, I want to thank you, so much for sharing your time and your insight with me and the TalkRL community today.
Robin:Thank you, Professor Bellemare.
Marc:And the same to you. Thank you. This has been a fantastic, opportunity.
Robin:Notes and links for this episode are at talkrl.com. If you like this show, I need your support. You can help in a few ways. One, subscribe on your favorite podcast platform. Subscriptions make a big difference.
Robin:Two, follow us on Twitter at TalkRL Podcast. We love retweets. Three, give us a five star rating on Apple Podcasts. If you don't think we deserve five stars, let us know on Twitter what we could do better. TalkRL.
Creators and Guests
