https://nzxhuong.github.io/enMon, 07 Apr 2025 17:36:37 GMTNikola (getnikola.com)http://blogs.law.harvard.edu/tech/rsshttps://nzxhuong.github.io/posts/prioritized-experience-replay-and-importance-sampling/Ngo Truong<div><p>During the time I’m learning about Deep Q-Learning (DQN), I was stumbling on one thought: with experience replay, we sample transitions in the buffer uniformly. But the way we append <em>every</em> transition means it's likely that most of the transitions are 'bad' experiences, especially early on. Like in the cliff walking problem, early in training we're just running around randomly and falling off the cliff a lot. This leads to the buffer being mostly filled with bad outcomes. Even when we <em>do</em> finally reach the goal and get a good transition, it's drowned out by the huge amount of bad stuff we already stored. This means most of the time we're sampling 'bad' experiences during training updates, and that doesn't feel very efficient.</p> <p>So I started thinking... is there a way to sample more of the 'good' stuff? That's when I found Prioritized Experience Replay (PER), proposed by DeepMind.</p> <p><a href="https://nzxhuong.github.io/posts/prioritized-experience-replay-and-importance-sampling/">Read more…</a> (5 min remaining to read)</p></div>importance samplingmathjaxprioritized experience replayreinforcement learninghttps://nzxhuong.github.io/posts/prioritized-experience-replay-and-importance-sampling/Wed, 02 Apr 2025 13:31:56 GMT