Found mistake in pytorch-lightning DQN example. How do I upload a fix?

[carsondenison]

I am learning deep RL, and as an exercise I looked through and tried to implement this example of DQN in pytorch lightning:

DQN example

I believe that I have found a bug and I am not sure how to flag it or how to upload a fix myself. This is my first time trying to contribute to an open source project and any advice will be greatly appreciated.

One line 106 of the section for the DQN Lightning Module, the expression for epsilon (which decays over the first eps_last_frame steps) is incorrect. The code is currently:
epsilon = max( self.hparams.eps_end, self.hparams.eps_start - self.global_step + 1 / self.hparams.eps_last_frame, )

I believe this is incorrect for the following reason: 0 <= epsilon <= 1, so self.global_step immediately overcomes the other two terms. There is no decay, only a single timestep at eps_start + 1/eps_last_frame, and then timesteps at eps_end. This also seems like an error because with the default values, eps_start + 1/eps_last_frame > 1.

The intended behavior is for the epsilon to decay linearly from eps_start to eps_end over the first eps_last_frame frames. I believe the second argument to max() should be:

self.hparams.eps_start - (self.global_step / self.hparams.eps_last_frame) * (self.hparams.eps_start - self.hparams.eps_end)

Who do I contact, or is there a way I can upload this fix myself?

[awaelchli]

Hello! This fix would be done in the Lightning Tutorials repo here: https://github.com/PyTorchLightning/lightning-tutorials

1. Fork the repo (button on top right)
2. Clone the repo from your fork
3. Go to the DQN example source code/notebook
4. Edit with your proposed fix
5. commit the changes and push them to your fork
6. Then go to the https://github.com/PyTorchLightning/lightning-tutorials/pulls and open a Pull Request.

Let me know if that helps and thanks for checking out the tutorials and the help!

[carsondenison]

Thank you!

I have forked and cloned the repo, and I now have an additional question.

The example is a colab notebook, but the repo only stores a .py file.

The first cell of the colab note book is:

! pip install --quiet "torchmetrics>=0.3" "torch>=1.6, <1.9" "pytorch-lightning>=1.3" "gym"

Which no longer works. Colab has updated the default version of torchvision, torchtext, and torchaudio to 0.11, and these must be downgraded to 0.10 for the install of pytorch-lightning to succeed in the colab environment. The command should be changed to:

! pip install --quiet "torch>=1.6, <1.9" "pytorch-lightning>=1.3" "gym" "torchvision<=0.10" "torchaudio<=0.10" "torchtext<=0.10"

However, this line is just missing from the .py file: dqn.py

Where would I fix this line? Can I put a colab terminal command in the .py file on github?

I really appreciate your help.

This is so exciting!

[carsondenison]

Also, I don't see unit tests for any of the examples. I've just changed the code, should I add a test_dqn.py file somewhere to verify my changes work as expected?

[awaelchli]

Glad you were able to open the PR! Nice job.

We don't store the notebooks directly and rather generate them from a template .py Python file. The comments in there determine what is shown in the to be generated notebook cells. About the requirements, we should ask @Borda.

should I add a test_dqn.py file somewhere to verify my changes work as expected?

That shouldn't be necessary. Afaik we don't have testing for the tutorial. But we make sure that the notebooks compile and are runnable.

[Borda]

This pip install is generated from requirements listed in meta.yaml so shall update packages and versions there...
see in this commit to your PR: Lightning-AI/tutorials@f0b4827