Need testing?

[0xsamgreen]

Hi, thank you for making this port :)

In you conversation with Danijar, I read that you're limited in your abilities to test because of GPU availability? I'm interested in building from your code, and I'd be happy to help run tests for you. I have four Titan Xps I could dedicate to it for a bit. My limitation is that I don't have a Mujoco license (I'm working on it) so testing would be limited to Gym environments.

[Kaixhin]

That would be much appreciated! Unfortunately I'm still waiting for the latest results from Danijar (fixing the bug in the RNN would improve upon the original results), and in order to check that this code is fine we'd want to compare results so that means DeepMind Control Suite.

If you're interested in getting baseline results for your own work then perhaps it would be interesting to have some results on Pendulum-v0 and MountainCarContinuous-v0 (both symbolic and visual observations) anyway? We currently have no idea what appropriate hyperparameters are, and as one of those people who also don't have easy access to MuJoCo licenses I know it'd be nice to have some completely open source reference results.

[0xsamgreen]

Hi @Kaixhin,

I'm running headless, and I'm able to run symbolic mode fine, but I can't run in non-sybolic mode. I have render set to False, but I get the following error:

                          Options
                          seed: 1
                          disable_cuda: False
                          env: Pendulum-v0
                          symbolic_env: False
                          max_episode_length: 1000
                          experience_size: 1000000
                          activation_function: relu
                          embedding_size: 1024
                          hidden_size: 200
                          belief_size: 200
                          state_size: 30
                          action_repeat: 2
                          action_noise: 0.3
                          episodes: 2000
                          seed_episodes: 5
                          collect_interval: 100
                          batch_size: 50
                          chunk_size: 50
                          overshooting_distance: 50
                          overshooting_kl_beta: 1
                          overshooting_reward_scale: 1
                          global_kl_beta: 0.1
                          free_nats: 2
                          learning_rate: 0.001
                          action_noise: 0.3
                          episodes: 2000
                          seed_episodes: 5
                          collect_interval: 100
                          batch_size: 50
                          chunk_size: 50
                          overshooting_distance: 50
                          overshooting_kl_beta: 1
                          overshooting_reward_scale: 1
                          global_kl_beta: 0.1
                          free_nats: 2
                          learning_rate: 0.001
                          grad_clip_norm: 1000
                          planning_horizon: 12
                          optimisation_iters: 10
                          candidates: 1000
                          top_candidates: 100
                          test_interval: 25
                          test_episodes: 10
                          checkpoint_interval: 25
                          checkpoint_experience: False
                          load_experience: False
                          load_checkpoint: 0
                          render: False
Traceback (most recent call last):
  File "main.py", line 85, in <module>
    observation, done, t = env.reset(), False, 0
  File "/home/sgreen/working/planet.pt/env.py", line 87, in reset
    return torch.tensor(cv2.resize(self._env.render(mode='rgb_array'), (64, 64), interpolation=cv2.INTER_LINEAR).transpose(2, 0, 1), dtype=torch.float32).div_(255).unsqueeze(dim=0)
  File "/ascldap/users/sgreen/anaconda3/envs/planet.pt/lib/python3.7/site-packages/gym/core.py", line 249, in render
    return self.env.render(mode, **kwargs)
  File "/ascldap/users/sgreen/anaconda3/envs/planet.pt/lib/python3.7/site-packages/gym/envs/classic_control/pendulum.py", line 61, in render
    from gym.envs.classic_control import rendering
  File "/ascldap/users/sgreen/anaconda3/envs/planet.pt/lib/python3.7/site-packages/gym/envs/classic_control/rendering.py", line 27, in <module>
    from pyglet.gl import *
  File "/ascldap/users/sgreen/anaconda3/envs/planet.pt/lib/python3.7/site-packages/pyglet/gl/__init__.py", line 239, in <module>
    import pyglet.window
  File "/ascldap/users/sgreen/anaconda3/envs/planet.pt/lib/python3.7/site-packages/pyglet/window/__init__.py", line 1896, in <module>
    gl._create_shadow_window()
  File "/ascldap/users/sgreen/anaconda3/envs/planet.pt/lib/python3.7/site-packages/pyglet/gl/__init__.py", line 208, in _create_shadow_window
    _shadow_window = Window(width=1, height=1, visible=False)
  File "/ascldap/users/sgreen/anaconda3/envs/planet.pt/lib/python3.7/site-packages/pyglet/window/xlib/__init__.py", line 166, in __init__
    super(XlibWindow, self).__init__(*args, **kwargs)
  File "/ascldap/users/sgreen/anaconda3/envs/planet.pt/lib/python3.7/site-packages/pyglet/window/__init__.py", line 501, in __init__
    display = get_platform().get_default_display()
  File "/ascldap/users/sgreen/anaconda3/envs/planet.pt/lib/python3.7/site-packages/pyglet/window/__init__.py", line 1845, in get_default_display
    return pyglet.canvas.get_display()
  File "/ascldap/users/sgreen/anaconda3/envs/planet.pt/lib/python3.7/site-packages/pyglet/canvas/__init__.py", line 82, in get_display
    return Display()
  File "/ascldap/users/sgreen/anaconda3/envs/planet.pt/lib/python3.7/site-packages/pyglet/canvas/xlib.py", line 86, in __init__
    raise NoSuchDisplayException('Cannot connect to "%s"' % name)
pyglet.canvas.xlib.NoSuchDisplayException: Cannot connect to "None"

Are you also running headless?

[Kaixhin]

I am not running headless, as I'm using gym's render functionality to get image-based observations. I'm afraid you'll need to run with either a real or fake display.

[0xsamgreen]

I have a MuJoCo license now! Is there a sweep of MuJoCo environment tests you would like run?

[Kaixhin]

Great news! I've started a run on walker-walk, so any of the others.

[0xsamgreen]

Will do. Other than the environment, should I use the default parser arguments of your latest commit?

[Kaixhin]

Latest commit should be same settings as PlaNet camera ready so yes - only other change is the recommended action repeat per environment, which you can see in env.py.

[0xsamgreen]

Thanks, I started a run on cartpole-swingup, finger-spin, cheetah-run, and ball_in_cup-catch.

[0xsamgreen]

Hi @Kaixhin, things are looking good! I will continue to train for another day and then update, but it seems that scores are meeting or approaching Danijar's. Thanks again for making this port.

cartpole-swingup

cheetah-run

cup-catch

finger-spin

[Kaixhin]

Awesome! If possible, do you mind finding a way to send me all the data once done (checkpoints and results)? Perhaps via a file sharing service, and I'll let you know once I've downloaded it all because it would take a lot of space. I'll also take the rewards and final model and make them available as a release. I've got results for walker-walk, so just kicked off a run for reacher-easy.

[0xsamgreen]

Here are the final test result plots! I'm looking into sharing the checkpoints and result logs.

cartpole-swingup

cheetah-run

cup-catch

finger-spin

[Kaixhin]

Awesome! Do let me know if you do something else with PlaNet, but I think the results are good. I'll close this once you get me all the data.

[Kaixhin]

@SG2 if you still have some capacity would you be able to run the same environments again with the latest commit? Among various improvements I've made changes to the image processing to match what was actually done in the original (I missed some of this originally). Should only take half the time now since I've set the default number of episodes to 1000 like in the camera ready.