Added SACD implementation. #190
Conversation
| alpha_lr: float, | ||
| device: torch.device, | ||
| ): | ||
| self.type = "discrete_policy" |
There was a problem hiding this comment.
How does this get handled by gym_environment? is there a respective update? This should just stay as policy no?
There was a problem hiding this comment.
The normalization in the gym environment was breaking things so wanted to make a loop without normalization. Sorry about the horrible naming
There was a problem hiding this comment.
how is it breaking things? it shouldn't break anything
There was a problem hiding this comment.
In the discrete setting, select_action_from_policy() and sample_action() return a tensor containing the integer index of the action to enact. However, the action returned from sample_action() is normalised, while the action returned from select_action_from_policy() is denormalised. To make this work, we would need to configure some sort of redundant reversal of normalisation/denormalisation for one of these functions so that run_action_on_emulator() receives only the normalised or original action format instead of both.
| ): | ||
| super().__init__() | ||
| if hidden_size is None: | ||
| hidden_size = [256, 256] |
There was a problem hiding this comment.
double check default hidden layer size
| if hidden_size is None: | ||
| hidden_size = [256, 256] | ||
| if log_std_bounds is None: | ||
| log_std_bounds = [-20, 2] |
There was a problem hiding this comment.
double check log_std_bounds default
There was a problem hiding this comment.
can remove log_std_bounds not even used here
| dist = torch.distributions.Categorical(action_probs) | ||
| action = dist.sample() | ||
| # Offset any values which are zero by a small amount so no nan nonsense | ||
| zero_offset = action_probs == 0.0 |
There was a problem hiding this comment.
to avoid confusion just through () around the (action_probs == 0.0)
| ): | ||
| super().__init__() | ||
| if hidden_size is None: | ||
| hidden_size = [256, 256] |
There was a problem hiding this comment.
double check hidden layer size defaults
There was a problem hiding this comment.
Where can we get this info? The paper?
There was a problem hiding this comment.
The paper or the code they provided
| actor_lr: Optional[float] = 3e-4 | ||
| critic_lr: Optional[float] = 3e-4 | ||
| alpha_lr: Optional[float] = 3e-4 | ||
|
|
||
| gamma: Optional[float] = 0.99 | ||
| tau: Optional[float] = 0.005 | ||
| reward_scale: Optional[float] = 1.0 |
There was a problem hiding this comment.
double check defaults from paper/code
…learning into alg/SAC-D
| alpha_lr: float, | ||
| device: torch.device, | ||
| ): | ||
| self.type = "discrete_policy" |
There was a problem hiding this comment.
how is it breaking things? it shouldn't break anything
| ): | ||
| super().__init__() | ||
| if hidden_size is None: | ||
| hidden_size = [256, 256] |
There was a problem hiding this comment.
The paper or the code they provided
| if hidden_size is None: | ||
| hidden_size = [256, 256] | ||
| if log_std_bounds is None: | ||
| log_std_bounds = [-20, 2] |
There was a problem hiding this comment.
can remove log_std_bounds not even used here
| import torch | ||
| from torch import nn | ||
|
|
||
| from cares_reinforcement_learning.util.common import SquashedNormal |
There was a problem hiding this comment.
can also remove the squashednormal import here
Tested on Cart-Pole. Attains maximum reward
based on this paper