Help in creating Adversarial Environment

[AizazSharif]

Hi @praveen-palanisamy,

I wanted to try the adversarial multi-agent example as mentioned in the related paper, but there are only 2 available examples and an adversarial example is not in it.
Kindly help in how to create such env and use it for training and testing.

Any information would be helpful.
Thanks

[praveen-palanisamy]

Hi @AizazSharif ,

An adversarial scenario was not explicitly evaluated/studied in the paper but it is easy to add one to MACAD-Gym.

1. Sample guide to create and new learning environment or scenario. This gives you the list of changes to create your own custom MACAD-Gym environment with a custom scenario.
2. Specifically for the adversarial multi-agent environment you are interested in, you can re-use this urban driving through a signalized intersection scenario where 3 cars are crossing the intersection. To make the Actor car2 behave adversarially to the car1 Actor for example, you could change the following reward function spec for car2:
   

   macad-gym/src/macad_gym/envs/intersection/urban_signal_intersection_3c.py

   Line 57 in d0a2a75

   "reward_function": "corl2017",

   
   to use your own custom reward function or simply create a negative version (mini-max's counter part) of car1's reward function.

For your reference, the Reward functions are implemented in reward.py.

I will be happy to help you if you need any further assistance in putting this together as a pull request.

[AizazSharif]

Hi @praveen-palanisamy,
Thanks for the help. I will go through the instructions and will update here soon. I asked about the Adversarial example since it's mentioned in the naming convention by the name 'Advrs'.

In urban driving through a signalized intersection scenario I wanted to ask if the multi-agents training are done through shared policy or separate. I am interested in working on separate policy training.

[praveen-palanisamy]

Yes, you could use separate policies for each of the agents.

An example where different policy parameters are used for each Car actor is available in the MACAD-Agents repository. It's implemented using Ray and specifically, the following lines shows how you could use different policy parameters for each of the agents:

https://github.com/praveen-palanisamy/macad-agents/blob/c7e48dd3dd1b9ce10dd5d52fc940300e3e328d67/src/macad_agents/rllib/impala_agent_independent.py#L312-L319

                "multiagent": {
                    "policy_graphs": {
                        id: default_policy()
                        for id in env_actor_configs["actors"].keys()
                    },
                    "policy_mapping_fn":
                    tune.function(lambda agent_id: agent_id),
                },

You can also use a custom/different Deep RL algorithm for each or the agents to train the policy.

[AizazSharif]

This is great thanks a lot for the help @praveen-palanisamy. Hopefully, I am able to do this before I create a feature pull request.

[AizazSharif]

Hi @praveen-palanisamy ,
I was going through the framework and was wondering how to change Deep RL policies within Macad-gym. It's visible in macad-agents (Impala, PPO) but I was unable to find it in macad-gym environments to add my own.

Any information would be helpful.
Thanks

[praveen-palanisamy]

Hey @AizazSharif ,
The Agent code which include the Deep RL policy implementation is kept out of the learning environment code MACAD-Gym to allow for modularity and plug-and-play with other RL libraries or RL environments. As you figured, sample Agent training scripts are provided in the MACAD-Agents repository.
MACAD-Agents works with MACAD-Gym and you can change the Deep RL Agent algorithms or policy definitions in MACAD-Agents repository.
Hope that helps.

[AizazSharif]

Thanks for the reply @praveen-palanisamy.
I am facing trouble while using Macad-agent impala examples. Even if I lower the resources the code runs but the cars don't move at all in default settings and without any training or crash Carla restarts the environment. Whereas in macad-gym, the agents make actions, and also we can have 2nd window which shows the agent's POV from the front.

If you have any suggestions or pointer regarding this issue it will be really helpful.

Thanks.

[praveen-palanisamy]

Okay so that does sound to be specific to the the Impala Agent which has it's own resource requirements on top of the environment/CARLA. Since this is a different issue, can you open one on the MACAD-Agents repository to keep things organized?

[praveen-palanisamy]

@AizazSharif : If you need further assistance on the original topic (Help in creating Adversarial Environments), we can continue it as an Idea Discussion.

[AizazSharif]

Hi @praveen-palanisamy ,
Thanks for moving it into a discussion. Regarding the Adversarial Environment, I wanted to ask how to add any gym (or stable-baseline) related model (PPO, etc) into basic_agent.py so that I can first train each agent in a multi-agent environment.

In macad-gym I can easily work on reward function, but without RL algorithm and training environment it looks difficult.

Any help would be appreciated.
Thanks.

[praveen-palanisamy]

You can use stable-baselines or any RL Agent algorithm with MACAD-Gym. A good place to start simple is using this example:

macad-gym/examples/basic_agent.py

Lines 1 to 41 in 38884ac

	#!/bin/env python
	import gym
	import macad_gym # noqa F401
	env = gym.make("HomoNcomIndePOIntrxMASS3CTWN3-v0")
	configs = env.configs
	env_config = configs["env"]
	actor_configs = configs["actors"]
	class SimpleAgent(object):
	def __init__(self, actor_configs):
	"""A simple, deterministic agent for an example
	Args:
	actor_configs: Actor config dict
	"""
	self.actor_configs = actor_configs
	self.action_dict = {}
	def get_action(self, obs):
	""" Returns `action_dict` containing actions for each agent in the env
	"""
	for actor_id in self.actor_configs.keys():
	# ... Process obs of each agent and generate action ...
	if env_config["discrete_actions"]:
	self.action_dict[actor_id] = 3 # Drive forward
	else:
	self.action_dict[actor_id] = [1, 0] # Full-throttle
	return self.action_dict
	agent = SimpleAgent(actor_configs) # Plug-in your agent or use MACAD-Agents
	for ep in range(2):
	obs = env.reset()
	done = {"__all__": False}
	step = 0
	while not done["__all__"]:
	obs, reward, done, info = env.step(agent.get_action(obs))
	print(f"Step#:{step} Rew:{reward} Done:{done}")
	step += 1
	env.close()

You can modify the SimpleAgent class or it's get_action(...) method to use any of the Agent class/methods from an RL library (preferably with built-in support for Multi-Agent RL environments).

[AizazSharif]

Yes I am currently using basic_agent.py to run my environments, but integrating the RL algorithm that supports multi-agent support seems an issue. Is there any work done that you can point out which can be integrated into macad gym (apart from macad agents)?

[AizazSharif]

This issue is highly dependent on #27 I think since you can import any TensorFlow-based model to create a training environment within macad-gym's basic_agent.py code.