Determine main process in distributed training

[PgLoLo]

Hello! During distributed training in DDP mode, how can I determine if the current working code is called from the main process, and not from the spawned processes (I don't want to perform my custom initialization twice)?

[akihironitta]

@PgLoLo You can use the trainer's property is_global_zero:

class YourLightningModule(LightningModule):
    def do_something(self, ...):
        if self.trainer.is_global_zero:
            ...

https://pytorch-lightning.readthedocs.io/en/1.5.9/common/trainer.html#is-global-zero

[PgLoLo]

Is it possible outside of Trainer/LightningModule? I configure logging dirs/checkpoint dirs/profile dirs in the main script, and the naming of these folders depends on the current time. Thus, during DDP I would configure them twice and basically create two sets with different names. I thought that something like torch.distributed.is_initialized would be False only on the first go through the main script, but unfortunately no.

[akihironitta]

@PgLoLo I believe this variable should work:
https://github.com/PyTorchLightning/pytorch-lightning/blob/4d72110b51525296eac16454e7b34e2c1b60cb7e/pytorch_lightning/utilities/distributed.py#L69-L70

[clumsy]

With default torchrun rendezvous strategy rank 0 is not guaranteed to be on the main node for multi-node training, isn't? @akihironitta

Let's say I override trainer.is_global_zero for the purposes of initializing Callbacks, e.g. like progress bar, is it possible it could break anything in Lightning where it assumes it's always on rank0?

[lengmo1996]

Have you solved this problem?