About average in progress bar with on_epoch

[Nokimann]

self.log("val_loss", loss, on_step=False, on_epoch=True, prog_bar=True, logger=True, batch_size=batch_size)
If the calculated loss is averaged on a batch,

then, is val_loss with on_epoch an average of the averaged loss?
So, double-averaged? Or does it take batch_size into account for averaging?

[rohitgr7]

it takes batch_size into account for averaging.

[Nokimann]

That means it accumulates loss*batch_size, then divides the accumulated loss*batch_size by the accumulated batch_size?

[rohitgr7]

yes

[rohitgr7]

here is the piece of code if you are interested: https://github.com/Lightning-AI/lightning/blob/7d1731096609adb97c3a686d1cb5000bbaea4832/src/pytorch_lightning/trainer/connectors/logger_connector/result.py#L247-L266

[Nokimann]

Then, I wonder if we may set the loss with specific reduction methods during calculating loss.
How does Lightning know the reduction method?

The code you showed seems to read the reduction method. But. we may manually divide the loss with reduction='sum' by batch_size in training_step().

Is the reduction method recognized by a user-defined input? (the part after if self.meta.is_mean_reduction: ...)

[Nokimann]

Oh, I found the reduce_fx in the docs! And TorchMetrics too!
Thank you. All solved!

[adosar]

@rohitgr7 I think the docs are a little bit unclear on what happens under the hood.

reduce_fx: Reduction function over step values for end of epoch. Uses torch.mean() by default and is not applied when a torchmetrics.Metric is logged.

Nevertheless, in the provided piece of code torch.mean isn't used at all during compute(). Moreover, based on the docs, one would assume that Lightning does the following under the hood:

reported_loss = torch.mean(step_values_over_epoch) 

where step_values_over_epoch is just a list containing the logged loss values.

However, by inspecting the source code, I think Lightning does the following when reduce_fx='mean':

# Before epoch starts
self.value = 0
self.dataset_size = 0  # E.g. the number of images in a dataset

# On training step
self.value += step_value * batch_size  # Undoes the reduction from the criterion (e.g. MSELoss etc)
self.dataset_size += batch_size  # Correctly accounts for batch_size being different by DataLoader (e.g. when drop_last=False)

# On epoch end
reported_loss = self.value / self.dataset_size