Default batch transfer behavior

[ziw-liu]

For CPU to CUDA batch transfer, when the batch is on host pinned memory, is the default trainer behavior set to non-blocking transfer automatically? Or do the users need to override the batch transfer hook for it to be non-blocking?

[ziw-liu]

It reads like it does:

pytorch-lightning/src/lightning/fabric/utilities/apply_func.py

Lines 98 to 108 in a944e77

	def batch_to(data: Any) -> Any:
	kwargs = {}
	# Don't issue non-blocking transfers to CPU
	# Same with MPS due to a race condition bug: https://github.com/pytorch/pytorch/issues/83015
	if isinstance(data, Tensor) and isinstance(device, torch.device) and device.type not in _BLOCKING_DEVICE_TYPES:
	kwargs["non_blocking"] = True
	data_output = data.to(device, **kwargs)
	if data_output is not None:
	return data_output
	# user wrongly implemented the `_TransferableDataType` and forgot to return `self`.
	return data