[Bug] Resuming training on a pretraining loop does not continue data loading from where it left off

[NanoCode012]

Please check that this issue hasn't been reported before.

• I searched previous Bug Reports didn't find any similar reports.

Expected Behavior

Loads the dataset, goes to the last processed sample from prior training, and continues training.

Current behaviour

Loads the dataset as usual, does not resume prior point in training in data loader, and continues training

Steps to reproduce

Start training, stop, resume.

Config yaml

No response

Possible solution

We need to save the dataloader's point and resume that when we resume training.

https://huggingface.co/docs/datasets/v3.2.0/stream#save-a-dataset-checkpoint-and-resume-iteration

Which Operating Systems are you using?

• Linux
• macOS
• Windows

Python Version

3.11

axolotl branch-commit

main

Acknowledgements

• My issue title is concise, descriptive, and in title casing.
• I have searched the existing issues to make sure this bug has not been reported yet.
• I am using the latest version of axolotl.
• I have provided enough information for the maintainers to reproduce and diagnose the issue.