Add a new experimental restart policy for large scale model training

Add a new experimental restart policy for large scale model training #3135