Multithreading for TNeuralDataLoadingFit.FitLoading

[gwiesenekker]

I see from the HypotenuseFitLoading example that GetTrainingPair supports a ThreadId argument. How do you add multi-threading support to the HypotenuseFitLoading example?

[gwiesenekker]

I have compiled the HypotenuseFitLoading example and the output ('Threads') suggests that multithreading is enabled and it looks like 'batchsize' controls the number of threads. This is not the common meaning of 'batchsize' in the context of NN training. How are the 'batchsize' and the number of threads related?

[joaopauloschuler]

@gwiesenekker , regarding "I see from the HypotenuseFitLoading example that GetTrainingPair supports a ThreadId argument. How do you add multi-threading support to the HypotenuseFitLoading example?", this example already explores parallel processing. The class TNeuralDataLoadingFit uses https://github.com/joaopauloschuler/neural-api/blob/master/neural/neuralthread.pas . The parameter ThreadId is there just in case that you need to make sure that your data each thread has some specific code (such as writes instead of pure data loading).

[joaopauloschuler]

In this API, the batch is distributed to parallel threads using the principle of data parallelism: https://www.telesens.co/2017/12/25/understanding-data-parallelism-in-machine-learning/ .

I tested this API with up to 64 CPU cores with 64 parallel threads. It should support more threads but I tested with up to 64 CPU cores. In this API, your batch size should be bigger than the number of cores.

[gwiesenekker]

Thanks. So where do you specify the number of threads? Looking at the source code it looks like the default number of threads is GetSystemThreadCount, correct? Why does changing 'batch size' in HypotenuseFitLoading to 8 change 'Threads' in the output to 8? Perhaps because the 'batch size' is 1 (one X, Y pair)?

[joaopauloschuler]

"Looking at the source code it looks like the default number of threads is GetSystemThreadCount, correct?"
In FPC, this is correct. In Delphi, TThread.ProcessorCount is used.

"Why does changing 'batch size' in HypotenuseFitLoading to 8 change 'Threads' in the output to 8? Perhaps because the 'batch size' is 1 (one X, Y pair)?"

This is a limitation in CAI. The number of threads needs to be equal or smaller than the batch size. In this API, larger batch sizes lead to better hardware usage (more efficient). Larger batch sizes tend to lead to smaller over fitting but may also make the learning harder. My own default value for problems that I have never touched before is 64 elements for batch size. I recommend 64 as a starting point. Some papers report batch sizes of up to 256 and even 512.

There are some experiments with batch size in this link: https://medium.com/mini-distill/effect-of-batch-size-on-training-dynamics-21c14f7a716e .

In this API, just to start "feeling" a new problem, I would try 64 for batch size, 0,001 or smaller for LR and 0.9 for inertia.