Equivalent to Dropout

[Dzandaa]

Hello,

Is there and equivalent to "dropout" in CAI?

Thank you.

B->

[joaopauloschuler]

Hello!

Have you tried TNNetDropout.Create(Rate: double; OneMaskPerbatch: integer = 1); ?

Glad to help.

[Dzandaa]

Hi,
Thank you, I missed it :)

Rate is from 0 to 100, right?

B->

[Dzandaa]

Hi again,

I have a directory with 8 sub-directories containing images (64x64) of faces expressions.

I try to train this network:

`
VGG13NN := TNNet.Create;
// VGG13 Custom Net
VGG13NN.AddLayer([
TNNetInput.Create(64, 64, 3),

TNNetConvolutionReLU.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1),
TNNetConvolutionReLU.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1),
TNNetMaxPool.Create(2),
// TNNetDropout.Create(25, 1),

TNNetConvolutionReLU.Create({Features=}128, {FeatureSize=}3, {Padding=}1, {Stride=}1),
TNNetConvolutionReLU.Create({Features=}128, {FeatureSize=}3, {Padding=}1, {Stride=}1),
TNNetMaxPool.Create(2),
// TNNetDropout.Create(25, 1),

TNNetConvolutionReLU.Create({Features=}256, {FeatureSize=}3, {Padding=}1, {Stride=}1),
TNNetConvolutionReLU.Create({Features=}256, {FeatureSize=}3, {Padding=}1, {Stride=}1),
TNNetConvolutionReLU.Create({Features=}256, {FeatureSize=}3, {Padding=}1, {Stride=}1),
TNNetMaxPool.Create(2),
// TNNetDropout.Create(25, 1),

TNNetConvolutionReLU.Create({Features=}256, {FeatureSize=}3, {Padding=}1, {Stride=}1),
TNNetConvolutionReLU.Create({Features=}256, {FeatureSize=}3, {Padding=}1, {Stride=}1),
TNNetConvolutionReLU.Create({Features=}256, {FeatureSize=}3, {Padding=}1, {Stride=}1),
TNNetMaxPool.Create(2),
// TNNetDropout.Create(25, 1),

TNNetFullConnectReLU.Create(1024),
// TNNetDropout.Create(50, 1),
TNNetFullConnectReLU.Create(1024),
// TNNetDropout.Create(50, 1),

TNNetLayerFullConnect.Create(8),
TNNetSoftMax.Create()
]);

VGG13NN.DebugStructure();
VGG13NN.SetLearningRate(0.001,0.9);
VGG13NN.SetL2Decay(0.0);
`
I load the images in volumes with this code:

`
LoadOK := False;
if(SelectPicturesDirectory.Execute) then
begin
// change ProportionToLoad to a smaller number if you don't have available 16GB of RAM.
ProportionToLoad := 0.1;

MLog.Append('Loading ' + Round(ProportionToLoad*100).ToString + ' % of The Faces Dataset into memory.');
CreateVolumesFromImagesFromFolder
(
 ImgTrainingVolumes, ImgValidationVolumes, ImgTestVolumes,
 {FolderName=}SelectPicturesDirectory.FileName, {pImageSubFolder=}'',
 {color_encoding=}0{RGB},
 {TrainingProp=}0.9*ProportionToLoad,
 {ValidationProp=}0.05*ProportionToLoad,
 {TestProp=}0.05*ProportionToLoad,
 {NewSizeX=}64, {NewSizeY=}64
);

MLog.Append('Training Images: ' +  ImgTrainingVolumes.Count.ToString);
MLog.Append('Validation Images: ' +  ImgValidationVolumes.Count.ToString);
MLog.Append('Test Images: ' +  ImgTestVolumes.Count.ToString);

LoadOK := True;

end;
`
It seems O.K.

And I train with this code:

`
VGG13Fit := TNeuralImageFit.Create();
MLog.Append('Computing...');

VGG13Fit.OnAfterEpoch := @Self.OnAfterEpoch;
VGG13Fit.OnAfterStep := @Self.OnAfterStep;
VGG13Fit.OnStart := @Self.OnStart;
VGG13Fit.FileNameBase := 'ExpressionDetection';
VGG13Fit.InitialLearningRate := 0.001;
VGG13Fit.LearningRateDecay := 0.01;
VGG13Fit.StaircaseEpochs := 10;
VGG13Fit.Inertia := 0.9;
VGG13Fit.L2Decay := 0.00001;

if HasOpenCL then
begin
VGG13Fit.EnableOpenCL(EasyOpenCL.PlatformIds[0], EasyOpenCL.Devices[0]);
VGG13NN.EnableOpenCL(EasyOpenCL.PlatformIds[0], EasyOpenCL.Devices[0]);
end;
IsFitting := True;

VGG13Fit.Fit(VGG13NN, ImgTrainingVolumes, ImgValidationVolumes, ImgTestVolumes, {NumClasses=}8, {batchsize=}SEBatch.Value, {epochs=}SEEpoch.Value);

IsFitting := False;
MLog.Append('Done');
`

Nothing happens, Never go to OnStart, OnAfterStep, OnAfterEpoch
But the program use more and more memory.

The test use 1332 images.

Any Idea?

Thank you.

B->

[joaopauloschuler]

Just for debugging, try to add VGG13Fit.MaxThreadNum := 1; before calling VGG13Fit and check memory usage. You'll have one full memory structure for each parallel thread.

[joaopauloschuler]

I would also recommend giving a try to this model:

      NN.AddLayer([
      TNNetInput.Create(64, 64, 3),
      TNNetConvolutionLinear.Create({Features=}64, {FeatureSize=}5, {Padding=}4, {Stride=}1),
      TNNetMaxPool.Create(2),
      TNNetMovingStdNormalization.Create(),
      TNNetConvolutionReLU.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1),
      TNNetConvolutionReLU.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1),
      TNNetMaxPool.Create(2),
      TNNetConvolutionReLU.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1),
      TNNetConvolutionReLU.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1),
      TNNetConvolutionReLU.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}2),
      TNNetDropout.Create(0.5),
      TNNetMaxPool.Create(2),
      TNNetFullConnectLinear.Create(8),
      TNNetSoftMax.Create()
    ]);

This a model that I used in a number of experiments at:
https://github.com/joaopauloschuler/pre-trained-neural-api-networks/

[joaopauloschuler]

You can increase MaxThreadNum bit by bit checking memory usage, if you like the idea.

[Dzandaa]

VGG13 Net.pdf

This is the model I tried to reproduce.

B->

[joaopauloschuler]

@Dzandaa , I'm curious if you had a chance to test with smaller number of threads and / or the other simpler model above.

[Dzandaa]

Hi,

Still testing :)

When you convert RGB images to TNeuralFloat, what is the range of NeuralFloat?

RGB = 0..255 NeuralFloat =?

Thank you.

B->

[Dzandaa]

Sorry, I didn't see that there is a NeuronalInputToRgbImg function :)

I've go some good results using your model.

I think mine is to big for my computer and graphic card.

To be continued.

B->

[joaopauloschuler]

@Dzandaa , absolute many thanks for sharing this beautiful screenshot!!!

In the case that you haven't looked at, I highly recommend having a look at:
https://github.com/joaopauloschuler/pre-trained-neural-api-networks/

In the case that you would like to try a more complex model, you could start with:

      TNNetInput.Create(64, 64, 3),
      TNNetConvolutionLinear.Create({Features=}64, {FeatureSize=}5, {Padding=}4, {Stride=}1),
      TNNetMaxPool.Create(2),
      TNNetMovingStdNormalization.Create(),

Then, you could continue with a "RESNET" style:
https://github.com/joaopauloschuler/neural-api/blob/master/examples/ResNet/CaiResNet20.lpr
https://github.com/joaopauloschuler/neural-api/tree/master/examples/ResNet

For using the trained NN in production, you can follow the example from (if you like):
https://github.com/joaopauloschuler/pre-trained-neural-api-networks/

It would be:

  procedure ClassifyOneImageSimple;
  var
    NN: TNNet;
    ImageFileName: string;
    NeuralFit: TNeuralImageFit;
  begin
    WriteLn('Loading Neural Network...');
    NN := TNNet.Create;
    NN.LoadFromFile('SimplePlantLeafDisease-20230720.nn');
    NeuralFit := TNeuralImageFit.Create;
    ImageFileName := 'plant/Apple___Black_rot/image (1).JPG';
    WriteLn('Processing image: ', ImageFileName);
    WriteLn(
      'The class of the image is: ',
      NeuralFit.ClassifyImageFromFile(NN, ImageFileName)
    );
    NeuralFit.Free;
    NN.Free;
  end;  

[Dzandaa]

Hi,
Thank you for the links.

I've already tested "Plant leaf Disease" and "Colorectal"

What I try to do with CAI, is to detect Atoms in a Spectrum

The input is a TSpectrum of 1024 values

type TSpectrum = array[0..1023] of TNeuralFloat;

And the output is the detected atom(s)
In this case I try to identify only 5 Atoms.

this is the model:

`NeuralNet.AddLayer([

TNNetInput.Create(1024),

TNNetConvolutionReLU.Create(32, 3, 1, 1, 1),

TNNetMaxPool.Create(2, 2, 0),

TNNetConvolutionReLU.Create(32, 3, 1, 1, 1),

TNNetMaxPool.Create(2, 2, 0),

TNNetConvolutionReLU.Create(5, 3, 1, 1, 1),

TNNetFullConnectLinear.Create(5)

]); `

Of course the Spectrum is just a simulation with some peaks :)

B->