Compressed model creates "Out of memory" error but the uncompressed model does not

[krishnapitike]

Hello,

I have trained a V-H potential with 3 layers: training net=240x240x240 and embedding net=50x100x200.

The uncompressed frozen model is about 15 MB size. However the compressed model is 550 MB size.

When I run a lammps simulation using a 1024 atoms model using the compressed model, then I get a "Out of memory" error. However, when I run the same simulation with an uncompressed model the simulation runs just fine. Is my assessment correct that the "Out of memory" is caused by large size of compressed model?

In another instance of training, with same training parameters, the compressed model is only 150 MB where the simulation runs without any error.

Could you please explain why the same training parameters may lead to different size compressed models?

[njzjz]

How much memory do you have? What is your training parameter?

[krishnapitike]

I have 512 GB memory and the training input file is as follows:

{
"_comment": " model parameters",
"model": {
"type_map": ["V","H"],
"descriptor" :{
"type": "se_e2_a",
"sel": "auto",
"rcut_smth": 0.50,
"rcut": 6.40,
"neuron": [25, 50, 100, 200],
"resnet_dt": false,
"axis_neuron": 16,
"seed": 14229,
"set_davg_zero": true,
"_comment": " end descriptor"
},
"fitting_net" : {
"neuron": [240, 240, 240, 240],
"resnet_dt": true,
"seed": 12991,
"atom_ener": [-0.02846697, -0.02953589],
"_comment": " end fitting_net"
},
"_comment": " end model"
},

"learning_rate"     :{
    "type":                     "exp",
    "decay_steps":      5000,
    "start_lr":                 0.001,
    "stop_lr":                  3.51e-8,
    "_comment":                 " end learning_rate"
},

"loss" :{
    "type":                     "ener",
    "start_pref_e":     0.02,
    "limit_pref_e":         1,
    "start_pref_f":         1000,
    "limit_pref_f":         1,
    "start_pref_v":         0.2,
    "limit_pref_v":         1,
    "_comment":                 " end loss"
},

[njzjz]

Do you mean you use the CPU cores and the CPU memory is out of memory?

[krishnapitike]

Yes

[njzjz]

Do you use any MPI ranks? It's a bit confusing to me that 512GB is very large for any models.