Increase default thread number?

[gustrd]

I just had some tests and I was able to massively increase the speed of generation by increasing the threads number.

I have an i7-12700H, with 14 cores and 20 logical processors. So by the rule (of logical processors / 2 - 1) I was not using 5 physical cores.

Also the number of threads seems to increase massively the speed of BLAS when using CLBLAST. My GPU don't get 100% processing utilization at the default thread number.

Would not be better to instead of limiting the thread number bellow the logical processors number, just decreasing the process priority, using "os.nice" and "psutil"?

Or could I define a different thread number, and default thread count, for BLAS than for the token generation?

Do you have some ideias of tests I can make to enhance this feature?

[LostRuins]

We already have an option for that, its called --psutil_set_threads and it does use the physical core count instead. But on some systems this is not supported, so it cannot be the default. Also some systems tend to work faster with fewer threads depending on memory speed.

[gustrd]

Thank you, it makes sense.

[LostRuins]

Likewise, it appears that os.nice() is non portable and does not work on windows or OSX.

[gustrd]

Yes, at then you need to use psutil. But I don't think anymore that's the issue.

I did additional research and the token generation has no additional performance gain at thread number higher than the physical cores, but it seems that clblast does.

I could not find yet where clblast thread number is defined, maybe it can benefit of a higher thread count?