Proxy inventory of park above 20 thousand computers

[erique-souza]

Bug reporting acknowledgment

Yes, I read it

Professional support

Still not applicable

Describe the bug

I have a scenario with a park of approximately 20 thousand assets that have glpi-agent installed, these agents are installed and pointing to a proxy server using the glpi-agent native proxy plugin, but from what we noticed after some analysis it is that even with a relatively high inventory frequency, which is currently 168 hours (7 days), there are a lot of requests in a short space of time, I believe that as a result the proxy's ssl service ends up falling, that is, becoming inoperative

As the proxy plugin does not have a more extensive configuration module as is the case with conventional web services such as apache, ngnix, etc.. I tested some configurations of the proxy plugin itself to try to minimize requests as well as increase the inventory frequency. , using the documentation at https://glpi-agent.readthedocs.io/en/latest/plugins/proxy-server-plugin.html as a basis

Also change the maxrate together with maxrate_period to a very high number
Change it so that instead of being the active proxy that receives the inventory and then forwards it, it is passive that receives the inventory and stores it internally in a folder, using the only_local_store = yes and local_store = /inventario function

And even testing for very high numbers max_proxy_threads = 10000 and max_pass_through = 10000

However, it seems that none of these measures have yet been able to solve the problem we are facing, which is: The Proxy Server Going Offline in its web version, on port 443, with SSL everything correctly active

Doing this when an agent is trying to communicate, many times, the vast majority of the time they are unable to do their inventory

I am attaching the log from the proxy side below, because in the proxy the requests are registered and some manage to execute as expected, which are the few that manage to communicate and record their file, and the vast majority end up being rejected from their inventory, with the error messages

[ssl server plugin] HTTPD can't start SSL session: Failed to upgrade socket to SSL: SSL wants a read first
[http server] HTTPD can't start SSL session

I am also sending below the proxy settings and also the ssl configuration on the proxy for analysis

glpi-agent.log

proxy_server_conf.txt

ssl_conf.txt

To reproduce

• Having a park with many glpi-agent requests pointing to the same proxy
• Check the inventory if it is carried out, with default settings or changeable as needed
• log monitoring

Expected behavior

Device inventory locally on the proxy

Operating system

Linux

GLPI Agent version

v1.11

GLPI version

Not applicable

GLPIInventory plugin or other plugin version

Not applicable

Additional context

No response

[g-bougard]

Hello @erique-souza

first of all, I see in your issue more a support request than a real agent issue. You're speaking about 20000 agents trying to submit requests to one proxy agent. This is definitively not a context for which this feature was developed. I see also few big mistakes in the proxy configuration showing you didn't understood few basic principles.

For example, you changed max_pass_through to 10000 and this has no sens: it means you permited to chain until 10000 glpi-agent proxies which seems very far from your real configuration. The default limitation of 5 is here to prevent wrong configuration in the case you not intentionally chained together 2 glpi-agent proxies in a loop. With your configuration, this case will result into a loop of 5000 requests between the 2 chain-together agents. Also setting max_proxy_threads to 10000 will probably exhaust your OS resources until make the proxy agent unresponsive.

So I'll first move your issue as a discussion before fully continuing.

[g-bougard]

Of course, when targeting such a heavy park, you should definitively need a professional support. If you don't have it, please try to earn a GLPI subscription.

About max_proxy_threads, it must be coherent with the installed CPU number. It must be never over 2 times the number of CPU.

If you want to maximize the throughput, with such a number of agent requests, you should better:

• or check if another solution could also do the job, like a reverse proxy other Apache
• or create more glpi-agent proxies and dispatch the load other them using a load-balancing system

[erique-souza]

Thanks for the guidance

The configuration numbers were test numbers, previously I was using numbers close to the standard

As on the agents I was receiving a lot of 500 errors when trying to communicate with the proxy, one of the first tests I carried out was to increase the maxrate and maxrate_period, as far as I understood based on the documentation on the website the maxrate was the limit of requests per IP that could happen from the same agent to the proxy server in a certain period and maxrate_period defined this period in seconds. The idea by increasing the maxrate was not to prohibit agents from contacting the proxy so as not to return error 500 and to allow local inventory on the proxy, but I did not see this expected effect

Now about max_proxy_threads I ran the "lscpu" command to check the number of cores per socket, in this case it is 2, so the maximum that could be used in the max_proxy_threads configuration is 4, correct? If the default is 10 in the agent then it means that I need more cores to process the amount of load that the park is currently requesting, I'm just describing it "out loud" to establish the concepts and also record the line of reasoning

I made these changes to adjust it correctly, as advised, this below is the result of the log after the changes

glpi-agent.log

Current proxy configuration file

config_proxy.txt

My goal is to understand where the problem is with the web service being unavailable from the proxy, and also allow agents to be able to contact the proxy to carry out their inventories without presenting return errors in the agent itself

This server is a virtualized server, if one of the problems, for example, is the number of sockets/processors/cores, we can evaluate whether we can increase this capacity to improve processing performance

If there is something else I can work on on the glpi-agent side, proxy file and glpi-agent sll file I can be carrying out tests to improve the agent system as a whole

[erique-souza]

In fact it may be that I have interpreted the cpu number incorrectly, there is a lot of information on the screen
The number of CPUs is 4 from what I'm observing, but each CPU has 2 cores, this would make it have 8 cores in total, so the configuration within max_proxy_threads could be 16, is my line of reasoning correct?

[g-bougard]

This is not correct: you have 2 sockets with 2 core each, so 4 CPUs. At that level, 8 or 10 is still correct for the max_proxy_threads. My rule is not absolute, this is just an advice. 10 is far more appropriate than 10000, or even 100.

Anyway, here, I still can say you that computer has not enough resource for the expected workload. You should probably allocate more cpu if you can.

About maxrate, you have to update it if the proxy answers too often with a "429 Too many requests" errors. In your case, you have 500 errors with SSL connection reset. Maybe your SSL support on the proxy or in agents is not well configured.

As I don't see any evidence of the glpi-agent version in your proxy log, be sure your proxy agent is up-to-date, there was an issue fixed in v1.10 related to proxy-server-plugin & ssl-server-plugin support.

[erique-souza]

Thanks for the feedback

I then adjusted the max_proxy_threads and max_pass_throught to the default 10 per hour, until I was able to increase the capacity of this server

Regarding the agent load, even though there are approximately 20 thousand devices with agents, theoretically they all communicate at different times due to the nature of the agent of only communicating every x amount of time, which is the defined inventory frequency. Regarding this, I wanted to confirm if the process below is correct:

• The Agent has been installed, if the runnow configuration is enabled, it will try to communicate with the server indicated after installation, otherwise, it will use the internal delaytime configuration, which by default is 3600, 1h
• If the agent is unable to contact at this first moment, the default is to try to contact again every 1h, which is 3600 seconds, and if it is unable to do so again, it repeats this process until it succeeds
• If it is able to contact, the delaytime does nothing else and starts to assume the inventory frequency stipulated by the server that is receiving the inventory (proxy or glpi-server)

Is this process I described above correct?

I'm asking because I would like to optimize the settings when installing the agents based on this inventory time information. Since I'm talking about a relatively large computer park, perhaps a longer delay time would help to avoid having too many requests in short periods of time, when installing the agent on, for example, 1000 new computers.

I believe that another relevant setting for this situation of a large park would be to remove runnow, precisely so as not to force the inventory after installation, and wait for that first hour of the 3600 seconds before the first attempt. Depending on how the installation method is being applied, it may be forcing the inventory before the time.

Is there any other setting that I'm not seeing, which could be passed via agent installation parameter, that could help in these situations of larger parks?

Would the error in question mentioned and corrected in 1.10 be this? fd43789#diff-bbd4b6a86bc65b6ac8e79e97afc61499158edb40f7bb404c70637b46d80a7ad2R15

[g-bougard]

Hi @erique-souza

your assertions are not all correct.

If runnow option is disabled, agent will wait by default between 30 minutes to 1 hour before trying to contact the server. The default is from the half of the value of delaytime option to the value itself. That first delay is random in that interval to avoid too many computers installed or starting at the same time to overflow the server with requests. After a successful contact with the server, agent will used delay configured in GLPI, so 7 days in your case if I understand well. In case of network error, agent will try again 1 minute after doubling the delay after each error, 2 minutes, than 4 minutes, 8m, 16m, ... until reaching the known error maxdelay which is initially setup by the value of delaytime.

In your context, you may consider to setup delaytime to half of a day setting it to 43200. In that case, disabling runnow will tell the agent to attempt a first connection between 6h to 12h and in case of error, the delay between each attempt will grow until 12h reducing the load on the server.

An important point about delaytime is it will be used again after a computer restart if the expected next server contact has passed: with a 7 days expiration set in server, this means the computer has to be shutdown at least 7 days. You have to consider this point if computers are not always up: for desktop computers only up 8 hours a day for example, you may have the delaytime initial mechanism enabled again each 7 days. And this may involve side-effects.

With 20000 computers in the park without errors and always up, with a 7 days expiration, you can expect at a long term to have one agent contact every 30 seconds (86400*7/20000). If agent can handle 10 requests by second, this is finally very large and safe. Indeed, even the default of one day should be safe, but you may need a large delaytime to avoid 20000 computers trying to contact the server if they all have been shutdown during the night. In that case, theoretically, you may have a maximum of 20000/3600 = 30 requests by second, but this can be higher if agents starts to receive errors as they will retry to connect soon.

In my point of view, you should configure agents with at least 3 or 6h (so 16200 or 32400) as delaytime. You can leave expiration to 7 days and start to reduce it to 4, than 2, than 1 day after everything is stabilized, leaving at least a week between each reduction to verify this is stable.

There may be some other settings to tune, but they depend on your installation process and even OS. But don't worry about them, focus on what you are aware now.

About the fixed error in v1.10, yes, this is about that one. Eventually be sure to use the most recent OS you can to host proxy agent to lower the risk of issues related to used libraries.

[erique-souza]

@g-bougard Still on the same subject, in the latest versions of the agent 1.11, for example, does the agent now send partial inventory by default?, if there have been changes since your last inventory?

I saw some agents reporting newer versions like this, but I was unsure if it was a new feature, as I didn't see any specific announcement on the topic, but it could be my fault for not having seen something like that, could you advise me if this really exists? today

[erique-souza]

I'm asking this, because in this specific park we discovered another aggravating factor for this situation that I reported at the beginning of the topic, the systems inventoried are Linux OpenSuse which, being Linux, is not a problem in itself, but the size of the information generated in the .json files for the proxy yes, I was checking the size of information transmitted to a .json file on average practically any Windows has a size between 100kb and 200kb on average, now Linux specifically from this park which has OpenSuse as a base, they have an average size of 1.2 MB is then 6x heavier than conventional Windows

If the agents in their newer versions already efficiently send partial data, without using a separate command line, it would help in these inventory processes of very large quantities of inventories

[g-bougard]

Hello @erique-souza

that's funny, I just found you commented in this announcement discussion: #669

I said there:

The full-inventory-postpone new feature is available for all.

Of course, this doesn't say it is enabled by default. But this is said in the blog post, linked in the same announcement. You missed it ;-)

So the feature is available since 1.8 and also enabled by default since 1.8.

Regarding the size of windows inventory vs linux inventory, I can say few things:

• on windows and on linux, software inventory is one of the biggest inventory part or category. So if no software was installed, uninstalled or updated between 2 inventories, this part should not appear in a partial inventory and drastically reduce the inventory size.
• on linux only, there's a category named "process" which reports current process list. It can be really big and as you can imagine it always changes so it is always reported in inventory even in partial. In GLPI 10 (this will change in GLPI 11 as you'll be able to see that process list), this category is indeed not analyzed (it will be in GLPI 11).

Indeed when using GLPI 10, you still can reduce linux inventory size by setting no-category = process. This will also work with GLPI 11, but maybe you'll also want to check the process list.

[erique-souza]

Hello @erique-souza

that's funny, I just found you commented in this announcement discussion: #669

I said there:

The full-inventory-postpone new feature is available for all.

Of course, this doesn't say it is enabled by default. But this is said in the blog post, linked in the same announcement. You missed it ;-)

So the feature is available since 1.8 and also enabled by default since 1.8.

Regarding the size of windows inventory vs linux inventory, I can say few things:

• on windows and on linux, software inventory is one of the biggest inventory part or category. So if no software was installed, uninstalled or updated between 2 inventories, this part should not appear in a partial inventory and drastically reduce the inventory size.
• on linux only, there's a category named "process" which reports current process list. It can be really big and as you can imagine it always changes so it is always reported in inventory even in partial. In GLPI 10 (this will change in GLPI 11 as you'll be able to see that process list), this category is indeed not analyzed (it will be in GLPI 11).

Indeed when using GLPI 10, you still can reduce linux inventory size by setting no-category = process. This will also work with GLPI 11, but maybe you'll also want to check the process list.

It's been a while since I remembered this post specifically, but in any case at the time I didn't really open the content separately, my fault, I let it go

Now that I got the hang of this file size reduction, I did some tests that I won't delve into here as it's not the topic about it, but I opened an issue and an idea about the topic in parallel after some tests
#792
#793

Indeed, after doing some tests, I checked some conclusions, the partial inventory process is fantastic, it greatly optimizes the inventory process itself

I tested it in conjunction with the tip you gave about the processes adding the parameter for not inventorying it, and I did some tests on a Linux Ubuntu

• On Ubuntu Full Default Inventory: 600kb +/- file size

• In Ubuntu Partial Inventory keeping processes: 75kb +/-

• In Ubuntu Partial Inventory, removing processes: 2kb +/-

So in the scenario that I described at the beginning of this topic, updating the agents from the current version of 1.5 to any version 1.8+ will certainly result in a significant improvement in the processing of this data, since the mass of the data is much smaller than I initially described

So once again, thank you for the guidance, these inclusions and recommendations were certainly very useful.

It will still take some time to be able to make all the changes we discussed above, upgrading the amount of processing + new proxy for segregation + updating the agent with new, more optimized parameters, but now we have a better path to follow than at the beginning of this topic.