Options to reduce home assistant and pfsense integration load

[markfrancisonly]

I'm not using most of the entities the pfsense integration generates, and I want to reduce the impact to my firewall. The fact is, I cannot adequately manage pfsense from home assistant and certainly don't want to recreate the pfsense dashboard in home assistant. I can see the appeal in having a summary snapshot card with cpu or bandwidth data, and the last few firewall entries, but primarily I need a method to automate tasks only pfsense can perform. Using the integration, this comes at a cost.

Currently the pfsense integration creates 380 entities in my home, 295 disabled and *85 enabled - simply too many.

How can the integration best be updated to reduce the number of entities generated? Are more filter options are needed in config setup? device_trackers entities have a config setup filter toggle, which I greatly appreciate since I'm running a unifi controller integration that provides the network-based device trackers I need.

Today, I'm only using the pfsense integration for WOL service... At peak usage, I've only used a single switch to toggle one firewall rule on/off. Due to this low usage and the huge number of entities, I uninstalled the integration today. Wow it felt great, but soon found that it was going to be a big headache to implement WOL broadcasting through my linux vlan/bonding/docker network infrastructure.... and pfsense can reliably send WOL packets.

and how do I reduce the data refresh load on my firewall? The data refresh interval can be set to a very large number but then the entity states would always be stale. Eliminating the update interval with configuration could work, but this would require a service to reload (update) data. Are there any other uses for having a manual data update service?

Curious what are others' thoughts

[travisghansen]

What load do the unused entities introduce to your system? What is load introduced on the firewall vs when the integration is fully disabled?

[markfrancisonly]

The frontend has a hass object that get's populated at load and refreshed periodically via websocket for cards to respond to entity state changes, but these are only active entity states.

Disabled entities do not load the system, but 'unavailable' entities are active entities.

also for me personally, I want to reduce the clutter and/or improve organization. I made the choice to move all entities into areas this weekend, to allow in-room behaviors. it was a chore even using csv files and python to make the updates. the effort left me with the feeling that fewer entities is better

[travisghansen]

A disabled entity should never be unavailable, so the number of unavailable entities should be extremely low (from this integration anyway).

Just so I understand you were trying to use some automatic tooling to set the rooms/areas vs using the UI? If you were using the UI aren't the disabled entries completely out of site for doing such a chore?

[markfrancisonly]

generally, yes the disabled entities only clog up the user entity table from a admin management perceptive, which isn't really a problem except with 380 entities it is difficult to see what the integration is providing. the entities are not grouped in any way other than by name, and the names are long and difficult to read.

As I mentioned, the pfsense integration is creating 85 pfsense active entities, and I'm no longer using any of them, but still like the idea of being able to toggle a rule on/off here in the future. Since I only use the WOL service to wake up my LG tv, and usually use my remote, I think I can live without that function for now

[travisghansen]

And disabling the 85 (or 84 of them) doesn’t solve your problem either?

To be clear I’m just trying to fully understand the root issue, not being critical of your desire/wish!

[markfrancisonly]

bumping my conclusion out of this deep thread...

[markfrancisonly]

just realized that I could select all to disable integration entities in ha. this and setting the scan interval to once a day, would be the easiest way to disable entity/refresh loading. after the scan interval any new firewall changes would create new entities and I'd need to repeat the process again.

in this case, if I wanted to toggle a firewall rule, I would first need to enable the switch, or use a custom component to enable the switch (such as ha-registry) and wait the interval for new data on the result, or does flipping a switch force a data refresh for that entity or all entities?

still feel like there is a missed opportunity for a manual data refresh service

[travisghansen]

I really like the idea of a service to trigger the refresh. From a very high level the integration has 2 'coordinators' which house the data:

• the primary coordinator for everything except device_tacker data
• the device_tracker coordinator

Each of them is on a distinct scan interval. During each scan interval each runs several api requests and stores all the results in a python dict / structure. The important part is that the data is considered whole only after a series of api requests..and those api requests always happen jointly during the same 'refresh'.

It really just depends on the method/service being called about whether the integration triggers a refresh upon action or not. For example if you disable/enable a pfsense service/daemon it will trigger a refresh of the data immediately after the action has succeeded. Others may not...I'd have to audit each to know for sure :( If you depend solely on the integration to trigger those updates then the data should remain relatively fresh, however if you make changes in pfsense directly the changes would not be reflected until a refresh happens.

[markfrancisonly]

Is there a clever way to create a generic integration service that executes an arbitrary function on pfsense? If folks keep requesting new services, the integration could easily end up with 20 or many more services, which would be undesirable both for a home assistant instance and future code maintenance.

a new generic service and manual data update service would do wonders to prevent indefinite scope expansion, minor requests would be answered without a new service declaration

[travisghansen]

Those already exist in exec_command and exec_php. Neither of them help tell the integration to refresh its data though.

[markfrancisonly]

for my use case, I would prefer to have only needed entities generated, and refresh only a targeted entity data, but I can see how someone else may want the integration to resync config and pull down the whole pfsense state in some cases.

these days, my pfsense 2.6 dashboard takes a full 5 second delay to load in the browser, it's super annoying. while I haven't figured out the problem yet, presumably pfsense is waiting on one of the dashboard widgets to refresh data. pulling too much data at once is an anti-pattern

[travisghansen]

It’s a balancing act between more requests and more fine-grained data retrieval. For the vast majority of sensors I developed it in such a way as to gather all that data in a single api request (precisely for the reason of keeping the number of requests down).

Pulling too little data has its drawbacks as well.