data-importer account caching not working ?

[MacPaille]

Today while looking at the logs for the csv import issue I did notice the DI is polling all the accounts from FF instance for each imported account.
I remember in the past I did set the env variable USE_CACHE to true but it seems the account caching is not working.
As I'm importing 13 accounts at each run of my instance every day it just takes a huge amount of time fetching the same list every time...
It seems it's also querying directly some names, not sure exactly why but I don't know all the logic in place :-)

DI logs

[2024-11-28 20:05:04] production.DEBUG: Start of shared import routine.  
[2024-11-28 20:05:04] production.DEBUG: First collect account information from Firefly III.  
[2024-11-28 20:05:04] production.DEBUG: Now in collectAccountTypes()  
[2024-11-28 20:05:04] production.DEBUG: No base url in getBaseUrl() session, will return config variable.  
[2024-11-28 20:05:04] production.DEBUG: No access token in hasAccessToken() session, will return config variable.  
[2024-11-28 20:07:41] production.DEBUG: Collected 2802 account(s) in collectAccountTypes()  
[2024-11-28 20:07:41] production.DEBUG: Now starting submission by calling API Submitter  

FF logs

  [28/Nov/2024:20:05:00 +0100] "POST /api/v1/transactions HTTP/1.1" 422 1026 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:05:01 +0100] "GET /api/v1/accounts/1664 HTTP/1.1" 200 1803 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:05:01 +0100] "GET /api/v1/currencies/default HTTP/1.1" 200 1236 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:05:01 +0100] "GET /api/v1/search/accounts?query=LA+MIE+DU+JOUR&field=name&page=1 HTTP/1.1" 200 3694 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:05:02 +0100] "GET /api/v1/search/accounts?query=LA+MIE+DU+JOUR&field=name&page=1 HTTP/1.1" 200 3694 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:05:02 +0100] "GET /api/v1/search/accounts?query=LA+MIE+DU+JOUR&field=name&page=1 HTTP/1.1" 200 3694 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:05:02 +0100] "GET /api/v1/search/accounts?query=1714&field=id&page=1 HTTP/1.1" 200 1942 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:05:02 +0100] "GET /api/v1/search/accounts?query=LA+MIE+DU+JOUR&field=name&page=1 HTTP/1.1" 200 3694 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:05:03 +0100] "GET /api/v1/search/accounts?query=LA+MIE+DU+JOUR&field=name&page=1 HTTP/1.1" 200 3694 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:05:03 +0100] "GET /api/v1/search/accounts?query=LA+MIE+DU+JOUR&field=name&page=1 HTTP/1.1" 200 3694 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:05:03 +0100] "GET /api/v1/search/accounts?query=BOULANGERIE+LOUISE+GENAPPE&field=name&page=1 HTTP/1.1" 200 1948 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:05:03 +0100] "GET /api/v1/search/accounts?query=BOULANGERIE+LOUISE+GENAPPE&field=name&page=1 HTTP/1.1" 200 1948 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:05:03 +0100] "GET /api/v1/search/accounts?query=2065&field=id&page=1 HTTP/1.1" 200 1945 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:05:04 +0100] "GET /api/v1/accounts?type=all&page=1&limit=250 HTTP/1.1" 200 220244 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:05:22 +0100] "GET /api/v1/accounts?type=all&page=2&limit=250 HTTP/1.1" 200 220695 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:05:38 +0100] "GET /api/v1/accounts?type=all&page=3&limit=250 HTTP/1.1" 200 220641 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:05:52 +0100] "GET /api/v1/accounts?type=all&page=4&limit=250 HTTP/1.1" 200 219970 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:06:08 +0100] "GET /api/v1/accounts?type=all&page=5&limit=250 HTTP/1.1" 200 220442 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:06:22 +0100] "GET /api/v1/accounts?type=all&page=6&limit=250 HTTP/1.1" 200 220677 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:06:37 +0100] "GET /api/v1/accounts?type=all&page=7&limit=250 HTTP/1.1" 200 220737 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:06:50 +0100] "GET /api/v1/accounts?type=all&page=8&limit=250 HTTP/1.1" 200 220513 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:07:02 +0100] "GET /api/v1/accounts?type=all&page=9&limit=250 HTTP/1.1" 200 221003 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:07:14 +0100] "GET /api/v1/accounts?type=all&page=10&limit=250 HTTP/1.1" 200 220258 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:07:26 +0100] "GET /api/v1/accounts?type=all&page=11&limit=250 HTTP/1.1" 200 220217 "-" "GuzzleHttp/7"
  [28/Nov/2024:20:07:38 +0100] "GET /api/v1/accounts?type=all&page=12&limit=250 HTTP/1.1" 200 48500 "-" "GuzzleHttp/7"

Has this cache variable changed name or does it work along another variable to be set too?

[JC5]

That particular piece of code doesn't use that variable, so no surprises there. But unfortunately, the Docker container has no cache volume so it would be pointless to use it.

But I'm surprised it takes this long. Can you tell me more? Are you importing from GoCardless? CSV?

[MacPaille]

A mix of both. I'm importing 4 accounts from GoCardLess and 9 from CSV. CSVs do not move a lot maybe once a week. The real important with our daily transactions are used with GCL.
Now it could be the performances of my container but can't really say what is the bottleneck to fetch these 250 accounts pages... I don't see any CPU/RAM blocking while running

Any plans to optimize this? I mean it's only fetching a list of accounts to be stored on a temporary text file haha. Or maybe we would need a redis container alongside it?

[JC5]

I've added some experimental caching in the latest develop build, but I've yet to see how it works.

[MacPaille]

Cool very good new I'll update the instance to this new version and check how it works 👍

[MacPaille]

From my testing the caching seems working fine. Takes some minutes to fetch all the accounts at the first account import, but then all the others that steps is completely skipped. This reduces the import time for me quite drastically. Thanks!
Not that time matters as it runs automatically during the night but still nice optimization!