Avoid random failures when deleting environment

Limactl is racy, trying to access files in other clusters directories
and failing when files were deleted. Until this issue is fixed in lima,
ensure that only single vm can be deleted at the same time.

Example failure:

    % drenv delete envs/regional-dr.yaml
    2024-09-13 05:59:57,159 INFO    [rdr] Deleting environment
    2024-09-13 05:59:57,169 INFO    [dr1] Deleting lima cluster
    2024-09-13 05:59:57,169 INFO    [dr2] Deleting lima cluster
    2024-09-13 05:59:57,169 INFO    [hub] Deleting lima cluster
    2024-09-13 05:59:57,255 WARNING [dr2] no such process
    2024-09-13 05:59:57,265 WARNING [dr2] remove /Users/nsoffer/.lima/dr2/ssh.sock: no such file or directory
    2024-09-13 05:59:57,265 WARNING [hub] remove /Users/nsoffer/.lima/hub/ssh.sock: no such file or directory
    2024-09-13 05:59:57,297 ERROR   [dr1] open /Users/nsoffer/.lima/dr2/lima.yaml: no such file or directory
    2024-09-13 05:59:57,297 ERROR   [hub] open /Users/nsoffer/.lima/dr2/lima.yaml: no such file or directory
    2024-09-13 05:59:57,298 ERROR   Command failed
    Traceback (most recent call last):
      ...
    drenv.commands.Error: Command failed:
       command: ('limactl', '--log-format=json', 'delete', '--force', 'dr1')
       exitcode: 1
       error:

Note how delete command for "dr1" and "hub" are failing to read lima.yaml
of cluster "dr2":

    2024-09-13 05:59:57,297 ERROR   [dr1] open /Users/nsoffer/.lima/dr2/lima.yaml: no such file or directory
    2024-09-13 05:59:57,297 ERROR   [hub] open /Users/nsoffer/.lima/dr2/lima.yaml: no such file or directory

With the lock, we run single limactl process at a time, so it cannot
race with other clusters.

Signed-off-by: Nir Soffer <[email protected]>

test/drenv/providers/lima/__init__.py

@@ -7,6 +7,7 @@
 import os
 import subprocess
 import tempfile
+import threading
 import time
 from functools import partial
 
@@ -33,6 +34,11 @@
     "service_cluster_ip_range",
 )
 
+# limactl delete is racy, trying to access lima.yaml in other clusters and
+# fails when the files are deleted by another limactl process. Until limactl is
+# fixed, ensure only single concurent delete.
+_delete_vm_lock = threading.Lock()
+
 # Provider scope
 
 
@@ -118,7 +124,9 @@ def delete(profile):
     start = time.monotonic()
     logging.info("[%s] Deleting lima cluster", profile["name"])
 
-    _delete_vm(profile)
+    with _delete_vm_lock:
+        _delete_vm(profile)
+
     _delete_additional_disks(profile)
     _remove_kubeconfig(profile)