Configuring Additional Ephemeral Storage for Talos Linux Worker Nodes

[OrellBuehler]

Hello,

Background:
I'm using an OpenStack-based infrastructure provider that offers VMs with "system" (included) disks of 20GB, 50GB, or 80GB. My Talos Linux worker nodes require more ephemeral storage due to numerous containers downloading large amounts of (temporary) data and having large container images (>10GB). Even with an 80GB system disk, the nodes run out of ephemeral storage / disk space.

Current Setup:
For testing, I created a VM with a 20GB system disk (/dev/vda) and an additional 40GB disk (/dev/vdb).

Issue:
I'm unable to properly configure the additional disk (/dev/vdb) for ephemeral storage or as the installation disk ignoring / not using the "system" (included) disk (/dev/vda) at all.

Attempted Configurations:

1. Using diskSelector:

version: v1alpha1
debug: false
persist: true
machine:
    type: worker
    kubelet:
        image: ghcr.io/siderolabs/kubelet:v1.31.2
        defaultRuntimeSeccompProfileEnabled: true
        disableManifestsDirectory: true
    # disks:
    #     - device: /dev/vdb
    #       partitions:
    #         - mountpoint: /var/ephemeral
    install:
        diskSelector:
            size: '>= 30GB'
        image: ghcr.io/siderolabs/installer:v1.8.3
        wipe: true

Result: diskSelector doesn't seem to work for the install disk.

talosctl -n 10.0.1.234 get disks
NODE         NAMESPACE   TYPE   ID      VERSION   SIZE     READ ONLY   TRANSPORT   ROTATIONAL   WWID   MODEL          SERIAL
10.0.1.234   runtime     Disk   loop0   1         4.1 kB   true
10.0.1.234   runtime     Disk   loop1   1         75 MB    true
10.0.1.234   runtime     Disk   sr0     1         543 kB   false       ata         true                QEMU DVD-ROM
10.0.1.234   runtime     Disk   vda     1         22 GB    false       virtio      true
10.0.1.234   runtime     Disk   vdb     1         43 GB    false       virtio      true

talosctl -n 10.0.1.234 get dv
NODE         NAMESPACE   TYPE               ID      VERSION   TYPE        SIZE     DISCOVERED   LABEL       PARTITIONLABEL
10.0.1.234   runtime     DiscoveredVolume   loop1   1         disk        75 MB    squashfs
10.0.1.234   runtime     DiscoveredVolume   sr0     1         disk        543 kB   iso9660      config-2
10.0.1.234   runtime     DiscoveredVolume   vda     2         disk        22 GB    gpt
10.0.1.234   runtime     DiscoveredVolume   vda1    1         partition   105 MB   vfat                     EFI
10.0.1.234   runtime     DiscoveredVolume   vda2    1         partition   1.0 MB                            BIOS
10.0.1.234   runtime     DiscoveredVolume   vda3    1         partition   982 MB   xfs          BOOT        BOOT
10.0.1.234   runtime     DiscoveredVolume   vda4    1         partition   1.0 MB                            META
10.0.1.234   runtime     DiscoveredVolume   vda5    2         partition   92 MB    xfs          STATE       STATE
10.0.1.234   runtime     DiscoveredVolume   vda6    2         partition   20 GB    xfs          EPHEMERAL   EPHEMERAL
10.0.1.234   runtime     DiscoveredVolume   vdb     1         disk        43 GB

---

2. Partitioning the mounted disk:

version: v1alpha1
debug: false
persist: true
machine:
    type: worker
    kubelet:
        image: ghcr.io/siderolabs/kubelet:v1.31.2
        defaultRuntimeSeccompProfileEnabled: true
        disableManifestsDirectory: true
    disks:
        - device: /dev/vdb
          partitions:
            - mountpoint: /var/ephemeral
    install:
        diskSelector:
            size: '>= 30GB'
        image: ghcr.io/siderolabs/installer:v1.8.3
        wipe: true

Result: A partition is created but not assigned to anything.

talosctl -n 10.0.4.4 get disks
NODE       NAMESPACE   TYPE   ID      VERSION   SIZE     READ ONLY   TRANSPORT   ROTATIONAL   WWID   MODEL          SERIAL
10.0.4.4   runtime     Disk   loop0   1         4.1 kB   true
10.0.4.4   runtime     Disk   loop1   1         75 MB    true
10.0.4.4   runtime     Disk   sr0     1         543 kB   false       ata         true                QEMU DVD-ROM
10.0.4.4   runtime     Disk   vda     1         22 GB    false       virtio      true
10.0.4.4   runtime     Disk   vdb     1         43 GB    false       virtio      true

talosctl -n 10.0.4.4 get dv
NODE       NAMESPACE   TYPE               ID      VERSION   TYPE        SIZE     DISCOVERED   LABEL       PARTITIONLABEL
10.0.4.4   runtime     DiscoveredVolume   loop1   1         disk        75 MB    squashfs
10.0.4.4   runtime     DiscoveredVolume   sr0     1         disk        543 kB   iso9660      config-2
10.0.4.4   runtime     DiscoveredVolume   vda     2         disk        22 GB    gpt
10.0.4.4   runtime     DiscoveredVolume   vda1    1         partition   105 MB   vfat                     EFI
10.0.4.4   runtime     DiscoveredVolume   vda2    1         partition   1.0 MB                            BIOS
10.0.4.4   runtime     DiscoveredVolume   vda3    1         partition   982 MB   xfs          BOOT        BOOT
10.0.4.4   runtime     DiscoveredVolume   vda4    1         partition   1.0 MB                            META
10.0.4.4   runtime     DiscoveredVolume   vda5    2         partition   92 MB    xfs          STATE       STATE
10.0.4.4   runtime     DiscoveredVolume   vda6    2         partition   20 GB    xfs          EPHEMERAL   EPHEMERAL
10.0.4.4   runtime     DiscoveredVolume   vdb     2         disk        43 GB    gpt
10.0.4.4   runtime     DiscoveredVolume   vdb1    2         partition   43 GB    xfs

---

3. Setting the install disk explicitly:

version: v1alpha1
debug: false
persist: true
machine:
    type: worker
    kubelet:
        image: ghcr.io/siderolabs/kubelet:v1.31.2
        defaultRuntimeSeccompProfileEnabled: true
        disableManifestsDirectory: true
    # disks:
    #     - device: /dev/vdb
    #       partitions:
    #         - mountpoint: /var/ephemeral
    install:
        disk: /dev/vdb
        diskSelector:
            size: '>= 30GB'
        image: ghcr.io/siderolabs/installer:v1.8.3
        wipe: true

Result: The install disk doesn't change.

talosctl -n 10.0.3.164 get disks
NODE         NAMESPACE   TYPE   ID      VERSION   SIZE     READ ONLY   TRANSPORT   ROTATIONAL   WWID   MODEL          SERIAL
10.0.3.164   runtime     Disk   loop0   1         4.1 kB   true
10.0.3.164   runtime     Disk   loop1   1         75 MB    true
10.0.3.164   runtime     Disk   sr0     1         543 kB   false       ata         true                QEMU DVD-ROM
10.0.3.164   runtime     Disk   vda     1         22 GB    false       virtio      true
10.0.3.164   runtime     Disk   vdb     1         43 GB    false       virtio      true

talosctl -n 10.0.3.164 get dv
NODE         NAMESPACE   TYPE               ID      VERSION   TYPE        SIZE     DISCOVERED   LABEL       PARTITIONLABEL
10.0.3.164   runtime     DiscoveredVolume   loop1   1         disk        75 MB    squashfs
10.0.3.164   runtime     DiscoveredVolume   sr0     1         disk        543 kB   iso9660      config-2
10.0.3.164   runtime     DiscoveredVolume   vda     2         disk        22 GB    gpt
10.0.3.164   runtime     DiscoveredVolume   vda1    1         partition   105 MB   vfat                     EFI
10.0.3.164   runtime     DiscoveredVolume   vda2    1         partition   1.0 MB                            BIOS
10.0.3.164   runtime     DiscoveredVolume   vda3    1         partition   982 MB   xfs          BOOT        BOOT
10.0.3.164   runtime     DiscoveredVolume   vda4    1         partition   1.0 MB                            META
10.0.3.164   runtime     DiscoveredVolume   vda5    2         partition   92 MB    xfs          STATE       STATE
10.0.3.164   runtime     DiscoveredVolume   vda6    2         partition   20 GB    xfs          EPHEMERAL   EPHEMERAL
10.0.3.164   runtime     DiscoveredVolume   vdb     1         disk        43 GB

---

Questions:

1. Why doesn't the install disk change when specified in the configuration?
2. Based on the information I found on the doucmentation I am not sure if we can set a second disk (/dev/vdb) for EPHEMERAL storage in Talos Linux yet? But how would one apply a different volume config (https://www.talos.dev/v1.8/reference/configuration/block/volumeconfig/#VolumeConfig.provisioning), the documentation is not clear where to configure this (in the machine section?)?

Additional Information:

• Talos version: 1.8.3
• Kubernetes version: v1.31.2
• Talos image: https://factory.talos.dev/image/376567988ad370138ad8b2698212367b8edcb69b5fd68c80be1f2ec7d603b4ba/v1.8.3/openstack-amd64.raw.xz

Any guidance or solutions would be greatly appreciated. Thank you!

[smira]

It's not clear whether you wiped the VM between the attempts, but machine.install section is only used at the initial install time. Moreover, on OpenStack you're using a disk image, so there's no install process at all, and Talos system disk is the one OpenStack puts the disk image to.

Result: A partition is created but not assigned to anything.

Not sure what you mean, it is mounted to the path you specified.

Based on the information I found on the doucmentation I am not sure if we can set a second disk (/dev/vdb) for EPHEMERAL

See https://www.talos.dev/v1.8/talos-guides/configuration/disk-management/#machine-configuration

You can simply add the document by splitting it with --- (YAML document separator) or use it as a config patch.

Keep in mind, it should be present in the initial configuration, otherwise Talos will create default EPHEMERAL.

[OrellBuehler]

Hello @smira,

Thanks four your reply.

Yes I did recreate completely new instances and volumes for each of the scenarios.

I added the document at the end of the config and it works now! I've tried this a while ago but then it did not work (probably because of some other configuration issue) and I've not retried this. But thank you so much for your assistance.