Skip to content

Commit

Permalink
added storage page to ocpv
Browse files Browse the repository at this point in the history
Closes: #91
Signed-off-by: Robert Guske <[email protected]>
  • Loading branch information
rguske committed Jan 10, 2025
1 parent f72113c commit 9285a85
Show file tree
Hide file tree
Showing 5 changed files with 184 additions and 140 deletions.
Binary file added content/kubevirt/csi-driver-support.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
138 changes: 0 additions & 138 deletions content/kubevirt/local-storage.md

This file was deleted.

182 changes: 182 additions & 0 deletions content/kubevirt/storage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,182 @@
---
title: Storage
linktitle: Storage
description: Page for OCP Virtualization Storage options
tags: ['cnv', 'kubevirt', 'storage', 'ocp-v', 'csi']
---

# OpenShift Virtualization Storage

OpenShift Virtualization uses the Persistent Volume (PV) paradigm from Kubernetes to provide storage for virtual machines. More specifically, each VM disk is stored in a dedicated persistent volume provisioned and managed by a CSI (Container Storage Interface) driver provided by Red Hat or a supported storage vendor. This is different from the paradigm used by other virtualization offerings where a single storage device, e.g. an iSCSI/FC LUN or NFS export, is used to store many virtual machine disks.

Live migration of virtual machines requires that all virtual machine disks for the VM being migrated use PVCs with the read-write-many (RWX) access mode.

???+ Important

OCP Virtualization works differently for storage than vSphere and RHV: The reuse of existing storage is the aim by customers across the board but only little is known about the differences in the consumption of storage with OpenShift Virtualization. This reveals to be a major challenge for most of the vSphere to OCP-V projects.

In vSphere plain storage consumption, the storage capacity is provided using large volumes (aka described as LUNs) as datastores and then the vmdk "block" devices carved out as files coming from vmfs residing on the datastores. A similar approach is used by RHV.

In OCP Virtualization instead, every persistent volume for a virtual disk coming from a block storage must be provided as a real world block storage device. This usually leads to trouble on the OCP nodes as well as on the enterprise storage: the number of PVs used for VMs define the number of block volumes to create on the enterprise storage subsystems, to be handled in SAN mappings, and to be managed by FC/iSCSI and multi-pathing on the OCP nodes.

## Overview of the CSI Components

The diagram below provides a high-level overview of the CSI components in an OpenShift cluster. The CSI driver coordinates the creation and management of the volume on the storage device, mounting of the volume to the node hosting the virtual machine, and implementing features like snapshots and volume resizing at the storage volume level.

* *Source: [Docs Red Hat - OCP 4.17 Using CSI](https://docs.openshift.com/container-platform/4.17/storage/container_storage_interface/persistent-storage-csi.html)*

![CSI Storage](storage_csi.png)

## CSI drivers supported by OpenShift Container Platform

OpenShift Container Platform installs certain CSI drivers by default, giving users storage options that are not possible with in-tree volume plugins.

The following table describes the CSI drivers that are installed with OpenShift Container Platform supported by OpenShift Container Platform and which CSI features they support, such as volume snapshots and resize.

### *Applies to OCP-V version 4.17*

* *Source: [Docs Red Hat - OCP 4.17 Using CSI](https://docs.openshift.com/container-platform/4.17/storage/container_storage_interface/persistent-storage-csi.html)*

![CSI Driver Support](csi-driver-support.png)

* (1) Requires vSphere version 7.0 Update 3 or later for both vCenter Server and ESXi.
* Does not support fileshare volumes.

* (2) Offline volume expansion: minimum required vSphere version is 6.7 Update 3 P06
* Online volume expansion: minimum required vSphere version is 7.0 Update 2.

* (3) Does not support offline snapshots or resize. Volume must be attached to a running pod.

* (4) Azure File cloning does not supports NFS protocol. It supports the azurefile-csi storage class, which uses SMB protocol.
* Azure File cloning and snapshot are Technology Preview features.

## Configuring local storage for virtual machines

Official documentation: [Configuring local storage for virtual machines
](https://docs.openshift.com/container-platform/latest/cnv/cnv_virtual_machines/cnv_virtual_disks/cnv-configuring-local-storage-for-vms.html)

## Create a backing directory on each node

Label your CNV nodes:

```code
oc label node/compute-0 node-role.kubernetes.io/cnv
```

## Create MachineConfigPool

!!! important
One node can only apply one MachineConfigPool! That's why you have to include all worker machineconfigurations.

```yaml
oc apply -f - <<EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfigPool
metadata:
name: cnv
spec:
machineConfigSelector:
matchExpressions:
- {key: machineconfiguration.openshift.io/role, operator: In, values: [worker,cnv]}
nodeSelector:
matchLabels:
node-role.kubernetes.io/cnv: ""
EOF
```

Source: [custom-pools](https://github.com/openshift/machine-config-operator/blob/master/docs/custom-pools.md)

## Create MachineConfig

!!! note
Machine Config Operator do not support the Ignition filesystem.directory method.
[Supported vs Unsupported Ignition config changes](https://github.com/openshift/machine-config-operator/blob/master/docs/MachineConfigDaemon.md#supported-vs-unsupported-ignition-config-changes)

```yaml
oc apply -f - <<EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
name: 50-cnv-local-storage
labels:
machineconfiguration.openshift.io/role: cnv
spec:
config:
ignition:
version: 2.2.0
storage:
filesystems:
- name: storage
mount:
device: /dev/vdb
format: xfs
wipe_filesystem: false
systemd:
units:
- contents: |
[Unit]
Description=Create mountpoint /var/srv/storage
Before=kubelet.service
[Service]
ExecStart=/bin/mkdir -p /var/srv/storage
[Install]
WantedBy=var-srv-storage.mount
enabled: true
name: create-mountpoint-var-srv-storage.service
- name: var-srv-storage.mount
enabled: true
contents: |
[Unit]
Before=local-fs.target
[Mount]
What=/dev/vdb
Where=/var/srv/storage
Type=xfs
[Install]
WantedBy=local-fs.target
- contents: |
[Unit]
Description=Set SELinux chcon for hostpath provisioner
Before=kubelet.service
[Service]
ExecStart=/usr/bin/chcon -Rt container_file_t /var/srv/storage
[Install]
WantedBy=multi-user.target
enabled: true
name: hostpath-provisioner.service
EOF
```

## Create HostPathProvisioner

```yaml
oc apply -f - <<EOF
apiVersion: hostpathprovisioner.kubevirt.io/v1alpha1
kind: HostPathProvisioner
metadata:
name: hostpath-provisioner
spec:
imagePullPolicy: IfNotPresent
pathConfig:
path: "/var/srv/storage"
useNamingPrefix: "false"
EOF
```

## Create StorageClass

```yaml
oc apply -f - <<EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: hostpath-provisioner
provisioner: kubevirt.io/hostpath-provisioner
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
EOF
```
Binary file added content/kubevirt/storage_csi.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 2 additions & 2 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -252,8 +252,8 @@ nav:
- Descheduler: kubevirt/descheduler/index.md
- Templates: kubevirt/template.md
- Ansible: kubevirt/ansible/README.md
- Networking: kubevirt/networking/index.md
- Local-storage: kubevirt/local-storage.md
- Networking: kubevirt/networking.md
- Storage: kubevirt/storage.md
- NFS CSI Driver: kubevirt/nfs-csi-driver.md

- Certificates: certificate.md
Expand Down

0 comments on commit 9285a85

Please sign in to comment.