Introduce wait time between resizes

[mohammed-uddin]

What should the feature do:
Re-opening this issue: #274

What is use case behind this feature:
Bringing this back to life.

I've got the same issue.

Say I have a threshold of 10%. And I increase capacity by 3Gi up to a max of 50Gi.

My pvc uses 91% of the storage so as expected a resize occurs. My new capacity is 13GI.

1 hour later my pvc once again fills up beyond 90%. The resizer attempts to scale up my PVC to 16Gi but because I'm using Amazon's storageClass this is not permitted for another 5 hours (1 resize per 6 hours). Say my controller has an interval of 30 second. So 30 seconds later the pvc-resizer sees that I'm using more than 90% and attempts a resize again this time 19Gi. This process repeats again and now I'm 22Gi. This keeps happening until I've scaled up to 50Gi of storage. After the 6 hour mark from the initial resize the resize applies and now I'm stuck with a giant 50Gi pvc.

[mohammed-uddin]

Maybe a generic fix here is:

if (spec.resources.requests.storage == status.capacity.storage) {
  do_the_resizing()
}

that way no resizing is queued up while one is already pending.

thoughts on that idea?

[mohammed-uddin]

@ushitora-anqou Should pvc-autoresizer do

.spec.resources.requests.storage = status.capacity.storage + resize.topolvm.io/increase

right now it seems as though this project does.

.spec.resources.requests.storage = .spec.resources.requests.storage + resize.topolvm.io/increase

[llamerada-jp]

@mohammed-uddin
In the story you described, I noticed that pvc-autoresizer was behaving unexpectedly. pvc-autoresizer is designed not to re-expand unexpanded PVC, based on the following logic. If it were working as expected, it wouldn't process as you describe.

pvc-autoresizer/internal/runners/pvc_autoresizer.go

Lines 195 to 207 in f2fee52

	preCap, exist := pvc.Annotations[pvcautoresizer.PreviousCapacityBytesAnnotation]
	if exist {
	preCapInt64, err := strconv.ParseInt(preCap, 10, 64)
	if err != nil {
	log.V(logLevelWarn).Info("failed to parse pre_cap_bytes annotation", "error", err.Error())
	// lint:ignore nilerr ignores this because invalid annotations should be allowed.
	return nil
	}
	if preCapInt64 == vs.CapacityBytes {
	log.Info("waiting for resizing...", "capacity", vs.CapacityBytes)
	return nil
	}
	}

The kubelet_volume_stats_capacity_bytes metrics values are stored in the PVC's resize.topolvm.io/pre_capacity_bytes annotation, and the expanding process should be skipped until these metrics values change. There may be unexpected behavior hidden there, so could you please check how the annotations and metrics are changing in your environment?

[mohammed-uddin]

@llamerada-jp You're right, thanks for the clarification.