RFE: kueue-viz: Add some compiled historical observability metrics in kueue-viz

[akram]

What would you like to be added:
The current kueue-viz dashboard is more about having a live view on how kueue is behaving.
It could be interesting to have a few additional historical metrics:

• average wait time for jobs to get resources
• preemption rate per queue
• resource in contention per resource flavor
• most borrowed resource within cohort

Why is this needed:
kueue admin or users may be interested in knowing how their kueue cluster is behaving with jobs and resources to anticipate peaks, loads or even provisioning of new resources.

Completion requirements:
The best way to store or compute these metrics could be with prometheus. But, if this is not not feasible, we may need to think about having a data storage beside the backend.

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[mimowo]

/kind dashboard