New namespace template

[gianrubio]

• Setup namespace templating

• Fix device naming to accept devices like /dev/xvda9 (based on http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html)

[tyranron]

@gianrubio, thank you for contribution!

Not sure about device=~\"^/dev/(sd|hd|xvd).*$\", as it will allow other partitions (different from 9) to be included.
I decided to use /dev/[sf]da9 strictly because of CoreOS Disk Layout, as far as I'm using CoreOS.
Would you be so kind to comment this moment more wide? What is your opinion?

[gianrubio]

@tyranron you're absolutely right about the partition 9. However you can mount devices as /dev/xvd.
Look on my current cluster

What do you think to use /dev/([sf]|xv)da9 ?

[tyranron]

What do you think to use /dev/([sf]|xv)da9 ?

@gianrubio 👍👍👍

[tyranron]

@gianrubio I've dived into it a bit closely and found some problems:

1. Unlike Node templating, the namespace label is not present on every metric. Actually, it is present only on k8s containers. So I removed Namespace templating from queries not related to k8s.
2. cAdvisor's container abstraction is based on lmctfy's so containers are inherently nested hierarchically.
   That means that containers metrics with id="/" show total system usage, and we are using that in our "Total Usage" stats. Those metrics do not have namespace label. So for templating by Namespace in "Total Usage" stats we need to sum per-container usage now, like id=~"^/[^/]+/[^/]+$". It's okay when we choose concrete Namespace, but numbers differ when we choose Namespace: All, because now queries do not collect system usage that is not related to containers:

   # This is not 0
   sum(container_fs_usage_bytes{device=~"^/dev/([sf]|xv)da9$",id="/"}) - sum(container_fs_usage_bytes{device=~"^/dev/([sf]|xv)da9$",id=~"^/[^/]+/[^/]+$"})
   

   Ideally, we need to use id="/" for Namespace: All, and id=~"^/[^/]+/[^/]+$" in other cases. But Grafana templating does not provide such capabilities.
   I think we need to customize Prometheus relabel_config a bit more here.