Multicluster Mesh Addon

The multicluster-mesh-addon is an enhanced service mesh addon created with addon-framework, it is used to manage(discovery, deploy and federate) service meshes across multiple clusters in Red Hat Advanced Cluster Management for Kubernetes. With multicluster-mesh-addon, you can unify the configuration and operation of your services spanning from single cluster to multiple clusters in hybrid cloud.

Core Concepts

To simplify the configuration and operation of service meshes, we created the following three custom resource definitions (CRDs) than you can configure from the hub cluster of Red Hat Advanced Cluster Management for Kubernetes. Behind the scenes, the multicluster-mesh-addon translates these high level objects into low level istio resources and then applied into the managed clusters.

Mesh - a mesh resource is mapping to a physical service mesh in a managed cluster, it contains the desired state and status of the backend service mesh.

For each physical service mesh in a managed cluster, a mesh resource is created in the managed cluster namespace of hub cluster. An example of mesh resource would resemble the following yaml snippet:

apiVersion: mesh.open-cluster-management.io/v1alpha1
kind: Mesh
metadata:
  name: cluster1-istio-system-default
spec:
  cluster: cluster1
  controlPlane:
    components: ["base", "istiod", "istio-ingress"]
    namespace: istio-system
    profiles: ["default"]
    version: 1.13.2
  meshMemberRoll: ["istio-apps"]
  meshProvider: Upstream Istio
  meshConfig:
    trustDomain: cluster.local

MeshDeployment - meshdeployment resource is used to deploy physical service meshes to managed cluster(s), it supports deploying multiple physical service meshes to different managed clusters with one mesh template.

An example of meshdeployment resource would resemble the following yaml snippet:

apiVersion: mesh.open-cluster-management.io/v1alpha1
kind: MeshDeployment
metadata:
  name: istio
spec:
  clusters: ["cluster1", "cluster2"]
  controlPlane:
    components: ["base", "istiod", "istio-ingress"]
    namespace: mesh-system
    profiles: ["default"]
    version: 1.13.2
  meshMemberRoll: ["mesh-apps"]
  meshProvider: Upstream Istio

MeshFederation - meshfederation resource is used to federate service meshes so that the physical service meshes located in one cluster or different clusters to securely share and manage traffic between meshes while maintaining strong administrative boundaries in a multi-tenant environment.

An example of meshfederation resource would resemble the following yaml snippet:
```
apiVersion: mesh.open-cluster-management.io/v1alpha1
kind: MeshFederation
metadata:
  name: istio-federation
spec:
  meshPeers:
  - peers:
    - name: cluster1-istio
      cluster: cluster1
    - name: cluster2-istio
      cluster: cluster2
  trustConfig:
    trustType: Complete
```

Getting Started

Prerequisites

Ensure docker 18.03+ is installed.
Ensure golang 1.17+ is installed.
Prepare an environment of Red Hat Advanced Cluster Management for Kubernetes and login to the hub cluster with oc command line tool.
Make sure at least one managed cluster imported to the hub cluster of Red Hat Advanced Cluster Management for Kubernetes.
For mesh federation support, make sure at least two managed clusters are imported and the cloud provider must support the network load balancer IP so that the meshes spanning across managed clusters can be connected.

Build and Deploy

Build and push docker image:

make docker-build docker-push IMAGE=quay.io/<your_quayio_username>/multicluster-mesh-addon:latest

Deploy the multicluster-mesh-addon to hub cluster:

make deploy IMAGE=quay.io/<your_quayio_username>/multicluster-mesh-addon:latest

How to use

Note: For now, Openshift Service Mesh on Openshift managed cluster(s) and upstream Istio on *KS managed cluster(s) are supported.

Mesh Discovery:

If you have installed an Openshift Service Mesh in an openshift managed cluster, then you should find a mesh resource created in its namespace of hub cluster:

# oc -n ocp1 get mesh
NAME                       CLUSTER    VERSION    PROVIDER                PEERS    AGE
ocp1-istio-system-basic    ocp1       v2.1       Openshift Service Mesh           18s

If you have installed an istio service mesh in a managed cluster, then you should also find a mesh resource created in its namespace of hub cluster:

# oc -n eks1 get mesh
NAME                         CLUSTER    VERSION   PROVIDER                PEERS    AGE
eks1-istio-system-default    eks1       1.13.2    Upstream Istio                   50s

Mesh Deployment:

To deploy new Openshift Service Mesh to managed clusters, create a meshdeployment resource in hub cluster by specifying Openshift Service Mesh meshProvider and selecting the openshift managed cluster(s). For example, create the following meshdeployment resource to deploy new Openshift Service Mesh to Openshift managed cluster ocp1 and ocp2:

cat << EOF | oc apply -f -
apiVersion: mesh.open-cluster-management.io/v1alpha1
kind: MeshDeployment
metadata:
  name: ossm
  namespace: open-cluster-management
spec:
  clusters: ["ocp1", "ocp2"]
  controlPlane:
    components: ["prometheus", "istio-discovery", "istio-ingress", "mesh-config", "telemetry-common", "tracing"]
    namespace: mesh-system
    profiles: ["default"]
    version: v2.1
  meshMemberRoll: ["bookinfo"]
  meshProvider: Openshift Service Mesh
EOF

To deploy new istio service mesh(es) to managed clusters, create a meshdeployment resource by specifying Upstream Istio meshProvider and selecting the *KS managed cluster(s). For example, create the following meshdeployment resource to deploy new istio service mesh(es) to managed cluster eks1 and eks1:

cat << EOF | oc apply -f -
apiVersion: mesh.open-cluster-management.io/v1alpha1
kind: MeshDeployment
metadata:
  name: istio
  namespace: open-cluster-management
spec:
  clusters: ["eks1", "eks2"]
  controlPlane:
    components: ["base", "istiod", "istio-ingress"]
    namespace: istio-system
    profiles: ["default"]
    version: 1.13.2
  meshMemberRoll: ["bookinfo"]
  meshProvider: Upstream Istio
EOF

Then verify the service meshes are created:

# oc get mesh -A
NAMESPACE   NAME          CLUSTER   VERSION   PROVIDER                 PEERS   AGE
eks1        eks1-istio    eks1      1.13.2    Upstream Istio                   13s
eks2        eks2-istio    eks2      1.13.2    Upstream Istio                   13s
ocp1        ocp1-ossm     ocp1      v2.1      Openshift Service Mesh           57s
ocp2        ocp2-ossm     ocp2      v2.1      Openshift Service Mesh           57s

Mesh Federation:

To federate the Openshift Service Mesh in Openshift managed clusters, create a meshfederation resource in hub cluster by specifying the peers of Openshift Service Mesh and trustType of Limited. For example, federate ocp1-ossm and ocp2-ossm created in last step by creating a meshfederation resource with the following command:
```
cat << EOF | oc apply -f -
apiVersion: mesh.open-cluster-management.io/v1alpha1
kind: MeshFederation
metadata:
  name: ossm-federation
  namespace: open-cluster-management
spec:
  meshPeers:
  - peers:
    - name: ocp1-ossm
      cluster: ocp1
    - name: ocp2-ossm
      cluster: ocp2
  trustConfig:
    trustType: Limited
EOF
```
To federate the istio service meshes in *KS managed clusters, create a meshfederation resource in hub cluster by specifying the peers of istio mesh and and trustType of Complete. For example, federate eks1-istio and eks1-istio created in last step by creating a meshfederation resource with the following command:
```
cat << EOF | oc apply -f -
apiVersion: mesh.open-cluster-management.io/v1alpha1
kind: MeshFederation
metadata:
  name: istio-federation
  namespace: open-cluster-management
spec:
  meshPeers:
  - peers:
    - name: eks1-istio
      cluster: eks1
    - name: eks2-istio
      cluster: eks2
  trustConfig:
    trustType: Complete
EOF
```
Finally, deploy Bookinfo application spanning across managed clusters with the following different instructions to verify the federated meshes are working as expected:

Note: currently the verify steps have to be executed in the managed cluster, the work for the service discovery and service federation is still in progress.
- Mesh federation verify for Openshift Service Mesh
- Mesh federation verify for upstream Istio

TroubleShooting

If the traffic across meshes/clusters can't be routed successfully after creating MeshFederation resource, follow the following steps to find the root cause for different mesh providers:

For Openshift Service Mesh provider:

Make sure the services are exported from source mesh and imported into target mesh.
Make sure the ServiceMeshPeer resource is created and its remotes connected status is true for each peer mesh.
Make sure the mesh CA are exchanged by checking the CA configmap named <peer-mesh-name>-ca-root-cert is created in each mesh.
Make sure the ingress gateway named <peer-mesh-name>-ingress and egress gateway named <peer-mesh-name>-egress for peer mesh traffic is created in each mesh.

For upstream Istio provider:

Make sure the services are imported to target mesh cluster by creating corresponding ServiceEntry resources.
Make sure the eastwest gateway for each mesh is created and has a public loader balancer IP allocated.
Make sure ServiceEntry resource for remote service has the loader balancer IP of eastwest gateway for the peer mesh.
Make sure the secret named cacerts for the intermediate CA is created for each mesh.

If the cross meshes/clusters traffic is still not routed successfully after all the checks above, then open the access logs for the gateways by editting the MeshDeployment resources to add the following section to the spec:

  meshConfig:
    proxyConfig:
      accessLogging:
        file: /dev/stdout

Then check the access logs for the ingress/egress/eastwest gateway for each mesh.

Future Work

Services and workloads discovery
Federate services across meshes
Deploy application across meshes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

Multicluster Mesh Addon

Core Concepts

Getting Started

Prerequisites

Build and Deploy

How to use

TroubleShooting

Future Work

Files

README.md

Latest commit

History

README.md

File metadata and controls

Multicluster Mesh Addon

Core Concepts

Getting Started

Prerequisites

Build and Deploy

How to use

TroubleShooting

Future Work