From 66494413973d1face705aab39ba0e5ea5e749f32 Mon Sep 17 00:00:00 2001 From: Ben Harosh Date: Tue, 23 Apr 2024 12:56:07 -0700 Subject: [PATCH] Add configuration support for DataDog multi-regions --- README.md | 284 +++++++++++++++++++------------- docker-build/docker.env | 4 +- fluent.conf.rt | 4 +- fluent.conf.xray | 4 +- helm/artifactory-ha-values.yaml | 3 + helm/artifactory-values.yaml | 3 + helm/jfrog_helm.env | 3 +- helm/xray-values.yaml | 3 + jfrog.env | 3 +- 9 files changed, 187 insertions(+), 124 deletions(-) diff --git a/README.md b/README.md index bea3817..1c75048 100644 --- a/README.md +++ b/README.md @@ -4,11 +4,12 @@ The following document describes how to configure Datadog to gather logs, metric ## Versions Supported -This integration is last tested with Artifactory 7.71.11 and Xray 3.88.12 versions. +This integration is last tested with Artifactory 7.77.8 and Xray 3.92.7 versions. ## Table of Contents `Note! You must follow the order of the steps throughout Datadog Configuration` + 1. [Datadog Setup](#datadog-setup) 2. [JFrog Metrics Setup](#jfrog-metrics-setup) 3. [Fluentd Installation](#fluentd-installation) @@ -32,49 +33,49 @@ Once datadog is setup, we can access logs via Logs > Search. We can also select * Add all attributes as facets from Facets > Add on the left side of the screen in Logs > search ## JFrog Metrics Setup -To enable metrics in Artifactory, make the following configuration changes to the [Artifactory System YAML](https://www.jfrog.com/confluence/display/JFROG/Artifactory+System+YAML) -```yaml + +Metrics collection is diabled by default in Artifactory by default. To enable metrics in Artifactory, make the following configuration changes to the [Artifactory System YAML](https://www.jfrog.com/confluence/display/JFROG/Artifactory+System+YAML). + artifactory: - metrics: - enabled: true - openMetrics: - enabled: true -``` +metrics: +enabled: true +openMetrics: +enabled: true Once this configuration is done and the application is restarted, metrics will be available in Open Metrics Format -Metrics are enabled by default in Xray. -For kubernetes based installs, openMetrics are enabled in the helm install commands listed below +:bulb: Metrics are enabled by default in Xray. + +:bulb: For kubernetes based installs, openMetrics collection is enabled in the helm install commands listed in the sections below ## Fluentd Installation ### OS / Virtual Machine + Ensure you have access to the Internet from a virtual machine (VM). We recommend installation through FluentD's native OS based package installs: -| OS | Package Manager | Link | -|----------------|------------------------|------------------------------------------------------| -| CentOS/RHEL | Linux - RPM (YUM) | https://docs.fluentd.org/installation/install-by-rpm | -| Debian/Ubuntu | Linux - APT | https://docs.fluentd.org/installation/install-by-deb | -| MacOS/Darwin | MacOS - DMG | https://docs.fluentd.org/installation/install-by-dmg | -| Windows | Windows - MSI | https://docs.fluentd.org/installation/install-by-msi | -| Gem Install** | MacOS & Linux - Gem | https://docs.fluentd.org/installation/install-by-gem | + +| OS | Package Manager | Link | +| ------------- | ------------------- | ---------------------------------------------------- | +| CentOS/RHEL | Linux - RPM (YUM) | https://docs.fluentd.org/installation/install-by-rpm | +| Debian/Ubuntu | Linux - APT | https://docs.fluentd.org/installation/install-by-deb | +| MacOS/Darwin | MacOS - DMG | https://docs.fluentd.org/installation/install-by-dmg | +| Windows | Windows - MSI | https://docs.fluentd.org/installation/install-by-msi | +| Gem Install** | MacOS & Linux - Gem | https://docs.fluentd.org/installation/install-by-gem | ##### Gem based install + For a Gem-based install, the Ruby Interpreter must be setup first. You can install the Ruby Interpreter by doing the following: 1. Install Ruby Version Manager (RVM) outlined in the [RVM documentation](https://rvm.io/rvm/install#installation-explained). - * Use the `SUDO` command for multi-user installation. For more information, see the [RVM troubleshooting documentation](https://rvm.io/support/troubleshooting#sudo). + * Use the `SUDO` command for multi-user installation. For more information, see the [RVM troubleshooting documentation](https://rvm.io/support/troubleshooting#sudo). 2. After the RVM installation is complete, execute the command 'rvm -v' to verify. - 3. Install Ruby v2.7.0 or above with the command `rvm install `, (for example, `rvm install 2.7.5`). - 4. Verify the Ruby installation, execute `ruby -v`, gem installation `gem -v` and `bundler -v` to ensure all the components are intact. - 5. Install the FluentD gem with the command `gem install fluentd`. - 6. After FluentD is successfully installed, install the following plugins. -```shell +```bash gem install fluent-plugin-concat gem install fluent-plugin-datadog gem install fluent-plugin-jfrog-siem @@ -83,10 +84,12 @@ gem install fluent-plugin-jfrog-send-metrics ``` ##### Configure Fluentd + We rely on environment variables to stream log files to your observability dashboards. Ensure that you fill in the `.env` file with the correct values. You can download the `.env` file [here](https://raw.githubusercontent.com/jfrog/log-analytics-datadog/master/jfrog.env). * **JF_PRODUCT_DATA_INTERNAL**: The environment variable JF_PRODUCT_DATA_INTERNAL must be defined to the correct location. For each JFrog service, you can find its active log files in the `$JFROG_HOME//var/log` directory * **DATADOG_API_KEY**: API Key from [Datadog](https://app.datadoghq.com/organization-settings/api-keys) +* **DATADOG_API_HOST**: Your DataDog host based on your [DataDog Site Parameter from this list](https://docs.datadoghq.com/getting_started/site/#access-the-datadog-site) * **JPD_URL**: Artifactory JPD URL with the format `http://` * **JPD_ADMIN_USERNAME**: Artifactory username for authentication * **JFROG_ADMIN_TOKEN**: Artifactory [Access Token](https://jfrog.com/help/r/how-to-generate-an-access-token-video/artifactory-creating-access-tokens-in-artifactory) for authentication @@ -94,142 +97,175 @@ We rely on environment variables to stream log files to your observability dashb Apply the `.env` files and run the fluentd wrapper with the following command, and note that the argument points to the `fluent.conf.*` file previously configured: -```shell +```bash source jfrog.env ./fluentd $JF_PRODUCT_DATA_INTERNAL/fluent.conf. ``` ### Docker + In order to run FluentD as a docker image to send the logs, violations, and metrics data to Datadog, execute the following commands on the host that runs the docker. 1. Execute the `docker version` and `docker ps` commands to verify that the Docker installation is functional. - 2. If the version and process are listed successfully, build the intended docker image for Datadog using the docker file. You can download [this Dockerfile]https://raw.githubusercontent.com/jfrog/log-analytics-datadog/master/docker-build/Dockerfile to any directory that has write permissions. - 3. Download the `docker.env` file needed to run `Jfrog/FluentD` Docker Images for Datadog. You can download [this docker.env]https://raw.githubusercontent.com/jfrog/log-analytics-datadog/master/docker-build/docker.env to the directory where the docker file was downloaded. - 4. Execute the following command to build the docker image: `docker build --build-arg SOURCE="JFRT" --build-arg TARGET="DATADOG" -t `. For example: - ```shell - docker build --build-arg SOURCE="JFRT" --build-arg TARGET="DATADOG" -t jfrog/fluentd-datadog-rt .' - ``` - + ```bash + docker build --build-arg SOURCE="JFRT" --build-arg TARGET="DATADOG" -t jfrog/fluentd-datadog-rt .' + ``` 5. Fill out the necessary information in the docker.env file: * **JF_PRODUCT_DATA_INTERNAL**: The environment variable JF_PRODUCT_DATA_INTERNAL must be defined to the correct location. For each JFrog service you will find its active log files in the `$JFROG_HOME//var/log` directory * **DATADOG_API_KEY**: API Key from [Datadog](https://docs.datadoghq.com/account_management/api-app-keys/) + * **DATADOG_API_HOST**: Your DataDog host based on your [DataDog Site Parameter from this list](https://docs.datadoghq.com/getting_started/site/#access-the-datadog-site) * **JPD_URL**: Artifactory JPD URL of the format `http://` * **JPD_ADMIN_USERNAME**: Artifactory username for authentication * **JFROG_ADMIN_TOKEN**: Artifactory [Access Token](https://jfrog.com/help/r/how-to-generate-an-access-token-video/artifactory-creating-access-tokens-in-artifactory) for authentication * **COMMON_JPD**: This flag should be set as true only for non-kubernetes installations or installations where JPD base URL is same to access both Artifactory and Xray (ex: https://sample_base_url/artifactory or https://sample_base_url/xray) - 6. Execute 'docker run -it --name jfrog-fluentd-datadog-rt -v :/var/opt/jfrog/artifactory --env-file docker.env ' The `` should be an absolute path where the Jfrog Artifactory Logs folder resides, such as a Docker based Artifactory Installation like`/var/opt/jfrog/artifactory/var/logs` on the docker host. For example: - ```shell - docker run -it --name jfrog-fluentd-datadog-rt -v $JFROG_HOME/artifactory/var/:/var/opt/jfrog/artifactory --env-file docker.env jfrog/fluentd-datadog-rt - ``` - + ```bash + docker run -it --name jfrog-fluentd-datadog-rt -v $JFROG_HOME/artifactory/var/:/var/opt/jfrog/artifactory --env-file docker.env jfrog/fluentd-datadog-rt + ``` ### Kubernetes Deployment with Helm + The recommended installation method for Kubernetes is to utilize the helm chart with the associated values.yaml in this repo. + | Product | Example Values File | -|----------------|---------------------------------| +| -------------- | ------------------------------- | | Artifactory | helm/artifactory-values.yaml | | Artifactory HA | helm/artifactory-ha-values.yaml | | Xray | helm/xray-values.yaml | Add JFrog Helm repository: -```shell +```bash helm repo add jfrog https://charts.jfrog.io helm repo update ``` -Replace placeholders with your ``masterKey`` and ``joinKey``. To generate each of them, use the command -``openssl rand -hex 32`` -#### Artifactory ⎈: +Throughout the exampled helm installations we'll use `jfrog-dd` as an example namespace. That said, you can use a different or existing namespace instead by setting the following environment variable -1. Skip this step if you already have Artifactory installed. Else, install Artifactory using the command below - ```shell - helm upgrade --install artifactory jfrog/artifactory \ - --set artifactory.masterKey=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF \ - --set artifactory.joinKey=EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE \ - --set artifactory.license.secret=artifactory-license \ - --set artifactory.license.dataKey=artifactory.cluster.license \ - --set artifactory.metrics.enabled=true \ - --set artifactory.openMetrics.enabled=true - ``` - -2. Create a secret for JFrog's admin token - [Access Token](https://jfrog.com/help/r/how-to-generate-an-access-token-video/artifactory-creating-access-tokens-in-artifactory) using any of the following methods - ```shell - kubectl create secret generic jfrog-admin-token --from-file=token= - - OR - - kubectl create secret generic jfrog-admin-token --from-literal=token= - ``` -3. For Artifactory installation, download the .env file from [here](https://github.com/jfrog/log-analytics-datadog/raw/master/helm/jfrog_helm.env). Fill in the jfrog_helm.env file with correct values. +```bash +export INST_NAMESPACE=jfrog-dd +``` - * **JF_PRODUCT_DATA_INTERNAL**: Helm based installs will already have this defined based upon the underlying Docker images. Not a required field for k8s installation - * **DATADOG_API_KEY**: API Key from [Datadog](https://app.datadoghq.com/organization-settings/api-keys) - * **JPD_URL**: Artifactory JPD URL of the format `http://` - * **JPD_ADMIN_USERNAME**: Artifactory username for authentication - * **COMMON_JPD**: This flag should be set as true only for non-Kubernetes installations or installations where the JPD base URL is the same to access both Artifactory and Xray (for example, `https://sample_base_url/artifactory` or `https://sample_base_url/xray`) +If you don't have an existing namespace for the deployment, create it and set the kubectl context to use this namespace + +```bash +kubectl create namespace $INST_NAMESPACE +kubectl config set-context --current --namespace=$INST_NAMESPACE +``` - Apply the .env files using the helm command below +Generate ``masterKey`` and ``joinKey`` for the installation - ```shell - source jfrog_helm.env - ``` -4. Postgres password is required to upgrade Artifactory. Run the following command to get the current password - ```shell - POSTGRES_PASSWORD=$(kubectl get secret artifactory-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode) +```bash +export JOIN_KEY=$(openssl rand -hex 32) +export MASTER_KEY=$(openssl rand -hex 32) +``` + +#### Artifactory ⎈: + +1. Skip this step if you already have Artifactory installed. Else, install Artifactory using the command below + + ```bash + helm upgrade --install artifactory jfrog/artifactory \ + --set artifactory.masterKey=$MASTER_KEY \ + --set artifactory.joinKey=$JOIN_KEY \ + --set artifactory.license.secret=artifactory-license \ + --set artifactory.license.dataKey=artifactory.cluster.license \ + --set artifactory.metrics.enabled=true \ + --set artifactory.openMetrics.enabled=true \ + -n $INST_NAMESPACE ``` -5. Upgrade Artifactory installation using the command below - ```shell - helm upgrade --install artifactory jfrog/artifactory \ - --set artifactory.masterKey=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF \ - --set artifactory.joinKey=EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE \ - --set artifactory.metrics.enabled=true --set artifactory.openMetrics.enabled=true \ - --set databaseUpgradeReady=true --set postgresql.postgresqlPassword=$POSTGRES_PASSWORD --set nginx.service.ssloffload=true \ - --set datadog.api_key=$DATADOG_API_KEY \ - --set jfrog.observability.jpd_url=$JPD_URL \ - --set jfrog.observability.username=$JPD_ADMIN_USERNAME \ - --set jfrog.observability.common_jpd=$COMMON_JPD \ - -f helm/artifactory-values.yaml - ``` + + :bulb: Metrics collection is disabled by default in Artifactory. Please make sure that you are following the above `helm upgrade` command to enable them in Artifactory by setting to true both `artifactory.metrics.enabled` and `artifactory.openMetrics.enabled` + + 2. Create a secret for JFrog's admin token - [Access Token](https://jfrog.com/help/r/how-to-generate-an-access-token-video/artifactory-creating-access-tokens-in-artifactory) using any of the following methods + + ```bash + kubectl create secret generic jfrog-admin-token --from-file=token= + + OR + + kubectl create secret generic jfrog-admin-token --from-literal=token= + ``` + 3. For Artifactory installation, download the .env file from [here](https://github.com/jfrog/log-analytics-datadog/raw/master/helm/jfrog_helm.env). Fill in the jfrog_helm.env file with correct values. + + * **JF_PRODUCT_DATA_INTERNAL**: Helm based installs will already have this defined based upon the underlying Docker images. Not a required field for k8s installation + * **DATADOG_API_KEY**: API Key from [Datadog](https://app.datadoghq.com/organization-settings/api-keys) + * **DATADOG_API_HOST**: Your DataDog host based on your [DataDog Site Parameter from this list](https://docs.datadoghq.com/getting_started/site/#access-the-datadog-site) + * **JPD_URL**: Artifactory JPD URL of the format `http://` + * **JPD_ADMIN_USERNAME**: Artifactory username for authentication + * **COMMON_JPD**: This flag should be set as true only for non-Kubernetes installations or installations where the JPD base URL is the same to access both Artifactory and Xray (for example, `https://sample_base_url/artifactory` or `https://sample_base_url/xray`) + + Apply the .env files using the helm command below + + ```shell + source jfrog_helm.env + ``` + 4. Postgres password is required to upgrade Artifactory. Run the following command to get the current password + + ```bash + POSTGRES_PASSWORD=$(kubectl get secret artifactory-postgresql -n $INST_NAMESPACE -o jsonpath="{.data.postgresql-password}" | base64 --decode) + ``` + 5. Upgrade Artifactory installation using the command below + + ```bash + helm upgrade --install artifactory jfrog/artifactory \ + --set artifactory.masterKey=$MASTER_KEY \ + --set artifactory.joinKey=$JOIN_KEY \ + --set artifactory.metrics.enabled=true --set artifactory.openMetrics.enabled=true \ + --set databaseUpgradeReady=true --set postgresql.postgresqlPassword=$POSTGRES_PASSWORD --set nginx.service.ssloffload=true \ + --set datadog.api_key=$DATADOG_API_KEY \ + --set datadog.api_host=$DATADOG_API_HOST \ + --set jfrog.observability.jpd_url=$JPD_URL \ + --set jfrog.observability.username=$JPD_ADMIN_USERNAME \ + --set jfrog.observability.common_jpd=$COMMON_JPD \ + -f helm/artifactory-values.yaml \ + -n $INST_NAMESPACE + ``` #### Artifactory-HA ⎈: 1. For HA installation, please create a license secret on your cluster prior to installation. - ```shell + + ```bash kubectl create secret generic artifactory-license --from-file=artifactory.cluster.license ``` 2. Skip this step if you already have Artifactory installed. Else, install Artifactory using the command below - ```shell - helm upgrade --install artifactory-ha jfrog/artifactory-ha \ - --set artifactory.masterKey=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF \ - --set artifactory.joinKey=EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE \ - --set artifactory.license.secret=artifactory-license \ - --set artifactory.license.dataKey=artifactory.cluster.license \ - --set artifactory.metrics.enabled=true \ - --set artifactory.openMetrics.enabled=true - ``` + + ```bash + helm upgrade --install artifactory-ha jfrog/artifactory-ha \ + --set artifactory.masterKey=$MASTER_KEY \ + --set artifactory.joinKey=$JOIN_KEY \ + --set artifactory.license.secret=artifactory-license \ + --set artifactory.license.dataKey=artifactory.cluster.license \ + --set artifactory.metrics.enabled=true \ + --set artifactory.openMetrics.enabled=true \ + -n $INST_NAMESPACE + ``` + + :bulb: Metrics collection is disabled by default in Artifactory-HA. Please make sure that you are following the above `helm upgrade` command to enable them in Artifactory-HA by setting to true both `artifactory.metrics.enabled` and `artifactory.openMetrics.enabled` 3. Create a secret for JFrog's admin token - [Access Token](https://jfrog.com/help/r/how-to-generate-an-access-token-video/artifactory-creating-access-tokens-in-artifactory) using any of the following methods - ```shell + + ```bash kubectl create secret generic jfrog-admin-token --from-file=token= - + OR - + kubectl create secret generic jfrog-admin-token --from-literal=token= ``` 4. Download the .env file from [here](https://github.com/jfrog/log-analytics-datadog/raw/master/helm/jfrog_helm.env). Fill in the jfrog_helm.env file with correct values. * **JF_PRODUCT_DATA_INTERNAL**: Helm based installs will already have this defined based upon the underlying Docker images. Not a required field for k8s installation * **DATADOG_API_KEY**: API Key from [Datadog](https://app.datadoghq.com/organization-settings/api-keys) + * **DATADOG_API_HOST**: Your DataDog host based on your [DataDog Site Parameter from this list](https://docs.datadoghq.com/getting_started/site/#access-the-datadog-site) * **JPD_URL**: Artifactory JPD URL of the format `http://` * **JPD_ADMIN_USERNAME**: Artifactory username for authentication * **COMMON_JPD**: This flag should be set as true only for non-Kubernetes installations or installations where the JPD base URL is the same to access both Artifactory and Xray (for example, `https://sample_base_url/artifactory` or `https://sample_base_url/xray`) @@ -240,27 +276,32 @@ Replace placeholders with your ``masterKey`` and ``joinKey``. To generate each o source jfrog_helm.env ``` 5. Postgres password is required to upgrade Artifactory. Run the following command to get the current password + ```shell - POSTGRES_PASSWORD=$(kubectl get secret artifactory-ha-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode) + POSTGRES_PASSWORD=$(kubectl get secret artifactory-ha-postgresql -n $INST_NAMESPACE -o jsonpath="{.data.postgresql-password}" | base64 --decode) ``` 6. Upgrade Artifactory HA installation using the command below - ```text + + ```bash helm upgrade --install artifactory-ha jfrog/artifactory-ha \ - --set artifactory.masterKey=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF \ - --set artifactory.joinKey=EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE \ + --set artifactory.masterKey=$MASTER_KEY \ + --set artifactory.joinKey=$JOIN_KEY \ --set artifactory.metrics.enabled=true --set artifactory.openMetrics.enabled=true \ --set databaseUpgradeReady=true --set postgresql.postgresqlPassword=$POSTGRES_PASSWORD --set nginx.service.ssloffload=true \ - --set datadog.api_key=$DATADOG_API_KEY \ + --set datadog.api_key=$DATADOG_API_KEY \ + --set datadog.api_host=$DATADOG_API_HOST \ --set jfrog.observability.jpd_url=$JPD_URL \ --set jfrog.observability.username=$JPD_ADMIN_USERNAME \ --set jfrog.observability.common_jpd=$COMMON_JPD \ - -f helm/artifactory-ha-values.yaml + -f helm/artifactory-ha-values.yaml \ + -n $INST_NAMESPACE ``` #### Xray ⎈: Create a secret for JFrog's admin token - [Access Token](https://jfrog.com/help/r/how-to-generate-an-access-token-video/artifactory-creating-access-tokens-in-artifactory) using any of the following methods if it doesn't exist -```shell + +```bash kubectl create secret generic jfrog-admin-token --from-file=token= OR @@ -278,43 +319,56 @@ For Xray installation, download the .env file from [here](https://raw.githubuser Apply the .env files and then run the helm command below -```shell +```bash source jfrog_helm.env ``` +Generate a master key for xray + +```bash +export XRAY_MASTER_KEY=$(openssl rand -hex 32) +``` + Use the same `joinKey` as you used in Artifactory installation to allow Xray node to successfully connect to Artifactory. -```shell +```bash helm upgrade --install xray jfrog/xray --set xray.jfrogUrl=http://my-artifactory-nginx-url \ - --set xray.masterKey=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF \ - --set xray.joinKey=EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE \ - --set datadog.api_key=$DATADOG_API_KEY \ + --set xray.masterKey=$XRAY_MASTER_KEY \ + --set xray.joinKey=$JOIN_KEY \ + --set datadog.api_key=$DATADOG_API_KEY \ + --set datadog.api_host=$DATADOG_API_HOST \ --set jfrog.observability.jpd_url=$JPD_URL \ --set jfrog.observability.username=$JPD_ADMIN_USERNAME \ --set jfrog.observability.common_jpd=$COMMON_JPD \ - -f helm/xray-values.yaml + -f helm/xray-values.yaml \ + -n $INST_NAMESPACE ``` ### Dashboards -#### JFrog Artifactory Dashboard +#### JFrog Artifactory Dashboard + This dashboard is divided into three sections Application, Audit and Requests + * **Application** - This section tracks Log Volume(information about different log sources) and Artifactory Errors over time(bursts of application errors that may otherwise go undetected) * **Audit** - This section tracks audit logs help you determine who is accessing your Artifactory instance and from where. These can help you track potentially malicious requests or processes (such as CI jobs) using expired credentials. * **Requests** - This section tracks HTTP response codes, Top 10 IP addresses for uploads and downloads #### JFrog Artifactory Metrics dashboard + This dashboard tracks Artifactory System Metrics, JVM memory, Garbabe Collection, Database Connections, and HTTP Connections metrics #### JFrog Xray Logs dashboard -This dashboard provides a summary of access, service and traffic log volumes associated with Xray. Additionally, customers are also able to track various HTTP response codes, HTTP 500 errors, and log errors for greater operational insight + +This dashboard provides a summary of access, service and traffic log volumes associated with Xray. Additionally, customers are also able to track various HTTP response codes, HTTP 500 errors, and log errors for greater operational insight #### JFrog Xray Violations Dashboard + This dashboard provides an aggregated summary of all the license violations and security vulnerabilities found by Xray. Information is segment by watch policies and rules. Trending information is provided on the type and severity of violations over time, as well as, insights on most frequently occurring CVEs, top impacted artifacts and components. #### JFrog Xray Metrics Dashboard -This dashboard tracks System Metrics, and data metrics about Scanned Artifacts and Scanned Components +This dashboard tracks System Metrics, and data metrics about Scanned Artifacts and Scanned Components ## Demo Requirements diff --git a/docker-build/docker.env b/docker-build/docker.env index 8df7cfc..c15f636 100644 --- a/docker-build/docker.env +++ b/docker-build/docker.env @@ -1,9 +1,7 @@ JF_PRODUCT_DATA_INTERNAL=path_to_jfrog_logs DATADOG_API_KEY=change_me +DATADOG_API_HOST=datadoghq.com JPD_URL=https://change_me.jfrog.io JPD_ADMIN_USERNAME=admin JFROG_ADMIN_TOKEN=change_me COMMON_JPD=false - - - diff --git a/fluent.conf.rt b/fluent.conf.rt index 8d27854..a82570e 100644 --- a/fluent.conf.rt +++ b/fluent.conf.rt @@ -14,7 +14,7 @@ @type jfrog_send_metrics target_platform "DATADOG" apikey "#{ENV['DATADOG_API_KEY']}" - url "https://api.datadoghq.com/api/v2/series" + url "https://api.#{ENV['DATADOG_API_HOST']}/api/v2/series" # ddtags ["instance:test-artifactory", "cluster:GKE"] # ALL CALLHOME @@ -370,7 +370,7 @@ include_tag_key true dd_source jfrog_platform service jfrog_artifactory - host http-intake.logs.datadoghq.com + host "http-intake.logs.#{ENV['DATADOG_API_HOST']}" flush_interval 1s # frequency of the buffer flush diff --git a/fluent.conf.xray b/fluent.conf.xray index 2e07f52..d5b08b2 100644 --- a/fluent.conf.xray +++ b/fluent.conf.xray @@ -14,7 +14,7 @@ @type jfrog_send_metrics target_platform "DATADOG" apikey "#{ENV['DATADOG_API_KEY']}" - url "https://api.datadoghq.com/api/v2/series" + url "https://api.#{ENV['DATADOG_API_HOST']}/api/v2/series" # ddtags ["instance:test-artifactory", "cluster:GKE"] # ALL CALLHOME @@ -322,7 +322,7 @@ include_tag_key true dd_source jfrog_platform service jfrog_xray - host http-intake.logs.datadoghq.com + host "http-intake.logs.#{ENV['DATADOG_API_HOST']}" flush_interval 1s # frequency of the buffer flush diff --git a/helm/artifactory-ha-values.yaml b/helm/artifactory-ha-values.yaml index ad9a024..5e5ff26 100644 --- a/helm/artifactory-ha-values.yaml +++ b/helm/artifactory-ha-values.yaml @@ -36,10 +36,13 @@ artifactory: value: {{ .Values.jfrog.observability.common_jpd | quote }} - name: DATADOG_API_KEY value: {{ .Values.datadog.api_key }} + - name: DATADOG_API_HOST + value: {{ .Values.datadog.api_host }} - name: FLUENTD_CONF value: ../../../../{{ .Values.artifactory.persistence.mountPath }}/etc/fluentd/fluentd.conf datadog: api_key: DATADOG_API_KEY + api_host: DATADOG_API_HOST jfrog: observability: jpd_url: JPD_URL diff --git a/helm/artifactory-values.yaml b/helm/artifactory-values.yaml index 4aab5e0..b1ba450 100644 --- a/helm/artifactory-values.yaml +++ b/helm/artifactory-values.yaml @@ -36,10 +36,13 @@ artifactory: value: {{ .Values.jfrog.observability.common_jpd | quote }} - name: DATADOG_API_KEY value: {{ .Values.datadog.api_key }} + - name: DATADOG_API_HOST + value: {{ .Values.datadog.api_host }} - name: FLUENTD_CONF value: ../../../../{{ .Values.artifactory.persistence.mountPath }}/etc/fluentd/fluentd.conf datadog: api_key: DATADOG_API_KEY + api_host: DATADOG_API_HOST jfrog: observability: jpd_url: JPD_URL diff --git a/helm/jfrog_helm.env b/helm/jfrog_helm.env index cadf6b6..f8aed1d 100644 --- a/helm/jfrog_helm.env +++ b/helm/jfrog_helm.env @@ -1,4 +1,5 @@ export DATADOG_API_KEY=DATADOG_API_KEY +export DATADOG_API_HOST=datadoghq.com export JPD_URL=http://abc.jfrog.io export JPD_ADMIN_USERNAME=admin -export COMMON_JPD=false \ No newline at end of file +export COMMON_JPD=false diff --git a/helm/xray-values.yaml b/helm/xray-values.yaml index 030e7c5..f418101 100644 --- a/helm/xray-values.yaml +++ b/helm/xray-values.yaml @@ -42,8 +42,11 @@ common: value: {{ .Values.jfrog.observability.common_jpd | quote }} - name: DATADOG_API_KEY value: {{ .Values.datadog.api_key }} + - name: DATADOG_API_HOST + value: {{ .Values.datadog.api_host }} datadog: api_key: DATADOG_API_KEY + api_host: DATADOG_API_HOST jfrog: observability: jpd_url: JPD_URL diff --git a/jfrog.env b/jfrog.env index 9fc1d69..0a5238b 100644 --- a/jfrog.env +++ b/jfrog.env @@ -1,6 +1,7 @@ export JF_PRODUCT_DATA_INTERNAL=JF_PRODUCT_DATA_INTERNAL export DATADOG_API_KEY=DATADOG_API_KEY +export DATADOG_API_HOST=datadoghq.com export JPD_URL=http://abc.jfrog.io export JPD_ADMIN_USERNAME=admin export JFROG_ADMIN_TOKEN=JFROG_ADMIN_TOKEN -export COMMON_JPD=false \ No newline at end of file +export COMMON_JPD=false