Merge pull request #3 from flanksource/docupdate

Doc update

Pipfile

@@ -0,0 +1,20 @@
+[[source]]
+url = "https://pypi.org/simple"
+verify_ssl = true
+name = "pypi"
+
+[packages]
+jinja2 = "==3.1.0"
+mkdocs = "*"
+mkdocs-material = "*"
+markdown = "*"
+mkdocs-autolinks-plugin = "*"
+mkdocs-material-extensions = "*"
+mkdocs-markdownextradata-plugin = "*"
+mkdocs-nav-enhancements = "*"
+mkdocs-spellcheck = "*"
+
+[dev-packages]
+
+[requires]
+python_version = "3.8"

docs/apm-hub/kubernetes.md

@@ -0,0 +1 @@
+By default the logging agent will be able to stream logs in realtime using the Kubernetes logging API.

docs/apm-hub/overview.md

@@ -0,0 +1 @@
+# APM H

docs/reference/configuration/aws.md → docs/config-db/aws.md

@@ -1,46 +1,45 @@
 ## AWS
-This config type is used to scrape information about your AWS infrastructure.
 
-??? example
-    ```yaml
-    aws:
-      - region:
-          - eu-west-2
-          - us-east-1
-          - af-south-1
-          - ap-south-1
-          - eu-central-1
-    compliance: true
-    patch_states: false
-    trusted_advisor_check: false
-    patch_details: false
-    cost_reporting:
-      s3_bucket_path: s3://flanksource-cost-reports/query-results
-      database: athenacurcfn_flanksource_report
-      table: flanksource_report
-      region: af-south-1
-    inventory: true
-    exclude:
-      - Amazon EC2 Reserved Instances Optimization
-      - Savings Plan
-    transform:
-      exclude:
-        - jsonpath: $.tags
-        - jsonpath: $.privateDnsNameOptionsOnLaunch
-        - jsonpath: outpostArn
-        - jsonpath: mapCustomerOwnedIpOnLaunch
-        - jsonpath: subnetArn
-    ```
+This config type is used to scrape information about your AWS infrastructure.
 
+```yaml
+aws:
+  - region:
+      - eu-west-2
+      - us-east-1
+      - af-south-1
+      - ap-south-1
+      - eu-central-1
+compliance: true
+patch_states: false
+trusted_advisor_check: false
+patch_details: false
+cost_reporting:
+  s3_bucket_path: s3://flanksource-cost-reports/query-results
+  database: athenacurcfn_flanksource_report
+  table: flanksource_report
+  region: af-south-1
+inventory: true
+exclude:
+  - Amazon EC2 Reserved Instances Optimization
+  - Savings Plan
+transform:
+  exclude:
+    - jsonpath: $.tags
+    - jsonpath: $.privateDnsNameOptionsOnLaunch
+    - jsonpath: outpostArn
+    - jsonpath: mapCustomerOwnedIpOnLaunch
+    - jsonpath: subnetArn
+```
 
 | Field | Description | Scheme | Required |
 | ----- | ----------- | ------ | -------- |
 | `compliance` | Toggle scraping of compliance metadata | *bool* |
 | `patch_states` | Set boolean value for scraping of patch state status, and report | *bool*  |
 | `trusted_advisor_check` | Set boolean value for enabling/disabling trusted advisor | *bool* |
 | `patch_details` | Set boolean value to enable/disable scraping of patch details | *bool* |  |
-| [`cost_reporting`](#CostReporting) | Specify cost reporting for scraping of data | [*CostReporting*](#costreporting)
-| [`cloud_trail`](#cloudtrail-cloudtrail) | Set CloudTrail specifications | [*CloudTrail*](#cloudtrail-cloudtrail) |  |
+| [`cost_reporting`](#cost-reporting) | Specify cost reporting for scraping of data | [*CostReporting*](#cost-reporting)
+| [`cloud_trail`](#cloudtrail-cloudtrail-cloudtrail) | Set CloudTrail specifications | [*CloudTrail*](#cloudtrail-cloudtrail) |  |
 | `include` | Specify AWS resources to include for scraping | *\[\]string* |
 | `exclude` | Specify AWS resources to exclude from scraping | *\[\]string* |
 | `inventory` | Toggle scrape of *metadata* for AWS resources | *bool* |
@@ -52,11 +51,30 @@ This config type is used to scrape information about your AWS infrastructure.
 | `exclude` | Set events to be excluded from scraping | *\[\]string* |
 | `max_age` | Set maximum age of events for scraping | *string* |
 
-### CostReporting (`cost_reporting`)
+### Cost Reporting (`cost_reporting`)
 
 | Field | Description | Scheme | Required |
 | ----- | ----------- | ------ | -------- |
 | `s3_bucket_path` | Set path for S3 bucket to scrape published AWS billing reports | *string* |
 | `table` | Specify table containing cost and usage data | *string* |
 | `database` | Specify database containing cost and usage data | *string* |
-| `region` | Specify region for S3 bucket | *string* |
+| `region` | Specify region for S3 bucket | *string* |
+
+### Supported Resources
+
+- Account
+- CloudTrail
+- Config Rules
+- Cost & Usage Reporting
+- EBS
+- EC2
+- ECR
+- EFS
+- EKS
+- IAM
+- Load Balancers
+- RDS
+- Route53
+- Subnet
+- Trusted Advisir
+- VPC

docs/config-db/azure-devops.md

@@ -0,0 +1,19 @@
+
+The Azure Devops  scanner will create a new configuration item for each unique pipeline (combination of name and variables).
+
+Each time the pipeline is run it will create a change for that configuration item.
+
+```yaml
+azureDevops:
+- organization: <Org Name>
+  personalAccessToken:
+    valueFrom:
+      secretKeyRef:
+        name: ado-credentials
+        key: TOKEN
+  projects:
+    - <Project Name> # leave empty to select all projects
+  pipelines:
+    - <Pipeline Name> # leave empty to select all pipelines
+  type: Release # the change type the pipeline produces
+```

docs/reference/configuration/file.md → docs/config-db/file.md

@@ -1,21 +1,20 @@
 ## File
-The file config type is set to scrape configurations or configurations in common with the related elements that can be specified in the fields; `type`, and `id`. The paths to the configuration(s) to be scraped is set with the field `path` as a list.
 
-??? example 
-    ```yaml
-      file:
-        - type: $.Config.InstanceType
-          id: $.Config.InstanceId
-          path:
-            - config*.json
-            - test*.json
-    ```
+The file config type is set to scrape configurations or configurations in common with the related elements that can be specified in the fields; `type`, and `id`. The paths to the configuration(s) to be scraped is set with the field `path` as a list.
 
+```yaml
+  file:
+    - type: $.Config.InstanceType
+      id: $.Config.InstanceId
+      path:
+        - config*.json
+        - test*.json
+```
 
 | Field | Description | Scheme | Required |
 | ----- | ----------- | ------ | -------- |
 | [**BaseScraper**](#BaseScraper) | Specify inline fields, `id`, `name`, `items`, `type`, `transform`, and `format` | [BaseScraper](#basescraper) |
-| `url` | Specify URL e.g github repository containing the configuration(s) | *string* | 
+| `url` | Specify URL e.g github repository containing the configuration(s) | *string* |
 | `path` | Specify path to configuration(s) for scraping | *string* |
 | `ignore` | Set configurations to ignore | *string* |
 
@@ -38,17 +37,17 @@ The file config type is set to scrape configurations or configurations in common
 | [`include`](#Filter) | Specify fields to include in the configuration | [*Filter*](#filter)  |  |
 | [`exclude`](#filter) | Specify fields to remove from the config. It is useful for removing sensitive data and fields | [*Filter](#filter) |  |
 
-### Filter 
+### Filter
 
 | Field | Description | Scheme | Required |
 | ----- | ----------- | ------ | -------- |
 | `jsonPath` | Specify JSONPath expression for the fields | *string* |
 
-### Script 
+### Script
 
 | Field | Description | Scheme | Required |
 | ----- | ----------- | ------ | -------- |
 | `gotemplate` | Specify Go template for use in script | *string* |
 | `jsonPath` | Specify path to JSON element for use in script | *string* |
 | `expr` | Specify expression for use in script | *string* |
-| `javascript` | Specify javascript syntax for script | *string* |
+| `javascript` | Specify javascript syntax for script | *string* |

docs/reference/configuration/kubernetes-file.md → docs/config-db/kubernetes-file.md

@@ -1,34 +1,37 @@
-## KubernetesFile 
+## KubernetesFile
+
 The KubernetesFile config type is used to scrape configurations contained in your specified resource e.g Pod.
 
-??? example 
-    ```yaml
-      kubernetesFile:
-        - selector:
-          namespace: demo
-          kind: Statefulset
-          name: postgresql
-        files:
-          - path:
-              - /var/lib/postgresql/data/pgdata/postgresql.conf
-            format: properties
-    ```
+```yaml
+  kubernetesFile:
+    - selector:
+      namespace: demo
+      kind: Statefulset
+      name: postgresql
+    files:
+      - path:
+          - /var/lib/postgresql/data/pgdata/postgresql.conf
+        format: properties
+```
 
 ### KubernetesFile (`kubernetesFile`)
+
 | Field | Description | Scheme | Required |
 | ----- | ----------- | ------ | -------- |
 | [**BaseScraper**](#basescraper) | Specify inline fields, `id`, `name`, `items`, `type`, `transform`, and `format` | [*BaseScraper*](#basescraper) | yes |
 | [**`selector`**](#resourceselector-selector) | Specify Kubernetes resource for configuration based on `namespace`, `kind`, `name` and more. | *string* | yes |
 | `container` | Set container name | *string* |
 | [`files`](#PodFile) | Specify path to file contained in Pod | *string* |
 
-### PodFile 
+### PodFile
+
 | Field | Description | Scheme | Required |
 | ----- | ----------- | ------ | -------- |
 | `path` | Set path to file contained in Pod | *\[\]string* |  |
 | `format` | Specify format of file | *string* |
 
 ### ResourceSelector (`selector`)
+
 | Field | Description | Scheme | Required |
 | ----- | ----------- | ------ | -------- |
 | `namespace` | Specify Namespace for Kubernetes resource | *string* |  |
@@ -56,8 +59,8 @@ The KubernetesFile config type is used to scrape configurations contained in you
 | [`include`](#Filter) | Specify fields to include in the configuration | [*Filter*](#filter)  |  |
 | [`exclude`](#filter) | Specify fields to remove from the config. It is useful for removing sensitive data and fields | [*Filter*](#filter) |  |
 
-### Filter 
+### Filter
 
 | Field | Description | Scheme | Required |
 | ----- | ----------- | ------ | -------- |
-| `jsonPath` | Specify JSONPath expression for the fields | *string* |
+| `jsonPath` | Specify JSONPath expression for the fields | *string* |

docs/reference/configuration/kubernetes.md → docs/config-db/kubernetes.md

@@ -1,51 +1,44 @@
-## Kubernetes 
-The `kubernetes` config type scrapes the configurations of your Kubernetes resources as specified with the fields; `namespace`, `selector`, `fieldSelector` and more. 
+## Kubernetes
+
+The `kubernetes` config type scrapes the configurations of your Kubernetes resources as specified with the fields; `namespace`, `selector`, `fieldSelector` and more.
+
+```yaml
+  kubernetes:
+    - exclusions:
+        - Secret
+        - ReplicaSet
+        - APIService
+        - events
+        - endpoints.discovery.k8s.io
+        - endpointslices.discovery.k8s.io
+        - leases.coordination.k8s.io
+        - podmetrics.metrics.k8s.io
+        - nodemetrics.metrics.k8s.io
+        - customresourcedefinition
+        - controllerrevision
+        - certificaterequest
+        - orders.acme.cert-manager.io
+```
 
-??? example
-
-    ```yaml
-      kubernetes:
-        - exclusions:
-            - Secret
-            - ReplicaSet
-            - APIService
-            - events
-            - endpoints.discovery.k8s.io
-            - endpointslices.discovery.k8s.io
-            - leases.coordination.k8s.io
-            - podmetrics.metrics.k8s.io
-            - nodemetrics.metrics.k8s.io
-            - customresourcedefinition
-            - controllerrevision
-            - certificaterequest
-            - orders.acme.cert-manager.io
-    ```
-
-### Kubernetes (`kubernetes`)
 | Field | Description | Scheme | Required |
 | ----- | ----------- | ------ | -------- |
-| [**BaseScraper**](#BaseScraper) | Specify inline fields, `id`, `name`, `items`, `type`, `transform`, and `format` | [***BaseScraper***](#basescraper) | *yes* |
+| `id` | A static value or JSONPath expression to use as the ID for the resource. | *string* |  |
+| `name` | A static value or JSONPath expression to use as the Name for the resource. | *string* |  |
+| `items` | A JSONPath expression to use to extract individual items from the resource | *string* |  |
+| `type` | A static value or JSONPath expression to use as the type for the resource. | *string* |  |
+| [`transform`](#transform) | Specify field to transform result | *string* |
+| `format` | Format of config item, defaults to JSON, available options are JSON | *string* |
 | `namespace` | Specify namespace for scraping of Kubernetes resources | *string* |
 | `useCache` | Specify boolean value to toggle fetching results from Kube-apiserver or fetch response from etcd | *bool* |
-| `allowIncomplete` | 
+| `allowIncomplete` |
 | `scope` | Specify scope for scrape. e.g `cluster` for scraping at Cluster level | *string* |  |
 | `since` | Set time constraint for scraping resources within the set period | *string* |  |
 | `selector` | Specify Kubernetes resource to scrape based on selector. e.g `matchLabels` | *string*  |
 | `fieldSelector` | Specify Kubernetes resource based on value of resource fields. e.g `status.Phase=Running` | *string* |
-| `maxInflight` | Set value for maximum inflight requests | *int* | 
+| `maxInflight` | Set value for maximum inflight requests | *int* |
 | `exclusions` | Specify Kubernetes resources to be excluded from scraping | *\[\]string* |
 | **`kubeconfig`** | Specify kubeconfig for access to your Kubernetes Cluster |[*kommons.EnvVar*](https://pkg.go.dev/github.com/flanksource/kommons#EnvVar) | yes |
 
-### BaseScraper
-| Field | Description | Scheme | Required |
-| ----- | ----------- | ------ | -------- |
-| `id` | A static value or JSONPath expression to use as the ID for the resource. | *string* |  |
-| `name` | A static value or JSONPath expression to use as the Name for the resource. | *string* |  |
-| `items` | A JSONPath expression to use to extract individual items from the resource | *string* |  |
-| `type` | A static value or JSONPath expression to use as the type for the resource. | *string* |  |
-| [`transform`](#transform) | Specify field to transform result | *string* |
-| `format` | Format of config item, defaults to JSON, available options are JSON | *string* |
-
 ### Transform
 
 | Field | Description | Scheme | Required |
@@ -54,13 +47,13 @@ The `kubernetes` config type scrapes the configurations of your Kubernetes resou
 | [`include`](#Filter) | Specify fields to include in the configuration | [*Filter*](#filter)  |  |
 | [`exclude`](#filter) | Specify fields to remove from the config. It is useful for removing sensitive data and fields | [*Filter*](#filter) |  |
 
-### Filter 
+### Filter
 
 | Field | Description | Scheme | Required |
 | ----- | ----------- | ------ | -------- |
 | `jsonPath` | Specify JSONPath expression for the fields | *string* |
 
-### Script 
+### Script
 
 | Field | Description | Scheme | Required |
 | ----- | ----------- | ------ | -------- |

docs/config-db/overview.md

@@ -0,0 +1,15 @@
+## Configuration
+
+Mission Control is able to scan multiple configuration sources including
+
+- [AWS Cloud Resources](./aws)
+- [Azure Devops](./azure-devops.md) - Azure Devops Pipeline runs
+- [Files](./file)  -On a local filesystem, git or HTTP)
+- [Files - Kubernetes](/kubernetes-file) - Files inside a running kubernetes pod
+- [SQL](./sql.md) - Data available via query on MySQL, SQL Server and Postgres databases
+
+Each configuration has:
+
+- Configuration - Normally JSON, but XML and properties files are also available
+- Insights - Security, cost, performance and other recommendations from scanners including AWS Trusted Adviosr, AWS Config rules, etc..
+- Changes - Either changes directly on the config (recorded as diff change type) or changes identified via AWS Cloudtrail, etc...