Improve docs about Prow self-CD.

prow/more_prow.md

@@ -12,13 +12,31 @@ Some Prow components expose prometheus metrics that can be used for monitoring,
 
 ## Make Prow update and deploy itself!
 
-You can easily make your Prow instance automatically update itself when changes are made to its component's kubernetes resource files. This is achieved with a postsubmit job that `kubectl apply`s the resource files whenever they are changed (based on a `run_if_changed` regexp). In order to `kubectl apply` to the cluster, the job will need to supply credentials (e.g. a kubeconfig file or [GCP service account key-file](/prow/gcloud-deployer-service-account.sh)). Since this job requires priviledged credentials to deploy to the cluster, it is important that it is run in a separate build cluster that is isolated from all presubmit jobs. See the [documentation about separate build clusters](/prow/scaling.md#separate-build-clusters) for details. An example of such a job can be found [here](https://github.com/istio/test-infra/blob/45526926b4f1cd09147d54d23abc4a4258e62860/prow/cluster/jobs/istio/test-infra/istio.test-infra.trusted.master.yaml#L2-L28).
-Once you have a postsubmit deploy job, any changes to Prow component files are automatically applied to the cluster when the changes merge.
+You can easily make your Prow instance automatically update itself when changes
+are made to its component's kubernetes resource files. This is achieved with a
+postsubmit job that `kubectl apply`s the resource files whenever they are
+changed (based on a `run_if_changed` regexp). In order to `kubectl apply` to the
+cluster, the job will need to supply credentials (e.g. a kubeconfig file or
+[GCP service account key-file](/prow/gcloud-deployer-service-account.sh)). Since
+this job requires priviledged credentials to deploy to the cluster, it is
+important that it is run in a separate build cluster that is isolated from all
+presubmit jobs. See the
+[documentation about separate build clusters](/prow/scaling.md#separate-build-clusters)
+for details. An example of such a job can be found
+[here](https://github.com/istio/test-infra/blob/45526926b4f1cd09147d54d23abc4a4258e62860/prow/cluster/jobs/istio/test-infra/istio.test-infra.trusted.master.yaml#L2-L28).
+Once you have a postsubmit deploy job, any changes to Prow component files are
+automatically applied to the cluster when the changes merge. In order to ensure
+that all changes to production are properly approved, you can use OWNERS files
+with the [`approve` plugin](/prow/plugins/approve) and [`Tide`](/prow/cmd/tide).
 
 With the help of the [Prow Autobump utility](/prow/cmd/autobump#prow-autobump) you can easily create commits that update all references to Prow images to the latest image version that has been vetted by the https://prow.k8s.io instance. If your Prow component resource files live in GitHub, this utility can even automatically create/update a Pull Request that includes these changes. This works great when run as a periodic job since it will maintain a single open PR that is periodically updated to reference the most recent upstream version. See the [README](/prow/cmd/autobump#prow-autobump) for details and an example.
 
 Combining a postsubmit deploy job with a periodic job that runs the Prow Autobump utility allows Prow to be updated to the latest version by simply merging the automatically created Pull Request (or letting Tide merge it after it has been approved).
 
+### Deploy config changes automatically
+
+Prow can also automatically upload changes to files that correspond to Kubernetes ConfigMaps. This includes its own `config`, `plugins` and `job-config` config maps. Take a look at the [`updateconfig` plugin](/prow/plugins/updateconfig) and [`config-bootstrapper`](/prow/cmd/config-bootstrapper) for more details. Both of these tools share the [`updateconfig` plugin's plugin configuration](https://github.com/kubernetes/test-infra/blob/531f2a5e6b6fb60e3262340a86992029aa59808f/prow/plugins/config.go#L69). The plugin provides slightly better GitHub integration and is simpler to enable, but the config-bootstrapper is more flexible. It can be run in a postsubmit job to provide config upload on non-GitHub Prow instances or run after custom config generation is executed.
+
 ## Use other tools with Prow
 
 * If you find that your GitHub bot is running low on API tokens consider using [`ghproxy`](/ghproxy) to cache requests to GitHub and take advantage of the strange re-validation rules that allow for additional API token savings.

prow/scaling.md

@@ -10,14 +10,26 @@ You may also be interested in ["Getting more out of Prow"](/prow/more_prow.md).
 
 It is frequently not secure to run all ProwJobs in the same cluster that runs
 Prow's service components (`hook`, `plank`, etc.). In particular, ProwJobs that
-execute presubmit tests should typically be isolated from Prow's microservices.
-This isolation prevents a malicious PR author from modifying the presubmit test
-to do something evil like breaking out of the container and stealing secrets
-that live in the cluster.
-
-More than one build cluster can be used in order to isolate specific jobs from
-each other, improve scalability, and offer different node shapes.
-Instructions for configuring jobs to run in different clusters can be found [here](/prow/getting_started_deploy.md#Run-test-pods-in-different-clusters).
+execute presubmit tests for OSS projects should typically be isolated from
+Prow's microservices. This isolation prevents a malicious PR author from
+modifying the presubmit test to do something evil like breaking out of the
+container and stealing secrets that live in the cluster or DOSing a
+cluster-internal Prow component service.
+
+Any number of build clusters can be used in order to isolate specific jobs from
+each other, improve scalability, or allow tenants to provide and manage their
+own execution environments. Instructions for configuring jobs to run in
+different clusters can be found
+[here](/prow/getting_started_deploy.md#Run-test-pods-in-different-clusters).
+
+Production Prow instances should run most ProwJobs in a build cluster separate
+from the Prow service cluster (the cluster where the Prow components live). Any
+'trusted' jobs that require secrets or services that should not be exposed to
+presubmit jobs, such as publishing or deployment jobs, should run in a different
+cluster from the rest of the 'untrusted' jobs.
+It is common for the Prow service cluster to be reused as a build cluster for
+these 'trusted' jobs since they are typically fast and few in number so running
+and managing an additional build cluster would be wasteful.
 
 ### Pull Request Merge Automation
 
@@ -32,13 +44,24 @@ batch whenever possible.
 ### Config File Split
 
 If your Prow config starts to grow too large, consider splitting the job config
-files into more specific and easily reviewed files. To use this pattern simply
-aggregate all job configs in a directory of files with unique base names and
-supply the directory path to components via `--job-config-path`. 
-
-The [`updateconfig` plugin](/prow/plugins/updateconfig) supports this pattern by
-allowing multiple files to loaded into a single configmap under different keys
-(different files once mounted to a container).
+files into more specific and easily reviewed files. This is particularly useful
+for delegating ownership of ProwJob config to different users or groups via the
+use of OWNERS files with the [`approve` plugin](/prow/plugins/approve) and
+[`Tide`](/prow/cmd/tide). It is common to enforce custom config policies for
+jobs defined in certain files or directories via presubmit unit tests. This
+makes it safe for Prow admins to delegate job config ownership by enforcing
+limitations on what can be configured and by whom. For example, we use a golang
+unit test in a presubmit job to validate that all jobs that are configured to
+run in the `test-infra-trusted` build cluster are defined in a file controlled
+by test-infra oncall.
+([examples](https://github.com/kubernetes/test-infra/tree/5c388ffe5e45f44ac4b46a0d25e941d7fe22b126/config/tests/jobs))
+
+To use this pattern simply aggregate all job configs in a directory of files
+with unique base names and supply the directory path to components via
+`--job-config-path`. The [`updateconfig` plugin](/prow/plugins/updateconfig) and
+[`config-bootstrapper`](/prow/cmd/config-bootstrapper) support this pattern by
+allowing multiple files to be loaded into a single configmap under different
+keys (different files once mounted to a container).
 
 ### GitHub API Cache