diff --git a/.gitignore b/.gitignore index 777fbf123..500626375 100644 --- a/.gitignore +++ b/.gitignore @@ -14,4 +14,4 @@ config-terraform.sh *.out -example/*.secrets*.tfvars +*.secrets*.tfvars diff --git a/CHANGELOG.md b/CHANGELOG.md index f25effedf..bfd5cbfd0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,9 @@ and this project adheres to [Semantic Versioning](http://semver.org/). ## [Unreleased] +## [3.0.0] - 2019-03-29 +- Changed: The runner will register itself based on the registration token. No need to preregister the runner before running terraform. See the [README](README.md) for configuration and migration. #33 + ## [2.3.0] - 2019-03-27 - Bugfix: Added a profile for the docker machine runners. #41 - Changed: Changed the name of runner instance, added `docker-machine` to the name. @@ -102,7 +105,8 @@ and this project adheres to [Semantic Versioning](http://semver.org/). - Update default AMI's to The latest Amazon Linux AMI 2017.09.1 - released on 2018-01-17. - Minor updates in the example -[Unreleased]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/2.3.0...HEAD +[Unreleased]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/3.0.0...HEAD +[3.0.0]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/2.3.0...3.0.0 [2.3.0]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/2.2.1...2.3.0 [2.2.1]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/2.2.0...2.2.1 [2.2.0]: https://github.com/npalm/terraform-aws-gitlab-runner/compare/2.1.0...2.2.0 diff --git a/README.md b/README.md index 660487ff8..d3094ba69 100644 --- a/README.md +++ b/README.md @@ -1,21 +1,23 @@ [![Build Status](https://travis-ci.com/npalm/terraform-aws-gitlab-runner.svg?branch=master)](https://travis-ci.com/npalm/terraform-aws-gitlab-runner) [![Gitter](https://badges.gitter.im/terraform-aws-gitlab-runner/Lobby.svg)](https://gitter.im/terraform-aws-gitlab-runner/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) -# Terraform module for GitLab auto scaling runners on Spot instances +# Terraform module for GitLab auto scaling runners on AWS spot instances -This repo contains a terraform module and example to run a [GitLab CI multi runner](https://docs.gitlab.com/runner/) on AWS Spot instances. See the blog post at [040code](https://040code.github.io/2017/12/09/runners-on-the-spot/) for a detailed description of the setup. +> *NEW*: The runner will register itself automatically to GitLab. No need to register the runner first, see also the [examples](./examples) + +This repo contains a Terraform module and examples to run a [GitLab CI multi runner](https://docs.gitlab.com/runner/) on AWS Spot instances. See the blog post at [040code](https://040code.github.io/2017/12/09/runners-on-the-spot/) for a detailed description of the setup. ![GitLab Runners](https://github.com/npalm/assets/raw/master/images/2017-12-06_gitlab-multi-runner-aws.png) -The setup is based on the blog post: [Auto scale GitLab CI runners and save 90% on EC2 costs](https://about.gitlab.com/2017/11/23/autoscale-ci-runners/) The created runner will have by default a shared cache in S3 and logging is streamed to CloudWatch. The cache in S3 will expire in X days, see configuration. The logging can be disabled. +The setup is based on the blog post: [Auto scale GitLab CI runners and save 90% on EC2 costs](https://about.gitlab.com/2017/11/23/autoscale-ci-runners/) The created runner will have by default a shared cache in S3 and logging is streamed to CloudWatch. The cache in S3 will expire in X days, see configuration. The logging can be disabled. The post mentioned that you have to register the the runner before running the Terraform scripts. Since version 3+ this is not needed anymore you can pass the runner configuration including the runner registration token. -Besides the auto scaling option (docker+machine executor) the docker executor is supported as wel for a single node. +Besides the auto scaling option (docker+machine executor) the docker executor is supported as well for a single node. ## Prerequisites ### Terraform -Ensure you have Terraform installed, see `.terraform-version` for the used version. A handy tool to mange your terraform version is [tfenv](https://github.com/kamatama41/tfenv). +Ensure you have Terraform installed, see `.terraform-version` for the used version. A handy tool to mange your Terraform version is [tfenv](https://github.com/kamatama41/tfenv). On mac simple install tfenv using brew. @@ -23,7 +25,7 @@ On mac simple install tfenv using brew. brew install tfenv ``` -Next install a terraform version. +Next install a Terraform version. ```sh tfenv install @@ -31,7 +33,7 @@ tfenv install ### AWS -To run the terraform scripts you need to have AWS keys. +To run the Terraform scripts you need to have AWS keys. Example file: ```sh @@ -41,12 +43,12 @@ export AWS_SECRET_ACCESS_KEY=... ### Service linked roles -The gitlab runner ec2 instance needs the following sercice linked roles: +The gitlab runner EC2 instance needs the following service linked roles: - AWSServiceRoleForAutoScaling - AWSServiceRoleForEC2Spot -By default the ec2 instance is allowed to create the roles, by setting the option `allow_iam_service_linked_role_creation` to `false` you can deny the creation of roles by the instance. In that case you have to ensure the roles exists. You can create them manually or via terraform. +By default the EC2 instance is allowed to create the roles, by setting the option `allow_iam_service_linked_role_creation` to `false` you can deny the creation of roles by the instance. In that case you have to ensure the roles exists. You can create them manually or via terraform. ```hcl resource "aws_iam_service_linked_role" "spot" { @@ -60,13 +62,36 @@ resource "aws_iam_service_linked_role" "autoscaling" { ### Configuration GitLab runner token -Currently register a new runner is a manual process. See the GitLab Runner [documentation](https://docs.gitlab.com/runner/register/index.html#docker) for more details. +By default the runner is registered the first time. In previous version this was a manual process. The manual process is still supported but will be removed in future releases. The runner token will be stored in the parameter store. See [example](examples/runner-pre-registered/) for more details. -```sh -docker run -it --rm gitlab/gitlab-runner register +To register the runner automatically set the variable `gitlab_runner_registration_config["token"]` which you can find in your GitLab project, group or global settings. For a generic runner you can find the token in the admin section. By default the runner will be locked to project, not run untagged. Below an example of the configuration map. + +``` + gitlab_runner_registration_config = { + registration_token = "" + tag_list = " +token= +parameter-name=<${var.environment}>-<${var.secure_parameter_store_runner_token_key}> + +aws ssm put-parameter --overwrite --type SecureString --name "${parameter-name}" --value ${token} --region "${aws-region}" ``` +Once you have created the parameter, you have to remove the variable `runners_token` from your config. Then next time your gitlab runner instance is created it look up the token from the parameter store. + +Finally the runner still support the manual runner creation, no changes are required. Please keep in mind that this setup will be removed in future releases. -Provide the details in the interactive terminal. Once done the token can be found in the GitLab runners section, choose edit to get the token or see the config.toml file. ## Usage @@ -76,8 +101,8 @@ Update the variables in `terraform.tfvars` to your needs and add the following v ```hcl runner_name = "NAME_OF_YOUR_RUNNER" -gitlab_url = "GIT_LAB_URL" -runner_token = "RUNNER_TOKEN" +gitlab_url = "GITLAB_URL" +runner_token = "RUNNER_TOKEN" ``` The base image used to host the GitLab Runner agent is the latest available Amazon Linux HVM EBS AMI. In previous version of the module an hard coded list of AMI per region was available. This list is replaced by a search filter to find the latest AMI. By setting the filter for example to `amzn-ami-hvm-2018.03.0.20180622-x86_64-ebs` you can lock the version of the AMI. @@ -86,82 +111,97 @@ The base image used to host the GitLab Runner agent is the latest available Amaz ### Usage module. ```hcl -module "gitlab-runner" { +module "runner" { source = "npalm/gitlab-runner/aws" - version = "1.0.0" + version = "3.0.0" - aws_region = "${var.aws_region}" - environment = "${var.environment}" - ssh_public_key = "${file("${var.ssh_key_file}")}" + aws_region = "${var.aws_region}" + environment = "${var.environment}" + ssh_public_key = "${file("${var.ssh_key_file}")}" - vpc_id = "${module.vpc.vpc_id}" + vpc_id = "${module.vpc.vpc_id}" subnet_ids_gitlab_runner = "${module.vpc.private_subnets}" - subnet_id_runners = "${element(module.vpc.private_subnets, 0)}" - - runners_name = "${var.runner_name}" - runners_gitlab_url = "${var.gitlab_url}" - runners_token = "${var.runner_token}" - - # Optional - runners_off_peak_timezone = "Europe/Amsterdam" - runners_off_peak_periods = "[\"* * 0-9,17-23 * * mon-fri *\", \"* * * * * sat,sun *\"]" + subnet_id_runners = "${element(module.vpc.private_subnets, 0)}" + + runners_name = "my-spot-runner" + runners_gitlab_url = "https://www.gitlab.com" + + gitlab_runner_registration_config = { + registration_token = "" + tag_list = "docker_spot_runner" + description = "Docker AWS Spot runner" + locked_to_project = "true" + run_untagged = "false" + maximum_timeout = "3600" + } + + runners_off_peak_timezone = "Europe/Amsterdam" + runners_off_peak_idle_count = 0 + runners_off_peak_idle_time = 60 + + # working 9 to 5 :) + runners_off_peak_periods = "[\"* * 0-9,17-23 * * mon-fri *\", \"* * * * * sat,sun *\"]" } ``` + ## Inputs | Name | Description | Type | Default | Required | |------|-------------|:----:|:-----:|:-----:| -| allow_iam_service_linked_role_creation | Attach policy to runner instance to create service linked roles. | string | `true` | no | -| ami_filter | AMI filter to select the AMI used to host the gitlab runner agent. By default the pattern `amzn-ami-hvm-2018.03*-x86_64-ebs` is used for the name. Currently Amazon Linux 2 `amzn2-ami-hvm-2.0.????????-x86_64-ebs` looks *not* working for this configuration. | list | `` | no | -| ami_owners | A list of owners used to select the AMI for the instance. | list | `` | no | +| allow_iam_service_linked_role_creation | Boolean used to control attaching the policy to a runner instance to create service linked roles. | string | `true` | no | +| ami_filter | List of maps used to create the AMI filter for the Gitlab runner agent AMI. Currently Amazon Linux 2 `amzn2-ami-hvm-2.0.????????-x86_64-ebs` looks to *not* be working for this configuration. | list | `` | no | +| ami_owners | The list of owners used to select the AMI of Gitlab runner agent instances. | list | `` | no | | aws_region | AWS region. | string | - | yes | | aws_zone | AWS availability zone (typically 'a', 'b', or 'c'). | string | `a` | no | | cache_bucket_prefix | Prefix for s3 cache bucket name. | string | `` | no | | cache_expiration_days | Number of days before cache objects expires. | string | `1` | no | | cache_shared | Enables cache sharing between runners, false by default. | string | `false` | no | -| create_runners_iam_instance_profile | | string | `true` | no | +| create_runners_iam_instance_profile | Boolean to control the creation of the runners IAM instance profile | string | `true` | no | | docker_machine_instance_type | Instance type used for the instances hosting docker-machine. | string | `m4.large` | no | -| docker_machine_options | Additional to set options for docker machine. Each element of the list should be key and value. E.g. '["amazonec2-zone=a"]' | list | `` | no | +| docker_machine_options | List of additional options for the docker machine config. Each element of this list must be a key=value pair. E.g. '["amazonec2-zone=a"]' | list | `` | no | | docker_machine_spot_price_bid | Spot price bid. | string | `0.04` | no | -| docker_machine_user | User name for the user to create spot instances to host docker-machine. | string | `docker-machine` | no | +| docker_machine_user | Username of the user used to create the spot instances that host docker-machine. | string | `docker-machine` | no | | docker_machine_version | Version of docker-machine. | string | `0.16.1` | no | -| enable_cloudwatch_logging | Enable or disable the CloudWatch logging. | string | `1` | no | -| environment | A name that identifies the environment, will used as prefix and for tagging. | string | - | yes | -| gitlab_runner_version | Version for the gitlab runner. | string | `11.8.0` | no | -| instance_role_json | Instance role json for the runner agent ec2 instance to override the default. | string | `` | no | +| enable_cloudwatch_logging | Boolean used to enable or disable the CloudWatch logging. | string | `true` | no | +| enable_manage_gitlab_token | Manage the GitLab token in SSM, if `true` the token SSM parameter will be manged by terraform, a destroy removes the parameter for the runner token. When `false` the token will not be manged by terraform. The runner process will still store the token in SSM but a terraform destroy will not remove the token. | string | `true` | no | +| environment | A name that identifies the environment, used as prefix and for tagging. | string | - | yes | +| gitlab_runner_registration_config | Configuration used to register the runner. See the README for an example, or reference the examples in the examples directory of this repo. | map | `` | no | +| gitlab_runner_version | Version of the GitLab runner. | string | `11.8.0` | no | +| instance_role_json | Docker machine runner instance override policy, expected to be in JSON format. | string | `` | no | | instance_role_runner_json | Instance role json for the docker machine runners to override the default. | string | `` | no | -| instance_type | Instance type used for the gitlab-runner. | string | `t2.micro` | no | -| runners_concurrent | Concurrent value for the runners, will be used in the runner config.toml | string | `10` | no | -| runners_executor | The executor to use. Currently supports docker+machine or docker | string | `docker+machine` | no | -| runners_gitlab_url | URL of the gitlab instance to connect to. | string | - | yes | +| instance_type | Instance type used for the GitLab runner. | string | `t2.micro` | no | +| runners_concurrent | Concurrent value for the runners, will be used in the runner config.toml. | string | `10` | no | +| runners_executor | The executor to use. Currently supports `docker+machine` or `docker`. | string | `docker+machine` | no | +| runners_gitlab_url | URL of the GitLab instance to connect to. | string | - | yes | | runners_iam_instance_profile_name | IAM instance profile name of the runners, will be used in the runner config.toml | string | `` | no | -| runners_idle_count | Idle count of the runners, will be used in the runner config.toml | string | `0` | no | -| runners_idle_time | Idle time of the runners, will be used in the runner config.toml | string | `600` | no | +| runners_idle_count | Idle count of the runners, will be used in the runner config.toml. | string | `0` | no | +| runners_idle_time | Idle time of the runners, will be used in the runner config.toml. | string | `600` | no | | runners_image | Image to run builds, will be used in the runner config.toml | string | `docker:18.03.1-ce` | no | -| runners_limit | Limit for the runners, will be used in the runner config.toml | string | `0` | no | +| runners_limit | Limit for the runners, will be used in the runner config.toml. | string | `0` | no | | runners_monitoring | Enable detailed cloudwatch monitoring for spot instances. | string | `false` | no | -| runners_name | Name of the runner, will be used in the runner config.toml | string | - | yes | +| runners_name | Name of the runner, will be used in the runner config.toml. | string | - | yes | | runners_off_peak_idle_count | Off peak idle count of the runners, will be used in the runner config.toml. | string | `0` | no | | runners_off_peak_idle_time | Off peak idle time of the runners, will be used in the runner config.toml. | string | `0` | no | | runners_off_peak_periods | Off peak periods of the runners, will be used in the runner config.toml. | string | `` | no | | runners_off_peak_timezone | Off peak idle time zone of the runners, will be used in the runner config.toml. | string | `` | no | -| runners_output_limit | Set maximum build log size in kilobytes, by default set to 4096 (4MB) | string | `4096` | no | +| runners_output_limit | Sets the maximum build log size in kilobytes, by default set to 4096 (4MB) | string | `4096` | no | | runners_post_build_script | Commands to be executed on the Runner just after executing the build, but before executing after_script. | string | `` | no | | runners_pre_build_script | Script to execute in the pipeline just before the build, will be used in the runner config.toml | string | `` | no | | runners_pre_clone_script | Commands to be executed on the Runner before cloning the Git repository. this can be used to adjust the Git client configuration first, for example. | string | `` | no | -| runners_privilled | Runners will run in privilled mode, will be used in the runner config.toml | string | `true` | no | +| runners_privilled | Runners will run in privileged mode, will be used in the runner config.toml | string | `true` | no | | runners_request_concurrency | Limit number of concurrent requests for new jobs from GitLab (default 1) | string | `1` | no | -| runners_root_size | Runnner instance root size in GB. | string | `16` | no | -| runners_token | Token for the runner, will be used in the runner config.toml | string | - | yes | -| runners_use_private_address | Restrict runners to use only private address | string | `true` | no | -| ssh_public_key | Public SSH key used for the gitlab-runner ec2 instance. | string | - | yes | -| subnet_id_runners | Subnet used to hosts the docker-machine runners. | string | - | yes | -| subnet_ids_gitlab_runner | Subnet used for hosting the gitlab-runner. | list | - | yes | -| tags | Map of tags that will be added to created resources. By default resources will be tagged with name and environnemnt. | map | `` | no | -| userdata_post_install | User-data script snippet to insert after gitlab-runner install | string | `` | no | -| userdata_pre_install | User-data script snippet to insert before gitlab-runner install | string | `` | no | -| vpc_id | The VPC that is used for the instances. | string | - | yes | +| runners_root_size | Runner instance root size in GB. | string | `16` | no | +| runners_token | Token for the runner, will be used in the runner config.toml. | string | `__REPLACED_BY_USER_DATA__` | no | +| runners_use_private_address | Restrict runners to the use of a private IP address | string | `true` | no | +| secure_parameter_store_runner_token_key | The key name used store the Gitlab runner token in Secure Paramater Store | string | `runner-token` | no | +| ssh_public_key | Public SSH key used for the GitLab runner EC2 instance. | string | - | yes | +| subnet_id_runners | List of subnets used for hosting the gitlab-runners. | string | - | yes | +| subnet_ids_gitlab_runner | Subnet used for hosting the GitLab runner. | list | - | yes | +| tags | Map of tags that will be added to created resources. By default resources will be taggen with name and environemnt. | map | `` | no | +| userdata_post_install | User-data script snippet to insert after GitLab runner install | string | `` | no | +| userdata_pre_install | User-data script snippet to insert before GitLab runner install | string | `` | no | +| vpc_id | The target VPC for the docker-machine and runner instances. | string | - | yes | ## Outputs @@ -170,6 +210,7 @@ module "gitlab-runner" { | runner_agent_role | ARN of the rule used for the ec2 instance for the GitLab runner agent. | | runner_as_group_name | Name of the autoscaling group for the gitlab-runner instance | | runner_cache_bucket_arn | ARN of the S3 for the build cache. | +| runner_role | ARN of the rule used for the docker machine runners. | ## Example @@ -177,7 +218,7 @@ An example is provided, execute the following steps to run the sample. Ensure yo ### AWS keys -Keys are generated by terraform and stored in a directory `generated` in the example directory. +Keys are generated by Terraform and stored in a directory `generated` in the example directory. ### Configure GitLab @@ -191,7 +232,7 @@ Once done, lookup the token in GitLab and update the `terraform.tfvars` file. ## Create runner -Run `terraform init` to initialize terraform. Next you can run `terraform plan` to inspect the resources that will be created. +Run `terraform init` to initialise Terraform. Next you can run `terraform plan` to inspect the resources that will be created. To create the runner run: diff --git a/examples/.gitignore b/examples/.gitignore new file mode 100644 index 000000000..3f5ca68ad --- /dev/null +++ b/examples/.gitignore @@ -0,0 +1 @@ +terraform.tfvars diff --git a/examples/README.md b/examples/README.md new file mode 100644 index 000000000..2a094264c --- /dev/null +++ b/examples/README.md @@ -0,0 +1,8 @@ +# Examples + +Please see the readme in per example for more details: + +- [runner-default] The default setup, private subnet, auto register, runner on spot instances. +- [runner-public] Runner in a public subnet, auto register, runner on spot instances. +- [runner-docker] Runners run on the same instance as the agent. +- [runner-pre-registered] Runner needs to be preregistered, old setup DEPRECATED. diff --git a/examples/runner-default/README.md b/examples/runner-default/README.md index 7cb86509b..cc94b58fb 100644 --- a/examples/runner-default/README.md +++ b/examples/runner-default/README.md @@ -1,6 +1,6 @@ -# Example - Runner - Private subnets +# Example - Spot Runner - Private subnets -Example how create a gitlab runner, running in a private subnet. +Example how to run builds on spot instnaces in a private subent. ## Prerequisite The terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv). If you are not using tfenv please check `.terraform-version` for the tested version. diff --git a/examples/runner-default/main.tf b/examples/runner-default/main.tf index 0cb2fdba7..779e78bb0 100644 --- a/examples/runner-default/main.tf +++ b/examples/runner-default/main.tf @@ -1,6 +1,6 @@ module "vpc" { source = "terraform-aws-modules/vpc/aws" - version = "1.59.0" + version = "1.60.0" name = "vpc-${var.environment}" cidr = "10.0.0.0/16" @@ -11,6 +11,7 @@ module "vpc" { enable_nat_gateway = true single_nat_gateway = true + enable_s3_endpoint = true tags = { Environment = "${var.environment}" @@ -31,7 +32,15 @@ module "runner" { runners_name = "${var.runner_name}" runners_gitlab_url = "${var.gitlab_url}" - runners_token = "${var.runner_token}" + + gitlab_runner_registration_config = { + registration_token = "${var.registration_token}" + tag_list = "docker_spot_runner" + description = "runner default - auto" + locked_to_project = "true" + run_untagged = "false" + maximum_timeout = "3600" + } runners_off_peak_timezone = "Europe/Amsterdam" runners_off_peak_idle_count = 0 diff --git a/examples/runner-default/providers.tf b/examples/runner-default/providers.tf index 21363a663..8f3bc2aed 100644 --- a/examples/runner-default/providers.tf +++ b/examples/runner-default/providers.tf @@ -1,6 +1,6 @@ provider "aws" { region = "${var.aws_region}" - version = "2.2" + version = "2.4" } provider "template" { @@ -8,7 +8,7 @@ provider "template" { } provider "local" { - version = "1.1" + version = "1.2" } provider "null" { diff --git a/examples/runner-default/terraform.tfvars b/examples/runner-default/terraform.tfvars deleted file mode 100644 index 73723da7c..000000000 --- a/examples/runner-default/terraform.tfvars +++ /dev/null @@ -1,12 +0,0 @@ -key_name = "gitlab-runner" - -environment = "ci-test" - -aws_region = "eu-west-1" - -# Add the following variables: -runner_name = "docker.m3" - -gitlab_url = "https://gitlab.com" - -runner_token = "3939146918cced54ecf1dd08e6b87e" diff --git a/examples/runner-default/variables.tf b/examples/runner-default/variables.tf index bd7286242..c1693d39c 100644 --- a/examples/runner-default/variables.tf +++ b/examples/runner-default/variables.tf @@ -6,8 +6,8 @@ variable "aws_region" { variable "environment" { description = "A name that indentifies the environment, will used as prefix and for taggin." - default = "ci-runners" type = "string" + default = "runners-default" } variable "public_ssh_key_filename" { @@ -21,14 +21,13 @@ variable "private_ssh_key_filename" { variable "runner_name" { description = "Name of the runner, will be used in the runner config.toml" type = "string" + default = "default-auto" } variable "gitlab_url" { description = "URL of the gitlab instance to connect to." type = "string" + default = "https://gitlab.com" } -variable "runner_token" { - description = "Token for the runner, will be used in the runner config.toml" - type = "string" -} +variable "registration_token" {} diff --git a/examples/runner-docker/main.tf b/examples/runner-docker/main.tf index e9ae9cf84..6f1db551a 100644 --- a/examples/runner-docker/main.tf +++ b/examples/runner-docker/main.tf @@ -5,8 +5,9 @@ module "vpc" { name = "vpc-${var.environment}" cidr = "10.1.0.0/16" - azs = ["eu-west-1a"] - public_subnets = ["10.1.101.0/24"] + azs = ["eu-west-1a"] + public_subnets = ["10.1.101.0/24"] + enable_s3_endpoint = true tags = { Environment = "${var.environment}" @@ -30,5 +31,13 @@ module "runner" { runners_executor = "docker" runners_name = "${var.runner_name}" runners_gitlab_url = "${var.gitlab_url}" - runners_token = "${var.runner_token}" + + gitlab_runner_registration_config = { + registration_token = "${var.registration_token}" + tag_list = "docker_runner" + description = "runner docker - auto" + locked_to_project = "true" + run_untagged = "false" + maximum_timeout = "3600" + } } diff --git a/examples/runner-docker/providers.tf b/examples/runner-docker/providers.tf index 21363a663..8f3bc2aed 100644 --- a/examples/runner-docker/providers.tf +++ b/examples/runner-docker/providers.tf @@ -1,6 +1,6 @@ provider "aws" { region = "${var.aws_region}" - version = "2.2" + version = "2.4" } provider "template" { @@ -8,7 +8,7 @@ provider "template" { } provider "local" { - version = "1.1" + version = "1.2" } provider "null" { diff --git a/examples/runner-docker/terraform.tfvars b/examples/runner-docker/terraform.tfvars deleted file mode 100644 index 6b44a8979..000000000 --- a/examples/runner-docker/terraform.tfvars +++ /dev/null @@ -1,12 +0,0 @@ -key_name = "gitlab-runner" - -environment = "runner-docker" - -aws_region = "eu-west-1" - -# Add the following variables: -runner_name = "docker.m3" - -gitlab_url = "https://gitlab.com" - -runner_token = "3939146918cced54ecf1dd08e6b87e" diff --git a/examples/runner-docker/variables.tf b/examples/runner-docker/variables.tf index bd7286242..f73874535 100644 --- a/examples/runner-docker/variables.tf +++ b/examples/runner-docker/variables.tf @@ -6,7 +6,7 @@ variable "aws_region" { variable "environment" { description = "A name that indentifies the environment, will used as prefix and for taggin." - default = "ci-runners" + default = "runners-docker" type = "string" } @@ -21,14 +21,13 @@ variable "private_ssh_key_filename" { variable "runner_name" { description = "Name of the runner, will be used in the runner config.toml" type = "string" + default = "docker" } variable "gitlab_url" { description = "URL of the gitlab instance to connect to." type = "string" + default = "https://www.gitlab.com" } -variable "runner_token" { - description = "Token for the runner, will be used in the runner config.toml" - type = "string" -} +variable "registration_token" {} diff --git a/examples/runner-pre-registered/.terraform-version b/examples/runner-pre-registered/.terraform-version new file mode 100644 index 000000000..44ab23e43 --- /dev/null +++ b/examples/runner-pre-registered/.terraform-version @@ -0,0 +1 @@ +0.11.13 diff --git a/examples/runner-pre-registered/README.md b/examples/runner-pre-registered/README.md new file mode 100644 index 000000000..445ecb39d --- /dev/null +++ b/examples/runner-pre-registered/README.md @@ -0,0 +1,6 @@ +# Example - Spot Runner - Private subnets + +This is the previous default example. For this example you need to register the runner before running terraform and provide the runner token. Since version 3+ the runner can register itself by providing the registration token. This examples is to show case backwards compatibility. + +## Prerequisite +The terraform version is managed using [tfenv](https://github.com/Zordrak/tfenv). If you are not using tfenv please check `.terraform-version` for the tested version. diff --git a/examples/runner-pre-registered/generated/.gitkeep b/examples/runner-pre-registered/generated/.gitkeep new file mode 100644 index 000000000..e69de29bb diff --git a/examples/runner-pre-registered/key.tf b/examples/runner-pre-registered/key.tf new file mode 100644 index 000000000..84da24f1f --- /dev/null +++ b/examples/runner-pre-registered/key.tf @@ -0,0 +1,25 @@ +resource "tls_private_key" "ssh" { + algorithm = "RSA" +} + +resource "local_file" "public_ssh_key" { + depends_on = ["tls_private_key.ssh"] + + content = "${tls_private_key.ssh.public_key_openssh}" + filename = "${var.public_ssh_key_filename}" +} + +resource "local_file" "private_ssh_key" { + depends_on = ["tls_private_key.ssh"] + + content = "${tls_private_key.ssh.private_key_pem}" + filename = "${var.private_ssh_key_filename}" +} + +resource "null_resource" "file_permission" { + depends_on = ["local_file.private_ssh_key"] + + provisioner "local-exec" { + command = "${format("chmod 600 %s", var.private_ssh_key_filename)}" + } +} diff --git a/examples/runner-pre-registered/main.tf b/examples/runner-pre-registered/main.tf new file mode 100644 index 000000000..dd4699a24 --- /dev/null +++ b/examples/runner-pre-registered/main.tf @@ -0,0 +1,44 @@ +module "vpc" { + source = "terraform-aws-modules/vpc/aws" + version = "1.60.0" + + name = "vpc-${var.environment}" + cidr = "10.0.0.0/16" + + azs = ["eu-west-1a"] + private_subnets = ["10.0.1.0/24"] + public_subnets = ["10.0.101.0/24"] + + enable_nat_gateway = true + single_nat_gateway = true + + enable_s3_endpoint = true + + tags = { + Environment = "${var.environment}" + } +} + +module "runner" { + source = "../../" + + aws_region = "${var.aws_region}" + environment = "${var.environment}" + + ssh_public_key = "${local_file.public_ssh_key.content}" + + vpc_id = "${module.vpc.vpc_id}" + subnet_ids_gitlab_runner = "${module.vpc.private_subnets}" + subnet_id_runners = "${element(module.vpc.private_subnets, 0)}" + + runners_name = "${var.runner_name}" + runners_gitlab_url = "${var.gitlab_url}" + runners_token = "${var.runner_token}" + + runners_off_peak_timezone = "Europe/Amsterdam" + runners_off_peak_idle_count = 0 + runners_off_peak_idle_time = 60 + + # working 9 to 5 :) + runners_off_peak_periods = "[\"* * 0-9,17-23 * * mon-fri *\", \"* * * * * sat,sun *\"]" +} diff --git a/examples/runner-pre-registered/providers.tf b/examples/runner-pre-registered/providers.tf new file mode 100644 index 000000000..8f3bc2aed --- /dev/null +++ b/examples/runner-pre-registered/providers.tf @@ -0,0 +1,20 @@ +provider "aws" { + region = "${var.aws_region}" + version = "2.4" +} + +provider "template" { + version = "2.1" +} + +provider "local" { + version = "1.2" +} + +provider "null" { + version = "2.1" +} + +provider "tls" { + version = "1.2" +} diff --git a/examples/runner-pre-registered/variables.tf b/examples/runner-pre-registered/variables.tf new file mode 100644 index 000000000..bd7286242 --- /dev/null +++ b/examples/runner-pre-registered/variables.tf @@ -0,0 +1,34 @@ +variable "aws_region" { + description = "AWS region." + type = "string" + default = "eu-west-1" +} + +variable "environment" { + description = "A name that indentifies the environment, will used as prefix and for taggin." + default = "ci-runners" + type = "string" +} + +variable "public_ssh_key_filename" { + default = "generated/id_rsa.pub" +} + +variable "private_ssh_key_filename" { + default = "generated/id_rsa" +} + +variable "runner_name" { + description = "Name of the runner, will be used in the runner config.toml" + type = "string" +} + +variable "gitlab_url" { + description = "URL of the gitlab instance to connect to." + type = "string" +} + +variable "runner_token" { + description = "Token for the runner, will be used in the runner config.toml" + type = "string" +} diff --git a/examples/runner-public/README.md b/examples/runner-public/README.md index 5acd71f25..48d3faa62 100644 --- a/examples/runner-public/README.md +++ b/examples/runner-public/README.md @@ -1,4 +1,4 @@ -# Example - Runner - Public subnets +# Example - Spot Runner - Public subnets Example how create a gitlab runner, running in a public subnet. diff --git a/examples/runner-public/main.tf b/examples/runner-public/main.tf index f44d33f25..35d857184 100644 --- a/examples/runner-public/main.tf +++ b/examples/runner-public/main.tf @@ -1,6 +1,6 @@ module "vpc" { source = "terraform-aws-modules/vpc/aws" - version = "1.59.0" + version = "1.60.0" name = "vpc-${var.environment}" cidr = "10.1.0.0/16" @@ -30,5 +30,13 @@ module "runner" { runners_name = "${var.runner_name}" runners_gitlab_url = "${var.gitlab_url}" - runners_token = "${var.runner_token}" + + gitlab_runner_registration_config = { + registration_token = "${var.registration_token}" + tag_list = "docker_spot_runner" + description = "runner public - auto" + locked_to_project = "true" + run_untagged = "false" + maximum_timeout = "3600" + } } diff --git a/examples/runner-public/providers.tf b/examples/runner-public/providers.tf index 21363a663..8f3bc2aed 100644 --- a/examples/runner-public/providers.tf +++ b/examples/runner-public/providers.tf @@ -1,6 +1,6 @@ provider "aws" { region = "${var.aws_region}" - version = "2.2" + version = "2.4" } provider "template" { @@ -8,7 +8,7 @@ provider "template" { } provider "local" { - version = "1.1" + version = "1.2" } provider "null" { diff --git a/examples/runner-public/terraform.tfvars b/examples/runner-public/terraform.tfvars index 895b1cc1e..d9cf1f857 100644 --- a/examples/runner-public/terraform.tfvars +++ b/examples/runner-public/terraform.tfvars @@ -1,12 +1 @@ -key_name = "gitlab-runner" - -environment = "runner-public" - -aws_region = "eu-west-1" - -# Add the following variables: -runner_name = "docker.m3" - -gitlab_url = "https://gitlab.com" - -runner_token = "3939146918cced54ecf1dd08e6b87e" +registration_token = "6mf2Fu8n23DzhecKYzHL" diff --git a/examples/runner-public/variables.tf b/examples/runner-public/variables.tf index bd7286242..9d12cbbb1 100644 --- a/examples/runner-public/variables.tf +++ b/examples/runner-public/variables.tf @@ -6,7 +6,7 @@ variable "aws_region" { variable "environment" { description = "A name that indentifies the environment, will used as prefix and for taggin." - default = "ci-runners" + default = "runner-public" type = "string" } @@ -21,14 +21,13 @@ variable "private_ssh_key_filename" { variable "runner_name" { description = "Name of the runner, will be used in the runner config.toml" type = "string" + default = "public-auto" } variable "gitlab_url" { description = "URL of the gitlab instance to connect to." type = "string" + default = "https://gitlab.com" } -variable "runner_token" { - description = "Token for the runner, will be used in the runner config.toml" - type = "string" -} +variable "registration_token" {} diff --git a/main.tf b/main.tf index 74703525a..ca2800244 100644 --- a/main.tf +++ b/main.tf @@ -61,6 +61,23 @@ resource "aws_security_group_rule" "out_all" { security_group_id = "${aws_security_group.docker_machine.id}" } +# Parameter value is managed by the user-data script of the gitlab runner instance +resource "aws_ssm_parameter" "runner_registration_token" { + # count = "${var.manage_ssm_runner_token}" + + name = "${local.secure_parameter_store_runner_token_key}" + type = "SecureString" + value = "null" + + tags = "${local.tags}" + + lifecycle { + ignore_changes = [ + "value", + ] + } +} + data "template_file" "user_data" { template = "${file("${path.module}/template/user-data.tpl")}" @@ -82,19 +99,30 @@ data "template_file" "gitlab_runner" { template = "${file("${path.module}/template/gitlab-runner.tpl")}" vars { - gitlab_runner_version = "${var.gitlab_runner_version}" - docker_machine_version = "${var.docker_machine_version}" - runners_config = "${data.template_file.runners.rendered}" - runners_executor = "${var.runners_executor}" - pre_install = "${var.userdata_pre_install}" - post_install = "${var.userdata_post_install}" + gitlab_runner_version = "${var.gitlab_runner_version}" + docker_machine_version = "${var.docker_machine_version}" + runners_config = "${data.template_file.runners.rendered}" + runners_executor = "${var.runners_executor}" + pre_install = "${var.userdata_pre_install}" + post_install = "${var.userdata_post_install}" + runners_gitlab_url = "${var.runners_gitlab_url}" + runners_token = "${var.runners_token}" + secure_parameter_store_runner_token_key = "${local.secure_parameter_store_runner_token_key}" + secure_parameter_store_region = "${var.aws_region}" + gitlab_runner_registration_token = "${var.gitlab_runner_registration_config["registration_token"]}" + giltab_runner_description = "${var.gitlab_runner_registration_config["description"]}" + gitlab_runner_tag_list = "${var.gitlab_runner_registration_config["tag_list"]}" + gitlab_runner_locked_to_project = "${var.gitlab_runner_registration_config["locked_to_project"]}" + gitlab_runner_run_untagged = "${var.gitlab_runner_registration_config["run_untagged"]}" + gitlab_runner_maximum_timeout = "${var.gitlab_runner_registration_config["maximum_timeout"]}" } } locals { // Convert list to a string seperated and prepend by a comma - docker_machine_options_string = "${format(",%s", join(",", formatlist("%q", var.docker_machine_options)))}" - runners_off_peak_periods_string = "${var.runners_off_peak_periods == "" ? "" : format("OffPeakPeriods = %s", var.runners_off_peak_periods)}" + docker_machine_options_string = "${format(",%s", join(",", formatlist("%q", var.docker_machine_options)))}" + runners_off_peak_periods_string = "${var.runners_off_peak_periods == "" ? "" : format("OffPeakPeriods = %s", var.runners_off_peak_periods)}" + secure_parameter_store_runner_token_key = "${var.environment}-${var.secure_parameter_store_runner_token_key}" } data "template_file" "runners" { @@ -144,7 +172,6 @@ resource "aws_autoscaling_group" "gitlab_runner_instance" { name = "${var.environment}-as-group" vpc_zone_identifier = ["${var.subnet_ids_gitlab_runner}"] - # vpc_zone_identifier = ["${var.subnets}"] min_size = "1" max_size = "1" desired_capacity = "1" @@ -280,3 +307,27 @@ resource "aws_iam_role_policy_attachment" "service_linked_role" { role = "${aws_iam_role.instance.name}" policy_arn = "${aws_iam_policy.service_linked_role.arn}" } + +################################################################################ +### AWS Systems Manager access to store runner token once regsitered +################################################################################ +data "template_file" "ssm_policy" { + count = "${var.enable_manage_gitlab_token ? 1 : 0}" + + template = "${file("${path.module}/policies/instance-secure-parameter-role-policy.json")}" +} + +resource "aws_iam_policy" "ssm" { + count = "${var.enable_manage_gitlab_token ? 1 : 0}" + + name = "${var.environment}-ssm" + path = "/" + description = "Policy for runner token param access via SSM" + + policy = "${data.template_file.ssm_policy.rendered}" +} + +resource "aws_iam_role_policy_attachment" "ssm" { + role = "${aws_iam_role.instance.name}" + policy_arn = "${aws_iam_policy.ssm.arn}" +} diff --git a/policies/instance-secure-parameter-role-policy.json b/policies/instance-secure-parameter-role-policy.json new file mode 100644 index 000000000..0dbcea60a --- /dev/null +++ b/policies/instance-secure-parameter-role-policy.json @@ -0,0 +1,19 @@ +{ + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "ssm:PutParameter" + ], + "Resource": "*" + }, + { + "Effect": "Allow", + "Action": [ + "ssm:GetParameters" + ], + "Resource": "arn:aws:ssm:*" + } + ] +} \ No newline at end of file diff --git a/template/gitlab-runner.tpl b/template/gitlab-runner.tpl index c83256da9..31b6d3039 100644 --- a/template/gitlab-runner.tpl +++ b/template/gitlab-runner.tpl @@ -21,6 +21,23 @@ curl -L https://github.com/docker/machine/releases/download/v${docker_machine_ve cp /tmp/docker-machine /usr/local/bin/docker-machine && \ ln -s /usr/local/bin/docker-machine /usr/bin/docker-machine + +token=$(aws ssm get-parameters --names "${secure_parameter_store_runner_token_key}" --with-decryption --region "${secure_parameter_store_region}" | jq -r ".Parameters | .[0] | .Value") +if [[ `echo ${runners_token}` == "__REPLACED_BY_USER_DATA__" && `echo $token` == "null" ]] +then + token=$(curl --request POST -L "${runners_gitlab_url}/api/v4/runners" \ + --form "token=${gitlab_runner_registration_token}" \ + --form "tag_list=${gitlab_runner_tag_list}" \ + --form "description=${giltab_runner_description}" \ + --form "locked=${gitlab_runner_locked_to_project}" \ + --form "run_untagged=${gitlab_runner_run_untagged}" \ + --form "maximum_timeout=${gitlab_runner_maximum_timeout}" \ + | jq -r .token) + aws ssm put-parameter --overwrite --type SecureString --name "${secure_parameter_store_runner_token_key}" --value $token --region "${secure_parameter_store_region}" +fi + +sed -i.bak s/__REPLACED_BY_USER_DATA__/`echo $token`/g /etc/gitlab-runner/config.toml + ${post_install} service gitlab-runner restart diff --git a/template/logging.tpl b/template/logging.tpl index c96cd7490..0f984ab5e 100644 --- a/template/logging.tpl +++ b/template/logging.tpl @@ -24,6 +24,12 @@ log_group_name = ${environment} datetime_format = %b %d %H:%M:%S initial_position = start_of_file +[/var/log/user-data.log] +file = /var/log/user-data.log +log_stream_name = {instanceId}/user-data +log_group_name = ${environment} +initial_position = start_of_file + EOF # Set the region to send CloudWatch Logs data to (the region where the instance is located) diff --git a/variables.tf b/variables.tf index 8ee498986..203a4c120 100644 --- a/variables.tf +++ b/variables.tf @@ -10,33 +10,33 @@ variable "aws_zone" { } variable "environment" { - description = "A name that identifies the environment, will used as prefix and for tagging." + description = "A name that identifies the environment, used as prefix and for tagging." type = "string" } variable "vpc_id" { - description = "The VPC that is used for the instances." + description = "The target VPC for the docker-machine and runner instances." type = "string" } variable "subnet_id_runners" { - description = "Subnet used to hosts the docker-machine runners." + description = "List of subnets used for hosting the gitlab-runners." type = "string" } variable "subnet_ids_gitlab_runner" { - description = "Subnet used for hosting the gitlab-runner." + description = "Subnet used for hosting the GitLab runner." type = "list" } variable "instance_type" { - description = "Instance type used for the gitlab-runner." + description = "Instance type used for the GitLab runner." type = "string" default = "t2.micro" } variable "ssh_public_key" { - description = "Public SSH key used for the gitlab-runner ec2 instance." + description = "Public SSH key used for the GitLab runner EC2 instance." type = "string" } @@ -56,43 +56,44 @@ variable "docker_machine_version" { } variable "runners_name" { - description = "Name of the runner, will be used in the runner config.toml" + description = "Name of the runner, will be used in the runner config.toml." type = "string" } variable "runners_executor" { - description = "The executor to use. Currently supports docker+machine or docker" + description = "The executor to use. Currently supports `docker+machine` or `docker`." type = "string" default = "docker+machine" } variable "runners_gitlab_url" { - description = "URL of the gitlab instance to connect to." + description = "URL of the GitLab instance to connect to." type = "string" } variable "runners_token" { - description = "Token for the runner, will be used in the runner config.toml" + description = "Token for the runner, will be used in the runner config.toml." type = "string" + default = "__REPLACED_BY_USER_DATA__" } variable "runners_limit" { - description = "Limit for the runners, will be used in the runner config.toml" + description = "Limit for the runners, will be used in the runner config.toml." default = 0 } variable "runners_concurrent" { - description = "Concurrent value for the runners, will be used in the runner config.toml" + description = "Concurrent value for the runners, will be used in the runner config.toml." default = 10 } variable "runners_idle_time" { - description = "Idle time of the runners, will be used in the runner config.toml" + description = "Idle time of the runners, will be used in the runner config.toml." default = 600 } variable "runners_idle_count" { - description = "Idle count of the runners, will be used in the runner config.toml" + description = "Idle count of the runners, will be used in the runner config.toml." default = 0 } @@ -103,7 +104,7 @@ variable "runners_image" { } variable "runners_privilled" { - description = "Runners will run in privilled mode, will be used in the runner config.toml" + description = "Runners will run in privileged mode, will be used in the runner config.toml" type = "string" default = "true" } @@ -135,12 +136,13 @@ variable "runners_off_peak_periods" { } variable "runners_root_size" { - description = "Runnner instance root size in GB." + description = "Runner instance root size in GB." default = 16 } variable "create_runners_iam_instance_profile" { - default = true + description = "Boolean to control the creation of the runners IAM instance profile" + default = true } variable "runners_iam_instance_profile_name" { @@ -173,29 +175,29 @@ variable "runners_request_concurrency" { } variable "runners_output_limit" { - description = "Set maximum build log size in kilobytes, by default set to 4096 (4MB)" + description = "Sets the maximum build log size in kilobytes, by default set to 4096 (4MB)" default = "4096" } variable "userdata_pre_install" { - description = "User-data script snippet to insert before gitlab-runner install" + description = "User-data script snippet to insert before GitLab runner install" type = "string" default = "" } variable "userdata_post_install" { - description = "User-data script snippet to insert after gitlab-runner install" + description = "User-data script snippet to insert after GitLab runner install" type = "string" default = "" } variable "runners_use_private_address" { - description = "Restrict runners to use only private address" + description = "Restrict runners to the use of a private IP address" default = "true" } variable "docker_machine_user" { - description = "User name for the user to create spot instances to host docker-machine." + description = "Username of the user used to create the spot instances that host docker-machine." type = "string" default = "docker-machine" } @@ -218,14 +220,14 @@ variable "cache_shared" { } variable "gitlab_runner_version" { - description = "Version for the gitlab runner." + description = "Version of the GitLab runner." type = "string" default = "11.8.0" } variable "enable_cloudwatch_logging" { - description = "Enable or disable the CloudWatch logging." - default = 1 + description = "Boolean used to enable or disable the CloudWatch logging." + default = true } variable "tags" { @@ -235,18 +237,18 @@ variable "tags" { } variable "allow_iam_service_linked_role_creation" { - description = "Attach policy to runner instance to create service linked roles." + description = "Boolean used to control attaching the policy to a runner instance to create service linked roles." default = true } variable "docker_machine_options" { - description = "Additional to set options for docker machine. Each element of the list should be key and value. E.g. '[\"amazonec2-zone=a\"]'" + description = "List of additional options for the docker machine config. Each element of this list must be a key=value pair. E.g. '[\"amazonec2-zone=a\"]'" type = "list" default = [] } variable "instance_role_json" { - description = "Instance role json for the runner agent ec2 instance to override the default." + description = "Docker machine runner instance override policy, expected to be in JSON format." type = "string" default = "" } @@ -258,7 +260,7 @@ variable "instance_role_runner_json" { } variable "ami_filter" { - description = "AMI filter to select the AMI used to host the gitlab runner agent. By default the pattern `amzn-ami-hvm-2018.03*-x86_64-ebs` is used for the name. Currently Amazon Linux 2 `amzn2-ami-hvm-2.0.????????-x86_64-ebs` looks *not* working for this configuration." + description = "List of maps used to create the AMI filter for the Gitlab runner agent AMI. Currently Amazon Linux 2 `amzn2-ami-hvm-2.0.????????-x86_64-ebs` looks to *not* be working for this configuration." type = "list" default = [{ @@ -268,7 +270,32 @@ variable "ami_filter" { } variable "ami_owners" { - description = "A list of owners used to select the AMI for the instance." + description = "The list of owners used to select the AMI of Gitlab runner agent instances." type = "list" default = ["amazon"] } + +variable "gitlab_runner_registration_config" { + description = "Configuration used to register the runner. See the README for an example, or reference the examples in the examples directory of this repo." + type = "map" + + default = { + registration_token = "" + tag_list = "" + description = "" + locked_to_project = "" + run_untagged = "" + maximum_timeout = "" + } +} + +variable "secure_parameter_store_runner_token_key" { + type = "string" + description = "The key name used store the Gitlab runner token in Secure Paramater Store" + default = "runner-token" +} + +variable "enable_manage_gitlab_token" { + description = "Manage the GitLab token in SSM, if `true` the token SSM parameter will be manged by terraform, a destroy removes the parameter for the runner token. When `false` the token will not be manged by terraform. The runner process will still store the token in SSM but a terraform destroy will not remove the token." + default = true +}