General refactoring and automatic config accumulators (sous-chefs#403)

* Tidy up templates * Automatic config template resource generation * Test fixup for auto template resources * Resources to unified mode * Fix resource property collision * Add extra_options to grafana_config Closes sous-chefs#400 * Concat to existing settings rather than overwriting them * Add LDAP log filters to the main config file Fixes sous-chefs#392 * Start service immediately during testing * Azure AD auth needs path override * Require Chef 16 for resource partials * Kitchen allow log level to be set * Remove deprecated API resources * Refactor config file accumulation As we've removed the deprecated API resource then we don't need to worry about config writing timing so we can use a standard accumulator pattern to build the config files. * Template nil catch * Delete config_writer resource * Remove redundant sleep test recipe * Remove redundant instance_name With the migration to an standard accumulator pattern the instance_name functionality can be achieved by overriding the configuration file or directory etc. * Library refactoring * Correct config_auth default key path * Test strip global section * Remove etc/share directory creation as we use packages * Resource formatting and update options for Grafana 8.1 * Unify accumulator config method with action property * Automatic LDAP configuration accumalator template * Install deepsort gem for chefspec * Fixup plug library references * Test refactoring for api resource removal * LDAP config load_current_value support * Split log resource into separate types * Split metrics resource into separate types * Split auth resource into separate types * All resources converge_if_changed_support * Rationalise config resource action naming * Add expression, geomap, feature_toggle and date_format resources * load_current_value for extra_options property * Correct config actions * Remove redundant default values to use grafana defaults instead * Add load_existing option to init_config_template Allow the loading of the current configuration state to create a persistent accumulator pattern where configuration is not removed if the resource creating it is removed. * Move common resource actions into partial * Extract config path override to action_class method This allows us to use the common resource actions and load_current_value from the partial in 99% of cases. * Remove header comment * Load current state of config files into templates on init * Compact and deep sort ini config output * Add debug logging to ldap_config_file helper * Update LDAP resource for load current state changes * Add comments to the libraries * Remove enforce_idempotency overrides from kitchen config * Change skip property constant to method * Global documentation update * Add development guide * Refactor plugin resource to use converge_by * Common config resource delete action * Resource format fixup * Add missing grafanacom path override * Documentation update * Add jaeger and base external image resources * Add missing config resource properties * Add resource property/grafana property translation feature * Markdown link fixup * Add Okta auth resource * Allow overriding of the Grafana package name Fixes sous-chefs#380 * Update README links default branch name * Move LDAP server exist checks into resources * Fix missing prefix delete for ldap attribute resource load_current_value * Add Fedora installation support * Docs fixup * Use ternary for delete properties * Refactor helpers for a single override/translate/skip resource method Co-authored-by: Jason Field <[email protected]>
simonasr · Oct 20, 2021 · 1ce9568 · 1ce9568
1 parent c2bf67c
commit 1ce9568
Show file tree

Hide file tree

Showing 172 changed files with 3,769 additions and 5,767 deletions.
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -15,6 +15,8 @@ jobs:
         uses: actions/checkout@v2
       - name: Run Chef Delivery
         uses: actionshub/chef-delivery@main
+        with:
+          gems: "deepsort"
         env:
           CHEF_LICENSE: accept-no-persist
 
@@ -45,6 +47,7 @@ jobs:
           - 'debian-10'
           - 'centos-7'
           - 'centos-8'
+          - 'fedora-latest'
           - 'ubuntu-1804'
           - 'ubuntu-2004'
         suite:

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -4,6 +4,16 @@ This file is used to list changes made in each version of grafana.
 
 ## Unreleased
 
+- Automatic config Hash path creation
+- Automatic resource property fetching
+- Automatic config template resource generation
+- Add service resource
+- Add LDAP log filters to the main config file
+- Split log resource into seperate resources
+- Split auth resource into seperate resources
+- Update for Grafana v8.0 configuration
+- Config accumulators are now persistent (explicit delete action required to remove)
+
 ## 9.7.1 - *2021-08-30*
 
 - Standardise files with files in sous-chefs/repo-management

diff --git a/DEVELOPING.md b/DEVELOPING.md
@@ -0,0 +1,69 @@
+# Developing
+
+The refactor for version 10 of this cookbook makes some major changes to the underlying method of operation of the cookbook to improve operation and maintainability.
+
+In summary:
+
+- Accumulator config hash initialisation is automatic upon initial access
+- Accumulator config hash paths are automatically generated from the resource name
+- Property enumeration is automatic so adding a property to resource will result in that property being added to the configuration Hash
+- Most config resource functionality has been moved to the partially-abstract `_config_file` and `_config_file_ldap` partial resources
+  - Barring edge cases, all `config_*` resources inherit from these partials and contain only their properties
+  - `load_current_value` has been implemented for all resources
+- The `extra_options` hash property has been added to allow the additional of arbitrary configuration options (all config resources)
+- Config files are now generated by the `inifile` and `toml-rb` gems so the templates are purely placeholders
+
+## Adding a new configuration item to an existing resource
+
+Adding a new configuration item to an existing resource is as simple as adding the required item as a property to resource representing the section of the configuration file.
+
+## Adding a new configuration section
+
+To add a new configuration section a `config_` resource should be created where the name following the `config_` prefix is the name of the configuration section to represent. Configuration items under the section are then added as properties to this resource.
+
+The configuration resource **must** then inherit from the `_config_base` partial template via the addition of
+
+```ruby
+use 'partial/_config_file'
+```
+
+At the top of the resource declaration above the first property.
+
+See a pre-existing resource for an example of this pattern.
+
+### Overriding the automatic path
+
+If the name of the resource with the `'grafana_config_` prefix removed does not match the name of the configuration section or a nested configuration section is in use such as `section.subsection` then the automatic configuration path can be overridden by the addition of the `:resource_config_path_override` method.
+
+After the last resource property declaration, add the following code with the array contents set to the config Hash path set as you would pass to [Hash#dig](https://ruby-doc.org/core-3.0.2/Hash.html#method-i-dig) to create the required configuration path. Multiple nested Hashes are supported.
+
+```ruby
+def resource_config_path_override
+  %w(external_image_storage.s3).freeze
+end
+
+```
+
+### Excluding properties from the configuration
+
+If an additional property must be added for the resource to function but the property is not relevant for the Grafana config file then these properties can be marked to be skipped by the enumerator by the creation of a `:resource_config_properties_skip` method.
+
+```ruby
+def resource_config_properties_skip
+  %i(host).freeze
+end
+```
+
+### Property/configuration item name translation
+
+There are some cases where a Grafana configuration property name either conflicts with one of the base properties (`config_auth_ldap` property `:config_file`) or is a name that is disallowed by Chef (`config_external_image_storage` property `:provider`). To work around this issue there is a property translation mechanism implemented that can be used to alias the chef property name to the Grafana property name and vice-versa.
+
+This is performed by the creation of a `:resource_config_properties_translate` method as shown below.
+
+```ruby
+def resource_config_properties_translate
+  {
+    storage_provider: 'provider',
+  }.freeze
+end
+```
diff --git a/README.md b/README.md
@@ -35,54 +35,61 @@ We supply many different configuration resources, these all rely on the base con
 For any LDAP the base config resource is: `grafana_config_ldap`
 For any core configuration resources, the base config resource is: `grafana_config`
 
-**NOTE**: Inorder to write the above configuration resources to the disk use [grafana_config_writer](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_writer.md) at the end.
+**NOTE**: Inorder to write the above configuration resources to the disk use [grafana_config_writer](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_writer.md) at the end.
 
 ## Resources
 
-- [grafana_config](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config.md)
-- [grafana_config_alerting](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_alerting.md)
-- [grafana_config_auth](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_auth.md)
-- [grafana_config_auth_azuread](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_auth_azuread.md)
-- [grafana_config_dashboards](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_dashboards.md)
-- [grafana_config_database](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_database.md)
-- [grafana_config_dataproxy](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_dataproxy.md)
-- [grafana_config_emails](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_emails.md)
-- [grafana_config_enterprise](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_enterprise.md)
-- [grafana_config_explore](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_explore.md)
-- [grafana_config_log](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_log.md)
-- [grafana_config_ldap](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_ldap.md)
-- [grafana_config_ldap_group_mappings](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_ldap_group_mappings.md)
-- [grafana_config_ldap_servers](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_ldap_servers.md)
-- [grafana_config_metrics](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_metrics.md)
-- [grafana_config_panels](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_panels.md)
-- [grafana_config_paths](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_paths.md)
-- [grafana_config_plugins](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_plugins.md)
-- [grafana_config_quota](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_quota.md)
-- [grafana_config_remote_cache](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_remote_cache.md)
-- [grafana_config_rendering](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_rendering.md)
-- [grafana_config_security](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_security.md)
-- [grafana_config_server](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_server.md)
-- [grafana_config_session](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_session.md)
-- [grafana_config_smtp](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_smtp.md)
-- [grafana_config_snapshots](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_snapshots.md)
-- [grafana\_config\_external\_image\_storage\_s3](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_external_image_storage_s3.md)
-- [grafana_config_users](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_users.md)
-- [grafana_config_writer](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_config_writer.md)
-- [grafana_cookie_name](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_cookie_name.md)
-- [grafana_dashboard](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_dashboard.md)
-- [grafana_dashboard_template](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_dashboard_template.md)
-- [grafana_dashboards_backup](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_dashboards_backup.md)
-- [grafana_datasource](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_datasource.md)
-- [grafana_datasources_backup](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_datasources_backup.md)
-- [grafana_folder](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_folder.md)
-- [grafana_install](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_install.md)
-- [grafana_organization](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_organization.md)
-- [grafana_plugin](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_plugin.md)
-- [grafana_user](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_user.md)
+- [grafana_config](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config.md)
+- [grafana_config_alerting](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_alerting.md)
+- [grafana_config_auth_anonymous](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_auth_anonymous.md)
+- [grafana_config_auth_azuread](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_auth_azuread.md)
+- [grafana_config_auth_basic](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_auth_basic.md)
+- [grafana_config_auth_generic](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_auth_generic.md)
+- [grafana_config_auth_github](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_auth_github.md)
+- [grafana_config_auth_google](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_auth_google.md)
+- [grafana_config_auth_grafanacom](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_auth_grafanacom.md)
+- [grafana_config_auth_grafananet](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_auth_grafananet.md)
+- [grafana_config_auth_ldap](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_auth_ldap.md)
+- [grafana_config_auth_okta](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_auth_okta.md)
+- [grafana_config_auth_proxy](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_auth_proxy.md)
+- [grafana_config_dashboards](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_dashboards.md)
+- [grafana_config_database](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_database.md)
+- [grafana_config_dataproxy](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_dataproxy.md)
+- [grafana_config_emails](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_emails.md)
+- [grafana_config_enterprise](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_enterprise.md)
+- [grafana_config_explore](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_explore.md)
+- [grafana_config_expressions](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_expressions.md)
+- [grafana_config_external_image_storage_azure_blob](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_external_image_storage_azure_blob.md)
+- [grafana_config_external_image_storage_gcs](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_external_image_storage_gcs.md)
+- [grafana_config_external_image_storage_s3](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_external_image_storage_s3.md)
+- [grafana_config_external_image_storage_webdav](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_external_image_storage_webdav.md)
+- [grafana_config_ldap_attributes](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_ldap_attributes.md)
+- [grafana_config_ldap_group_mapping](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_ldap_group_mapping.md)
+- [grafana_config_ldap_server](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_ldap_server.md)
+- [grafana_config_log_console](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_log_console.md)
+- [grafana_config_log_file](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_log_file.md)
+- [grafana_config_log_syslog](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_log_syslog.md)
+- [grafana_config_log](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_log.md)
+- [grafana_config_metrics](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_metrics.md)
+- [grafana_config_metrics_graphite](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_metrics_graphite.md)
+- [grafana_config_panels](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_panels.md)
+- [grafana_config_paths](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_paths.md)
+- [grafana_config_plugins](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_plugins.md)
+- [grafana_config_quota](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_quota.md)
+- [grafana_config_remote_cache](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_remote_cache.md)
+- [grafana_config_rendering](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_rendering.md)
+- [grafana_config_security](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_security.md)
+- [grafana_config_server](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_server.md)
+- [grafana_config_session](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_session.md)
+- [grafana_config_smtp](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_smtp.md)
+- [grafana_config_snapshots](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_snapshots.md)
+- [grafana_config_users](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_config_users.md)
+- [grafana_install](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_install.md)
+- [grafana_plugin](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_plugin.md)
 
 ## Note on default session cookie name change
 
-The default cookie name changed from Grafana [5.4.5](https://github.com/grafana/grafana/blob/v5.4.5/pkg/setting/setting.go#L743) to [6.0.0](https://github.com/grafana/grafana/blob/v6.0.0/pkg/setting/setting.go#L664).  The name change was from `grafana_sess` to `grafana_session`.  This cookbook now defaults to `grafana_session`.  This can be a breaking change, so please be aware of this.  Please see the [`grafana_cookie_name` documentation](https://github.com/sous-chefs/grafana/tree/master/documentation/grafana_cookie_name.md) for details
+The default cookie name changed from Grafana [5.4.5](https://github.com/grafana/grafana/blob/v5.4.5/pkg/setting/setting.go#L743) to [6.0.0](https://github.com/grafana/grafana/blob/v6.0.0/pkg/setting/setting.go#L664).  The name change was from `grafana_sess` to `grafana_session`.  This cookbook now defaults to `grafana_session`.  This can be a breaking change, so please be aware of this.  Please see the [`grafana_cookie_name` documentation](https://github.com/sous-chefs/grafana/tree/main/documentation/grafana_cookie_name.md) for details
 
 ## Contributors
 

diff --git a/UPGRADING.md b/UPGRADING.md
@@ -1,5 +1,23 @@
 # Upgrading
 
+## 10.0.0
+
+### Version 10 is a major internal rewrite
+
+- All API resources have been removed
+- All resources have been refactored around an abstracted common base resource `_config_file`
+  - LDAP resources extend this via `_config_file_ldap`
+- Resource properties are automatically enumerated and added to the accumulator configuration file
+- The accumulator templates for the configuration files are now persistent via the loading of the current config state during creation
+  - Configuration is no longer automatically removed when the resource falls out of scope
+  - To remove configuration elements an explicit resource with `:delete` action is required
+    - If no properties are specified with the `:delete` action then the whole configuration section will be removed
+    - If properties are specified with the `:delete` action then only the specified properties will be removed
+  - Configuration files will not be overwritten with a blank or partial configuration when a run fails
+- The auth resource has been split into separate resources
+- The log resource has been split into separate resources
+- The LDAP configuration resource has been split and renamed to singular forms
+
 ## 9.0.0
 
 Ensure you are on chef 15.5 or greater as we are now using `chef_sleep` in tests
@@ -12,7 +30,7 @@ Remove any `source`, `cookbook`, `config_file` and `config_directory` overrides
 Remove any `cookbook` and `config_directory` overrides from the `grafana_config` resource
 Add the `grafana_config_writer` to the end of your config resources, this will create the config file on disk and restart grafana to allow any api calls to work straight after
 Change `ldap_config_servers` `host` property from the name property to a normal property (required)
-Change `ldap_config_group_mappings` `group_dn` property fromthe name property to a normal property (required)
+Change `ldap_config_group_mappings` `group_dn` property from the name property to a normal property (required)
 Added `instance_name` to above resources as name property, this should line up across all config resources
 Change `grafana_config_database` property `type` to a symbol
 Change `grafana_config_database` property `ssl_mode` to a symbol or true/false