Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failure to create a Root CA using Zookeeper as storage backend #29260

Open
AdriRRP opened this issue Dec 25, 2024 · 0 comments
Open

Failure to create a Root CA using Zookeeper as storage backend #29260

AdriRRP opened this issue Dec 25, 2024 · 0 comments

Comments

@AdriRRP
Copy link

AdriRRP commented Dec 25, 2024

Describe the bug
It is not possible to create a Root CA with the root user using Zookeeper as storage backend with mTLS.

To Reproduce
Steps to reproduce the behavior:

  1. Start Vault server with vault server -config=/opt/homebrew/etc/vault/vault-config.hcl
  2. In other terminal session, export env vars: export VAULT_ADDR=https://127.0.0.1:8200 VAULT_SKIP_VERIFY=true
  3. Init: vault operator init -key-shares=1 -key-threshold=1
  4. Unseal: vault operator unseal (with the unseal key obtained from the previous command).
  5. Login as root: vault login (with the token obtained from step 3).
  6. Enable PKI engine: vault secrets enable pki.
  7. Tune the PKI secrets engine to issue certificates with a maximum TTL: vault secrets tune -max-lease-ttl=87600h pki.
  8. Generate the example.com root CA: vault write pki/root/generate/internal common_name="example.com" issuer_name="root-2024" ttl=87600h key_bits=4096.
  9. See error:
Error writing data to pki/root/generate/internal: Error making API request.

URL: PUT https://127.0.0.1:8200/v1/pki/root/generate/internal
Code: 500. Errors:

* 1 error occurred:
	* error building CRLs: unable to persist updated cluster-local CRL config: failed to clean up internal CRL mapping: failed cleaning up orphaned CRL e8b334b1-82fd-daa1-f405-622600338209-delta: failed to acquire node data: zk: invalid path

Expected behavior
As explained in the Step 1 of Build your own certificate authority (CA) documentation, the Root CA should be created.

Environment:

  • Vault Server Version (retrieve with vault status): 1.18.3
  • Vault CLI Version (retrieve with vault version): 1.18.3
  • Server Operating System/Architecture: macOS Sequoia 15.2 (aarch64)

Vault server configuration file(s):

disable_mlock=true,

listener "tcp" {
  address       = "localhost:8200"
  tls_cert_file = "/opt/homebrew/etc/ssl/certs/vault/vault-cert.pem"
  tls_key_file  = "/opt/homebrew/etc/ssl/certs/vault/vault-key.pem"
  tls_client_ca_file = "/opt/homebrew/etc/ssl/certs/vault/ca-cert.pem"
  tls_min_version     = "tls12"
}

storage "zookeeper" {
  address             = "localhost:2281"
  path                = "vault/"
  redirect_addr       = "http://localhost:8200"
  tls_verify_ip       = false
  tls_enabled         = true
  tls_min_version     = "tls12"
  tls_ca_file         = "/opt/homebrew/etc/ssl/certs/zookeeper/ca-cert.pem"
  tls_cert_file       = "/opt/homebrew/etc/ssl/certs/zookeeper/zkclient-cert.pem"
  tls_key_file        = "/opt/homebrew/etc/ssl/certs/zookeeper/zkclient-key.pem"
  tls_skip_verify     = false
}

Additional context

  • Vault server logs:
2024-12-25 20:51:57.302723 I | connected to [::1]:2281
2024-12-25 20:51:57.311981 I | authenticated: id=72060054238330883, timeout=4000
2024-12-25 20:51:57.312041 I | re-submitting `0` credentials after reconnect
==> Vault server configuration:

Administrative Namespace:
             Api Address: http://localhost:8200
                     Cgo: disabled
         Cluster Address: https://localhost:8201
   Environment Variables: ANDROID_HOME, CARGO_HOME, COLORFGBG, COLORTERM, COMMAND_MODE, EDITOR, HOME, ITERM_PROFILE, ITERM_SESSION_ID, JAVA_HOME, LANG, LC_CTYPE, LC_TERMINAL, LC_TERMINAL_VERSION, LESS, LOGNAME, LSCOLORS, LaunchInstanceID, NVM_HOME, OLDPWD, PAGER, PATH, PWD, RUST_SRC_PATH, RVM_HOME, SDKMAN_CANDIDATES_API, SDKMAN_CANDIDATES_DIR, SDKMAN_DIR, SDKMAN_PLATFORM, SECURITYSESSIONID, SHELL, SHLVL, SSH_AUTH_SOCK, TERM, TERMINFO_DIRS, TERM_FEATURES, TERM_PROGRAM, TERM_PROGRAM_VERSION, TERM_SESSION_ID, TMPDIR, USER, VAULT_ADDR, VAULT_DISABLE_PKI_CONSTRAINTS_VERIFICATION, VAULT_NAMESPACE, VAULT_SKIP_VERIFY, VAULT_TOKEN, XPC_FLAGS, XPC_SERVICE_NAME, ZSH, _, __CFBundleIdentifier, __CF_USER_TEXT_ENCODING
              Go Version: go1.23.3
              Listener 1: tcp (addr: "localhost:8200", cluster address: "127.0.0.1:8201", disable_request_limiter: "false", max_request_duration: "1m30s", max_request_size: "33554432", tls: "enabled")
               Log Level:
                   Mlock: supported: false, enabled: false
           Recovery Mode: false
                 Storage: zookeeper (HA available)
                 Version: Vault v1.18.3, built 2024-12-16T14:00:53Z
             Version Sha: 7ae4eca5403bf574f142cd8f987b8d83bafcd1de

==> Vault server started! Log data will stream in below:

2024-12-25T20:51:57.260+0100 [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
2024-12-25T20:51:57.313+0100 [INFO]  incrementing seal generation: generation=1
2024-12-25T20:51:57.314+0100 [INFO]  core: Initializing version history cache for core
2024-12-25T20:51:57.314+0100 [INFO]  events: Starting event system
2024-12-25T20:52:25.353+0100 [INFO]  core: security barrier not initialized
2024-12-25T20:52:25.354+0100 [INFO]  core: seal configuration missing, not initialized
2024-12-25T20:52:25.358+0100 [INFO]  core: security barrier not initialized
2024-12-25T20:52:25.395+0100 [INFO]  core: security barrier initialized: stored=1 shares=1 threshold=1
2024-12-25T20:52:25.478+0100 [INFO]  core: post-unseal setup starting
2024-12-25T20:52:25.506+0100 [INFO]  core: loaded wrapping token key
2024-12-25T20:52:25.506+0100 [INFO]  core: successfully setup plugin runtime catalog
2024-12-25T20:52:25.506+0100 [INFO]  core: successfully setup plugin catalog: plugin-directory=""
2024-12-25T20:52:25.507+0100 [INFO]  core: no mounts; adding default mount table
2024-12-25T20:52:25.532+0100 [INFO]  core: successfully mounted: type=cubbyhole version="v1.18.3+builtin.vault" path=cubbyhole/ namespace="ID: root. Path: "
2024-12-25T20:52:25.536+0100 [INFO]  core: successfully mounted: type=system version="v1.18.3+builtin.vault" path=sys/ namespace="ID: root. Path: "
2024-12-25T20:52:25.538+0100 [INFO]  core: successfully mounted: type=identity version="v1.18.3+builtin.vault" path=identity/ namespace="ID: root. Path: "
2024-12-25T20:52:25.606+0100 [INFO]  core: successfully mounted: type=token version="v1.18.3+builtin.vault" path=token/ namespace="ID: root. Path: "
2024-12-25T20:52:25.617+0100 [INFO]  rollback: Starting the rollback manager with 256 workers
2024-12-25T20:52:25.617+0100 [INFO]  rollback: starting rollback manager
2024-12-25T20:52:25.618+0100 [INFO]  core: restoring leases
2024-12-25T20:52:25.625+0100 [INFO]  expiration: lease restore complete
2024-12-25T20:52:25.642+0100 [INFO]  identity: entities restored
2024-12-25T20:52:25.642+0100 [INFO]  identity: groups restored
2024-12-25T20:52:25.644+0100 [INFO]  core: usage gauge collection is disabled
2024-12-25T20:52:25.668+0100 [INFO]  core: Recorded vault version: vault version=1.18.3 upgrade time="2024-12-25 19:52:25.644507 +0000 UTC" build date=2024-12-16T14:00:53Z
2024-12-25T20:52:25.745+0100 [INFO]  core: post-unseal setup complete
2024-12-25T20:52:25.792+0100 [INFO]  core: root token generated
2024-12-25T20:52:25.796+0100 [INFO]  core: pre-seal teardown starting
2024-12-25T20:52:25.796+0100 [INFO]  rollback: stopping rollback manager
2024-12-25T20:52:25.796+0100 [INFO]  core: pre-seal teardown complete
2024-12-25T21:14:14.525+0100 [INFO]  core.cluster-listener.tcp: starting listener: listener_address=127.0.0.1:8201
2024-12-25T21:14:14.525+0100 [INFO]  core.cluster-listener: serving cluster requests: cluster_listen_address=127.0.0.1:8201
2024-12-25T21:14:14.525+0100 [INFO]  core: vault is unsealed
2024-12-25T21:14:14.525+0100 [INFO]  core: entering standby mode
2024-12-25T21:14:14.542+0100 [INFO]  core: acquired lock, enabling active operation
2024-12-25T21:14:14.566+0100 [INFO]  core: post-unseal setup starting
2024-12-25T21:14:14.574+0100 [INFO]  core: loaded wrapping token key
2024-12-25T21:14:14.574+0100 [INFO]  core: successfully setup plugin runtime catalog
2024-12-25T21:14:14.574+0100 [INFO]  core: successfully setup plugin catalog: plugin-directory=""
2024-12-25T21:14:14.575+0100 [INFO]  core: successfully mounted: type=system version="v1.18.3+builtin.vault" path=sys/ namespace="ID: root. Path: "
2024-12-25T21:14:14.576+0100 [INFO]  core: successfully mounted: type=identity version="v1.18.3+builtin.vault" path=identity/ namespace="ID: root. Path: "
2024-12-25T21:14:14.576+0100 [INFO]  core: successfully mounted: type=cubbyhole version="v1.18.3+builtin.vault" path=cubbyhole/ namespace="ID: root. Path: "
2024-12-25T21:14:14.579+0100 [INFO]  core: successfully mounted: type=token version="v1.18.3+builtin.vault" path=token/ namespace="ID: root. Path: "
2024-12-25T21:14:14.581+0100 [INFO]  rollback: Starting the rollback manager with 256 workers
2024-12-25T21:14:14.581+0100 [INFO]  rollback: starting rollback manager
2024-12-25T21:14:14.581+0100 [INFO]  core: restoring leases
2024-12-25T21:14:14.584+0100 [INFO]  expiration: lease restore complete
2024-12-25T21:14:14.584+0100 [INFO]  identity: entities restored
2024-12-25T21:14:14.584+0100 [INFO]  identity: groups restored
2024-12-25T21:14:14.586+0100 [INFO]  core: usage gauge collection is disabled
2024-12-25T21:14:14.587+0100 [INFO]  core: post-unseal setup complete
2024-12-25T21:21:07.039+0100 [INFO]  secrets.pki.pki_957e6b7d: 70ceb858-3733-3eeb-133d-d1411ea1148b: succeeded in migrating to issuer storage version 2
2024-12-25T21:21:07.050+0100 [INFO]  core: successful mount: namespace="" path=pki/ type=pki version="v1.18.3+builtin.vault"
2024-12-25T21:23:57.393+0100 [INFO]  core: mount tuning of leases successful: path=pki/
  • Zookeeper's zoo.cfg:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/opt/homebrew/var/run/zookeeper/data
clientPort=2181
client.secure=true
serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory
secureClientPort=2281
ssl.keyStore.location=/opt/homebrew/etc/ssl/certs/zookeeper/zkserver-keystore.jks
ssl.keyStore.password=zookeeper
ssl.trustStore.location=/opt/homebrew/etc/ssl/certs/zookeeper/zkserver-truststore.jks
ssl.trustStore.password=zookeepe
zookeeper.ssl.clientAuth=required
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant