diff --git a/.github/workflows/semantic-pull-request.yml b/.github/workflows/semantic-pull-request.yml index cb351ee2f9172..2ec7381172f6d 100644 --- a/.github/workflows/semantic-pull-request.yml +++ b/.github/workflows/semantic-pull-request.yml @@ -28,6 +28,7 @@ jobs: perf test build + guix ci chore revert diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 95e34313e6d8a..381dbb91a88d4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -131,6 +131,7 @@ include: - *utils* for changes to the utils and libraries - *wallet* for changes to the wallet code - *zmq* for changes to the ZMQ APIs + - *guix* for changes to the GUIX reproducible builds Examples: diff --git a/contrib/guix/INSTALL.md b/contrib/guix/INSTALL.md index a50b80f35719b..c7050b2ae3703 100644 --- a/contrib/guix/INSTALL.md +++ b/contrib/guix/INSTALL.md @@ -72,11 +72,11 @@ writing (July 2021). Guix is expected to be more widely packaged over time. For an up-to-date view on Guix's package status/version across distros, please see: https://repology.org/project/guix/versions -### Debian 11 (Bullseye)/Ubuntu 21.04 (Hirsute Hippo) +### Debian / Ubuntu Guix v1.2.0 is available as a distribution package starting in [Debian 11](https://packages.debian.org/bullseye/guix) and [Ubuntu -21.04](https://packages.ubuntu.com/hirsute/guix). +21.04](https://packages.ubuntu.com/search?keywords=guix). Note that if you intend on using Guix without using any substitutes (more details [here][security-model]), v1.2.0 has a known problem when building GnuTLS @@ -167,6 +167,10 @@ For reference, the graphic below outlines Guix v1.3.0's dependency graph: ![bootstrap map](https://user-images.githubusercontent.com/6399679/125064185-a9a59880-e0b0-11eb-82c1-9b8e5dc9950d.png) +#### Consider /tmp on tmpfs + +If you use an NVME (SSD) drive, you may encounter [cryptic build errors](#coreutils-fail-teststail-2inotify-dir-recreate). Mounting a [tmpfs at /tmp](https://ubuntu.com/blog/data-driven-analysis-tmp-on-tmpfs) should prevent this and may improve performance as a bonus. + #### Guile ##### Choosing a Guile version and sticking to it @@ -334,6 +338,8 @@ packages in Debian at the time of writing. |-----------------------|---------------------| | guile-gcrypt | libgcrypt-dev | | guile-git | libgit2-dev | +| guile-gnutls | (none) | +| guile-json | (none) | | guile-lzlib | liblz-dev | | guile-ssh | libssh-dev | | guile-sqlite3 | libsqlite3-dev | @@ -384,8 +390,9 @@ cd guix ``` You will likely want to build the latest release, however, if the latest release -when you're reading this is still 1.2.0 then you may want to use 95aca29 instead -to avoid a problem in the GnuTLS test suite. +when you're reading this is still 1.3.0 then you may want to use 998eda30 instead +to avoid the issues described in [#25099]( +https://github.com/bitcoin/bitcoin/pull/25099). ``` git branch -a -l 'origin/version-*' # check for the latest release @@ -609,6 +616,8 @@ systemctl enable guix-daemon systemctl start guix-daemon ``` +Remember to set `--no-substitute` in `$libdir/systemd/system/guix-daemon.service` and other customizations if you used them for `guix-daemon-original.service`. + ##### If you installed Guix via the Debian/Ubuntu distribution packages You will need to create a `guix-daemon-latest` service which points to the new @@ -717,6 +726,19 @@ $ bzcat /var/log/guix/drvs/../...-foo-3.6.12.drv.bz2 | less times, it may be `/tmp/...drv-1` or `/tmp/...drv-2`. Always consult the build failure output for the most accurate, up-to-date information. +### openssl-1.1.1l and openssl-1.1.1n + +OpenSSL includes tests that will fail once some certificate has expired. A workaround +is to change your system clock: + +```sh +sudo timedatectl set-ntp no +sudo date --set "28 may 2022 15:00:00" +sudo --login guix build --cores=1 /gnu/store/g9alz81w4q03ncm542487xd001s6akd4-openssl-1.1.1l.drv +sudo --login guix build --cores=1 /gnu/store/mw6ax0gk33gh082anrdrxp2flrbskxv6-openssl-1.1.1n.drv +sudo timedatectl set-ntp yes +``` + ### python(-minimal): [Errno 84] Invalid or incomplete multibyte or wide character This error occurs when your `$TMPDIR` (default: /tmp) exists on a filesystem @@ -774,7 +796,7 @@ The inotify-dir-create test fails on "remote" filesystems such as overlayfs as non-remote. A relatively easy workaround to this is to make sure that a somewhat traditional -filesystem is mounted at `/tmp` (where `guix-daemon` performs its builds). For +filesystem is mounted at `/tmp` (where `guix-daemon` performs its builds), see [/tmp on tmpfs](#consider-tmp-on-tmpfs). For Docker users, this might mean [using a volume][docker/volumes], [binding mounting][docker/bind-mnt] from host, or (for those with enough RAM and swap) [mounting a tmpfs][docker/tmpfs] using the `--tmpfs` flag. @@ -782,7 +804,7 @@ mounting][docker/bind-mnt] from host, or (for those with enough RAM and swap) Please see the following links for more details: - An upstream coreutils bug has been filed: [debbugs#47940](https://debbugs.gnu.org/cgi/bugreport.cgi?bug=47940) -- A Guix bug detailing the underlying problem has been filed: [guix-issues#47935](https://issues.guix.gnu.org/47935) +- A Guix bug detailing the underlying problem has been filed: [guix-issues#47935](https://issues.guix.gnu.org/47935), [guix-issues#49985](https://issues.guix.gnu.org/49985#5) - A commit to skip this test in Guix has been merged into the core-updates branch: [savannah/guix@6ba1058](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=6ba1058df0c4ce5611c2367531ae5c3cdc729ab4) @@ -799,3 +821,39 @@ Please see the following links for more details: [docker/volumes]: https://docs.docker.com/storage/volumes/ [docker/bind-mnt]: https://docs.docker.com/storage/bind-mounts/ [docker/tmpfs]: https://docs.docker.com/storage/tmpfs/ + +# Purging/Uninstalling Guix + +In the extraordinarily rare case where you messed up your Guix installation in +an irreversible way, you may want to completely purge Guix from your system and +start over. + +1. Uninstall Guix itself according to the way you installed it (e.g. `sudo apt + purge guix` for Ubuntu packaging, `sudo make uninstall` for a build from source). +2. Remove all build users and groups + + You may check for relevant users and groups using: + + ``` + getent passwd | grep guix + getent group | grep guix + ``` + + Then, you may remove users and groups using: + + ``` + sudo userdel + sudo groupdel + ``` + +3. Remove all possible Guix-related directories + - `/var/guix/` + - `/var/log/guix/` + - `/gnu/` + - `/etc/guix/` + - `/home/*/.config/guix/` + - `/home/*/.cache/guix/` + - `/home/*/.guix-profile/` + - `/root/.config/guix/` + - `/root/.cache/guix/` + - `/root/.guix-profile/` diff --git a/contrib/guix/README.md b/contrib/guix/README.md index 28d570ffa2c23..698deb47acfca 100644 --- a/contrib/guix/README.md +++ b/contrib/guix/README.md @@ -382,7 +382,7 @@ https://ci.guix.gnu.org is automatically used unless the `--no-substitutes` flag is supplied. This default list of substitute servers is overridable both on a `guix-daemon` level and when you invoke `guix` commands. See examples below for the various ways of adding dongcarl's substitute server after having [authorized -his signing key](#authorize-the-signing-keys). +his signing key](#step-1-authorize-the-signing-keys). Change the **default list** of substitute servers by starting `guix-daemon` with the `--substitute-urls` option (you will likely need to edit your init script): @@ -430,55 +430,6 @@ used. If you start `guix-daemon` using an init script, you can edit said script to supply this flag. - -# Purging/Uninstalling Guix - -In the extraordinarily rare case where you messed up your Guix installation in -an irreversible way, you may want to completely purge Guix from your system and -start over. - -1. Uninstall Guix itself according to the way you installed it (e.g. `sudo apt - purge guix` for Ubuntu packaging, `sudo make uninstall` for a build from source). -2. Remove all build users and groups - - You may check for relevant users and groups using: - - ``` - getent passwd | grep guix - getent group | grep guix - ``` - - Then, you may remove users and groups using: - - ``` - sudo userdel - sudo groupdel - ``` - -3. Remove all possible Guix-related directories - - `/var/guix/` - - `/var/log/guix/` - - `/gnu/` - - `/etc/guix/` - - `/home/*/.config/guix/` - - `/home/*/.cache/guix/` - - `/home/*/.guix-profile/` - - `/root/.config/guix/` - - `/root/.cache/guix/` - - `/root/.guix-profile/` - [b17e]: http://bootstrappable.org/ [r12e/source-date-epoch]: https://reproducible-builds.org/docs/source-date-epoch/ - -[guix/install.sh]: https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh -[guix/bin-install]: https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html -[guix/env-setup]: https://www.gnu.org/software/guix/manual/en/html_node/Build-Environment-Setup.html -[guix/substitutes]: https://www.gnu.org/software/guix/manual/en/html_node/Substitutes.html -[guix/substitute-server-auth]: https://www.gnu.org/software/guix/manual/en/html_node/Substitute-Server-Authorization.html -[guix/time-machine]: https://guix.gnu.org/manual/en/html_node/Invoking-guix-time_002dmachine.html - -[debian/guix-bullseye]: https://packages.debian.org/bullseye/guix -[ubuntu/guix-hirsute]: https://packages.ubuntu.com/hirsute/guix -[guix-docker]: https://github.com/dashpay/dash/tree/master/contrib/guix/Dockerfile - [env-vars-list]: #recognized-environment-variables diff --git a/contrib/guix/guix-attest b/contrib/guix/guix-attest index 84fb2840eaf9c..7c71ff3508260 100755 --- a/contrib/guix/guix-attest +++ b/contrib/guix/guix-attest @@ -213,8 +213,8 @@ mkdir -p "$outsigdir" exit 1 fi - temp_codesigned="$(mktemp)" - trap 'rm -rf -- "$temp_codesigned"' EXIT + temp_all="$(mktemp)" + trap 'rm -rf -- "$temp_all"' EXIT if (( ${#codesigned_fragments[@]} )); then # Note: all.SHA256SUMS attests to all of $sha256sum_fragments, but is @@ -222,20 +222,19 @@ mkdir -p "$outsigdir" cat "${sha256sum_fragments[@]}" \ | sort -u \ | sort -k2 \ - | sed 's/$/\r/' \ | basenameify_SHA256SUMS \ - > "$temp_codesigned" - if [ -e codesigned.SHA256SUMS ]; then + > "$temp_all" + if [ -e all.SHA256SUMS ]; then # The SHA256SUMS already exists, make sure it's exactly what we # expect, error out if not - if diff -u all.SHA256SUMS "$temp_codesigned"; then + if diff -u all.SHA256SUMS "$temp_all"; then echo "An all.SHA256SUMS file already exists for '${VERSION}' and is up-to-date." else shasum_already_exists all.SHA256SUMS exit 1 fi else - mv "$temp_codesigned" codesigned.SHA256SUMS + mv "$temp_all" all.SHA256SUMS fi else # It is fine to have the codesigned outputs be missing (perhaps the diff --git a/contrib/guix/guix-build b/contrib/guix/guix-build index 176eab32f93df..0d7ecfa271c56 100755 --- a/contrib/guix/guix-build +++ b/contrib/guix/guix-build @@ -233,7 +233,7 @@ host_to_commonname() { } # Determine the reference time used for determinism (overridable by environment) -SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log --format=%at -1)}" +SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git -c log.showSignature=false log --format=%at -1)}" # Precious directories are those which should not be cleaned between successive # guix builds diff --git a/contrib/guix/guix-codesign b/contrib/guix/guix-codesign index 45da43abf1c12..445ee91172d94 100755 --- a/contrib/guix/guix-codesign +++ b/contrib/guix/guix-codesign @@ -220,7 +220,7 @@ fi JOBS="${JOBS:-$(nproc)}" # Determine the reference time used for determinism (overridable by environment) -SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log --format=%at -1)}" +SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git -c log.showSignature=false log --format=%at -1)}" # Make sure an output directory exists for our builds OUTDIR_BASE="${OUTDIR_BASE:-${VERSION_BASE}/output}" diff --git a/contrib/guix/guix-verify b/contrib/guix/guix-verify index ffcfba736092c..feb0e55ea04b6 100755 --- a/contrib/guix/guix-verify +++ b/contrib/guix/guix-verify @@ -28,7 +28,11 @@ cmd_usage() { cat < ./contrib/guix/guix-verify + env GUIX_SIGS_REPO= [ SIGNER= ] ./contrib/guix/guix-verify + +Example overriding signer's manifest to use as base + + env GUIX_SIGS_REPO=/home/dongcarl/guix.sigs SIGNER=achow101 ./contrib/guix/guix-verify EOF } @@ -94,6 +98,17 @@ echo "--------------------" echo "" if (( ${#all_noncodesigned[@]} )); then compare_noncodesigned="${all_noncodesigned[0]}" + if [[ -n "$SIGNER" ]]; then + signer_noncodesigned="$OUTSIGDIR_BASE/$SIGNER/noncodesigned.SHA256SUMS" + if [[ -f "$signer_noncodesigned" ]]; then + echo "Using $SIGNER's manifest as the base to compare against" + compare_noncodesigned="$signer_noncodesigned" + else + echo "Unable to find $SIGNER's manifest, using the first one found" + fi + else + echo "No SIGNER provided, using the first manifest found" + fi for current_manifest in "${all_noncodesigned[@]}"; do verify "$compare_noncodesigned" "$current_manifest" @@ -114,6 +129,17 @@ echo "--------------------" echo "" if (( ${#all_all[@]} )); then compare_all="${all_all[0]}" + if [[ -n "$SIGNER" ]]; then + signer_all="$OUTSIGDIR_BASE/$SIGNER/all.SHA256SUMS" + if [[ -f "$signer_all" ]]; then + echo "Using $SIGNER's manifest as the base to compare against" + compare_all="$signer_all" + else + echo "Unable to find $SIGNER's manifest, using the first one found" + fi + else + echo "No SIGNER provided, using the first manifest found" + fi for current_manifest in "${all_all[@]}"; do verify "$compare_all" "$current_manifest" diff --git a/contrib/guix/libexec/build.sh b/contrib/guix/libexec/build.sh index 0755764869b88..e5fc7cb2752da 100644 --- a/contrib/guix/libexec/build.sh +++ b/contrib/guix/libexec/build.sh @@ -207,9 +207,9 @@ make -C depends --jobs="$JOBS" HOST="$HOST" \ ${SDK_PATH+SDK_PATH="$SDK_PATH"} \ x86_64_linux_CC=x86_64-linux-gnu-gcc \ x86_64_linux_CXX=x86_64-linux-gnu-g++ \ - x86_64_linux_AR=x86_64-linux-gnu-ar \ - x86_64_linux_RANLIB=x86_64-linux-gnu-ranlib \ - x86_64_linux_NM=x86_64-linux-gnu-nm \ + x86_64_linux_AR=x86_64-linux-gnu-gcc-ar \ + x86_64_linux_RANLIB=x86_64-linux-gnu-gcc-ranlib \ + x86_64_linux_NM=x86_64-linux-gnu-gcc-nm \ x86_64_linux_STRIP=x86_64-linux-gnu-strip \ qt_config_opts_x86_64_linux='-platform linux-g++ -xplatform bitcoin-linux-g++' \ FORCE_USE_SYSTEM_CLANG=1 diff --git a/contrib/guix/manifest.scm b/contrib/guix/manifest.scm index 648420f023f78..560b112f29fab 100644 --- a/contrib/guix/manifest.scm +++ b/contrib/guix/manifest.scm @@ -139,15 +139,17 @@ chain for " target " development.")) ;; https://gcc.gnu.org/install/configure.html (define (hardened-gcc gcc) (package-with-extra-configure-variable ( - package-with-extra-configure-variable gcc - "--enable-default-ssp" "yes") - "--enable-default-pie" "yes")) + package-with-extra-configure-variable ( + package-with-extra-configure-variable gcc + "--enable-initfini-array" "yes") + "--enable-default-ssp" "yes") + "--enable-default-pie" "yes")) (define* (make-bitcoin-cross-toolchain target #:key (base-gcc-for-libc base-gcc) (base-kernel-headers base-linux-kernel-headers) - (base-libc (hardened-glibc (make-glibc-without-werror glibc-2.28))) + (base-libc (hardened-glibc glibc-2.28)) (base-gcc (make-gcc-rpath-link (hardened-gcc base-gcc)))) "Convenience wrapper around MAKE-CROSS-TOOLCHAIN with default values desirable for building Dash Core release binaries." @@ -233,7 +235,7 @@ thus should be able to compile on most platforms where these exist.") (license license:gpl3+))) ; license is with openssl exception (define-public python-elfesteem - (let ((commit "87bbd79ab7e361004c98cc8601d4e5f029fd8bd5")) + (let ((commit "2eb1e5384ff7a220fd1afacd4a0170acff54fe56")) (package (name "python-elfesteem") (version (git-version "0.1" "1" commit)) @@ -246,8 +248,7 @@ thus should be able to compile on most platforms where these exist.") (file-name (git-file-name name commit)) (sha256 (base32 - "1nyvjisvyxyxnd0023xjf5846xd03lwawp5pfzr8vrky7wwm5maz")) - (patches (search-our-patches "elfsteem-value-error-python-39.patch")))) + "07x6p8clh11z8s1n2kdxrqwqm2almgc5qpkcr9ckb6y5ivjdr5r6")))) (build-system python-build-system) ;; There are no tests, but attempting to run python setup.py test leads to ;; PYTHONPATH problems, just disable the test @@ -500,15 +501,16 @@ and endian independent.") inspecting signatures in Mach-O binaries.") (license license:expat)))) -(define (make-glibc-without-werror glibc) - (package-with-extra-configure-variable glibc "enable_werror" "no")) - ;; https://www.gnu.org/software/libc/manual/html_node/Configuring-and-compiling.html +;; We don't use --disable-werror directly, as that would be passed through to bash, +;; and cause it's build to fail. (define (hardened-glibc glibc) (package-with-extra-configure-variable ( - package-with-extra-configure-variable glibc - "--enable-stack-protector" "strong") - "--enable-bind-now" "yes")) + package-with-extra-configure-variable ( + package-with-extra-configure-variable glibc + "enable_werror" "no") + "--enable-stack-protector" "strong") + "--enable-bind-now" "yes")) (define-public glibc-2.28 (package @@ -599,7 +601,7 @@ parse, modify and abstract ELF, PE and MachO formats.") ;; Scripting python-minimal ;; (3.9) ;; Git - git + git-minimal ;; Tests (fix-ppc64-nx-default lief)) (let ((target (getenv "HOST"))) diff --git a/contrib/guix/patches/elfsteem-value-error-python-39.patch b/contrib/guix/patches/elfsteem-value-error-python-39.patch deleted file mode 100644 index 21e1228afd83c..0000000000000 --- a/contrib/guix/patches/elfsteem-value-error-python-39.patch +++ /dev/null @@ -1,13 +0,0 @@ -diff --git a/examples/otool.py b/examples/otool.py -index 2b8efc0..d797b2e 100755 ---- a/examples/otool.py -+++ b/examples/otool.py -@@ -342,7 +342,7 @@ if __name__ == '__main__': - try: - e = macho_init.MACHO(raw, - parseSymbols = False) -- except ValueError, err: -+ except ValueError as err: - print("%s:" %file) - print(" %s" % err) - continue diff --git a/doc/release-process.md b/doc/release-process.md index c7a804e6793d8..eaec5c59409bd 100644 --- a/doc/release-process.md +++ b/doc/release-process.md @@ -145,6 +145,9 @@ Codesigner only: Sign the windows binaries: * Enter the passphrase for the key when prompted * `signature-win.tar.gz` will be created +Code-signer only: It is advised to test that the code signature attaches properly prior to tagging by performing the `guix-codesign` step. +However if this is done, once the release has been tagged in the bitcoin-detached-sigs repo, the `guix-codesign` step must be performed again in order for the guix attestation to be valid when compared against the attestations of non-codesigner builds. + Codesigner only: Commit the detached codesign payloads: ```sh