Merge pull request #5822 from PastaPastaPasta/backport-guix

backport: Guix related backports
dashpay · Feb 7, 2024 · b4cb17f · b4cb17f
2 parents 83ec7f2 + 69737c0
commit b4cb17f
Show file tree

Hide file tree

Showing 12 changed files with 124 additions and 96 deletions.
diff --git a/.github/workflows/semantic-pull-request.yml b/.github/workflows/semantic-pull-request.yml
@@ -28,6 +28,7 @@ jobs:
             perf
             test
             build
+            guix
             ci
             chore
             revert

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -131,6 +131,7 @@ include:
   - *utils* for changes to the utils and libraries
   - *wallet* for changes to the wallet code
   - *zmq* for changes to the ZMQ APIs
+  - *guix* for changes to the GUIX reproducible builds
 
 Examples:
 

diff --git a/contrib/guix/INSTALL.md b/contrib/guix/INSTALL.md
@@ -72,11 +72,11 @@ writing (July 2021). Guix is expected to be more widely packaged over time. For
 an up-to-date view on Guix's package status/version across distros, please see:
 https://repology.org/project/guix/versions
 
-### Debian 11 (Bullseye)/Ubuntu 21.04 (Hirsute Hippo)
+### Debian / Ubuntu
 
 Guix v1.2.0 is available as a distribution package starting in [Debian
 11](https://packages.debian.org/bullseye/guix) and [Ubuntu
-21.04](https://packages.ubuntu.com/hirsute/guix).
+21.04](https://packages.ubuntu.com/search?keywords=guix).
 
 Note that if you intend on using Guix without using any substitutes (more
 details [here][security-model]), v1.2.0 has a known problem when building GnuTLS
@@ -167,6 +167,10 @@ For reference, the graphic below outlines Guix v1.3.0's dependency graph:
 
 ![bootstrap map](https://user-images.githubusercontent.com/6399679/125064185-a9a59880-e0b0-11eb-82c1-9b8e5dc9950d.png)
 
+#### Consider /tmp on tmpfs
+
+If you use an NVME (SSD) drive, you may encounter [cryptic build errors](#coreutils-fail-teststail-2inotify-dir-recreate). Mounting a [tmpfs at /tmp](https://ubuntu.com/blog/data-driven-analysis-tmp-on-tmpfs) should prevent this and may improve performance as a bonus.
+
 #### Guile
 
 ##### Choosing a Guile version and sticking to it
@@ -334,6 +338,8 @@ packages in Debian at the time of writing.
 |-----------------------|---------------------|
 | guile-gcrypt          | libgcrypt-dev       |
 | guile-git             | libgit2-dev         |
+| guile-gnutls          | (none)              |
+| guile-json            | (none)              |
 | guile-lzlib           | liblz-dev           |
 | guile-ssh             | libssh-dev          |
 | guile-sqlite3         | libsqlite3-dev      |
@@ -384,8 +390,9 @@ cd guix
 ```
 
 You will likely want to build the latest release, however, if the latest release
-when you're reading this is still 1.2.0 then you may want to use 95aca29 instead
-to avoid a problem in the GnuTLS test suite.
+when you're reading this is still 1.3.0 then you may want to use 998eda30 instead
+to avoid the issues described in [#25099](
+https://github.com/bitcoin/bitcoin/pull/25099).
 
 ```
 git branch -a -l 'origin/version-*'  # check for the latest release
@@ -609,6 +616,8 @@ systemctl enable guix-daemon
 systemctl start guix-daemon
 ```
 
+Remember to set `--no-substitute` in `$libdir/systemd/system/guix-daemon.service` and other customizations if you used them for `guix-daemon-original.service`.
+
 ##### If you installed Guix via the Debian/Ubuntu distribution packages
 
 You will need to create a `guix-daemon-latest` service which points to the new
@@ -717,6 +726,19 @@ $ bzcat /var/log/guix/drvs/../...-foo-3.6.12.drv.bz2 | less
 times, it may be `/tmp/...drv-1` or `/tmp/...drv-2`. Always consult the build
 failure output for the most accurate, up-to-date information.
 
+### openssl-1.1.1l and openssl-1.1.1n
+
+OpenSSL includes tests that will fail once some certificate has expired. A workaround
+is to change your system clock:
+
+```sh
+sudo timedatectl set-ntp no
+sudo date --set "28 may 2022 15:00:00"
+sudo --login guix build --cores=1 /gnu/store/g9alz81w4q03ncm542487xd001s6akd4-openssl-1.1.1l.drv
+sudo --login guix build --cores=1 /gnu/store/mw6ax0gk33gh082anrdrxp2flrbskxv6-openssl-1.1.1n.drv
+sudo timedatectl set-ntp yes
+```
+
 ### python(-minimal): [Errno 84] Invalid or incomplete multibyte or wide character
 
 This error occurs when your `$TMPDIR` (default: /tmp) exists on a filesystem
@@ -774,15 +796,15 @@ The inotify-dir-create test fails on "remote" filesystems such as overlayfs
 as non-remote.
 
 A relatively easy workaround to this is to make sure that a somewhat traditional
-filesystem is mounted at `/tmp` (where `guix-daemon` performs its builds). For
+filesystem is mounted at `/tmp` (where `guix-daemon` performs its builds), see [/tmp on tmpfs](#consider-tmp-on-tmpfs). For
 Docker users, this might mean [using a volume][docker/volumes], [binding
 mounting][docker/bind-mnt] from host, or (for those with enough RAM and swap)
 [mounting a tmpfs][docker/tmpfs] using the `--tmpfs` flag.
 
 Please see the following links for more details:
 
 - An upstream coreutils bug has been filed: [debbugs#47940](https://debbugs.gnu.org/cgi/bugreport.cgi?bug=47940)
-- A Guix bug detailing the underlying problem has been filed: [guix-issues#47935](https://issues.guix.gnu.org/47935)
+- A Guix bug detailing the underlying problem has been filed: [guix-issues#47935](https://issues.guix.gnu.org/47935), [guix-issues#49985](https://issues.guix.gnu.org/49985#5)
 - A commit to skip this test in Guix has been merged into the core-updates branch:
 [savannah/guix@6ba1058](https://git.savannah.gnu.org/cgit/guix.git/commit/?id=6ba1058df0c4ce5611c2367531ae5c3cdc729ab4)
 
@@ -799,3 +821,39 @@ Please see the following links for more details:
 [docker/volumes]: https://docs.docker.com/storage/volumes/
 [docker/bind-mnt]: https://docs.docker.com/storage/bind-mounts/
 [docker/tmpfs]: https://docs.docker.com/storage/tmpfs/
+
+# Purging/Uninstalling Guix
+
+In the extraordinarily rare case where you messed up your Guix installation in
+an irreversible way, you may want to completely purge Guix from your system and
+start over.
+
+1. Uninstall Guix itself according to the way you installed it (e.g. `sudo apt
+   purge guix` for Ubuntu packaging, `sudo make uninstall` for a build from source).
+2. Remove all build users and groups
+
+   You may check for relevant users and groups using:
+
+   ```
+   getent passwd | grep guix
+   getent group | grep guix
+   ```
+
+   Then, you may remove users and groups using:
+
+   ```
+   sudo userdel <user>
+   sudo groupdel <group>
+   ```
+
+3. Remove all possible Guix-related directories
+    - `/var/guix/`
+    - `/var/log/guix/`
+    - `/gnu/`
+    - `/etc/guix/`
+    - `/home/*/.config/guix/`
+    - `/home/*/.cache/guix/`
+    - `/home/*/.guix-profile/`
+    - `/root/.config/guix/`
+    - `/root/.cache/guix/`
+    - `/root/.guix-profile/`
diff --git a/contrib/guix/README.md b/contrib/guix/README.md
@@ -382,7 +382,7 @@ https://ci.guix.gnu.org is automatically used unless the `--no-substitutes` flag
 is supplied. This default list of substitute servers is overridable both on a
 `guix-daemon` level and when you invoke `guix` commands. See examples below for
 the various ways of adding dongcarl's substitute server after having [authorized
-his signing key](#authorize-the-signing-keys).
+his signing key](#step-1-authorize-the-signing-keys).
 
 Change the **default list** of substitute servers by starting `guix-daemon` with
 the `--substitute-urls` option (you will likely need to edit your init script):
@@ -430,55 +430,6 @@ used.
 If you start `guix-daemon` using an init script, you can edit said script to
 supply this flag.
 
-
-# Purging/Uninstalling Guix
-
-In the extraordinarily rare case where you messed up your Guix installation in
-an irreversible way, you may want to completely purge Guix from your system and
-start over.
-
-1. Uninstall Guix itself according to the way you installed it (e.g. `sudo apt
-   purge guix` for Ubuntu packaging, `sudo make uninstall` for a build from source).
-2. Remove all build users and groups
-
-   You may check for relevant users and groups using:
-
-   ```
-   getent passwd | grep guix
-   getent group | grep guix
-   ```
-
-   Then, you may remove users and groups using:
-
-   ```
-   sudo userdel <user>
-   sudo groupdel <group>
-   ```
-
-3. Remove all possible Guix-related directories
-    - `/var/guix/`
-    - `/var/log/guix/`
-    - `/gnu/`
-    - `/etc/guix/`
-    - `/home/*/.config/guix/`
-    - `/home/*/.cache/guix/`
-    - `/home/*/.guix-profile/`
-    - `/root/.config/guix/`
-    - `/root/.cache/guix/`
-    - `/root/.guix-profile/`
-
 [b17e]: http://bootstrappable.org/
 [r12e/source-date-epoch]: https://reproducible-builds.org/docs/source-date-epoch/
-
-[guix/install.sh]: https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
-[guix/bin-install]: https://www.gnu.org/software/guix/manual/en/html_node/Binary-Installation.html
-[guix/env-setup]: https://www.gnu.org/software/guix/manual/en/html_node/Build-Environment-Setup.html
-[guix/substitutes]: https://www.gnu.org/software/guix/manual/en/html_node/Substitutes.html
-[guix/substitute-server-auth]: https://www.gnu.org/software/guix/manual/en/html_node/Substitute-Server-Authorization.html
-[guix/time-machine]: https://guix.gnu.org/manual/en/html_node/Invoking-guix-time_002dmachine.html
-
-[debian/guix-bullseye]: https://packages.debian.org/bullseye/guix
-[ubuntu/guix-hirsute]: https://packages.ubuntu.com/hirsute/guix
-[guix-docker]: https://github.com/dashpay/dash/tree/master/contrib/guix/Dockerfile
-
 [env-vars-list]: #recognized-environment-variables
diff --git a/contrib/guix/guix-attest b/contrib/guix/guix-attest
@@ -213,29 +213,28 @@ mkdir -p "$outsigdir"
         exit 1
     fi
 
-    temp_codesigned="$(mktemp)"
-    trap 'rm -rf -- "$temp_codesigned"' EXIT
+    temp_all="$(mktemp)"
+    trap 'rm -rf -- "$temp_all"' EXIT
 
     if (( ${#codesigned_fragments[@]} )); then
         # Note: all.SHA256SUMS attests to all of $sha256sum_fragments, but is
         #       not needed if there are no $codesigned_fragments
         cat "${sha256sum_fragments[@]}" \
             | sort -u \
             | sort -k2 \
-            | sed 's/$/\r/' \
             | basenameify_SHA256SUMS \
-                > "$temp_codesigned"
-        if [ -e codesigned.SHA256SUMS ]; then
+                > "$temp_all"
+        if [ -e all.SHA256SUMS ]; then
             # The SHA256SUMS already exists, make sure it's exactly what we
             # expect, error out if not
-            if diff -u all.SHA256SUMS "$temp_codesigned"; then
+            if diff -u all.SHA256SUMS "$temp_all"; then
                 echo "An all.SHA256SUMS file already exists for '${VERSION}' and is up-to-date."
             else
                 shasum_already_exists all.SHA256SUMS
                 exit 1
             fi
         else
-            mv "$temp_codesigned" codesigned.SHA256SUMS
+            mv "$temp_all" all.SHA256SUMS
         fi
     else
         # It is fine to have the codesigned outputs be missing (perhaps the

diff --git a/contrib/guix/guix-build b/contrib/guix/guix-build
@@ -233,7 +233,7 @@ host_to_commonname() {
 }
 
 # Determine the reference time used for determinism (overridable by environment)
-SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log --format=%at -1)}"
+SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git -c log.showSignature=false log --format=%at -1)}"
 
 # Precious directories are those which should not be cleaned between successive
 # guix builds

diff --git a/contrib/guix/guix-codesign b/contrib/guix/guix-codesign
@@ -220,7 +220,7 @@ fi
 JOBS="${JOBS:-$(nproc)}"
 
 # Determine the reference time used for determinism (overridable by environment)
-SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log --format=%at -1)}"
+SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git -c log.showSignature=false log --format=%at -1)}"
 
 # Make sure an output directory exists for our builds
 OUTDIR_BASE="${OUTDIR_BASE:-${VERSION_BASE}/output}"

diff --git a/contrib/guix/guix-verify b/contrib/guix/guix-verify
@@ -28,7 +28,11 @@ cmd_usage() {
 cat <<EOF
 Synopsis:
 
-    env GUIX_SIGS_REPO=<path/to/guix.sigs> ./contrib/guix/guix-verify
+    env GUIX_SIGS_REPO=<path/to/guix.sigs> [ SIGNER=<signer> ] ./contrib/guix/guix-verify
+
+Example overriding signer's manifest to use as base
+
+    env GUIX_SIGS_REPO=/home/dongcarl/guix.sigs SIGNER=achow101 ./contrib/guix/guix-verify
 
 EOF
 }
@@ -94,6 +98,17 @@ echo "--------------------"
 echo ""
 if (( ${#all_noncodesigned[@]} )); then
     compare_noncodesigned="${all_noncodesigned[0]}"
+    if [[ -n "$SIGNER" ]]; then
+        signer_noncodesigned="$OUTSIGDIR_BASE/$SIGNER/noncodesigned.SHA256SUMS"
+        if [[ -f "$signer_noncodesigned" ]]; then
+            echo "Using $SIGNER's manifest as the base to compare against"
+            compare_noncodesigned="$signer_noncodesigned"
+        else
+            echo "Unable to find $SIGNER's manifest, using the first one found"
+        fi
+    else
+        echo "No SIGNER provided, using the first manifest found"
+    fi
 
     for current_manifest in "${all_noncodesigned[@]}"; do
         verify "$compare_noncodesigned" "$current_manifest"
@@ -114,6 +129,17 @@ echo "--------------------"
 echo ""
 if (( ${#all_all[@]} )); then
     compare_all="${all_all[0]}"
+    if [[ -n "$SIGNER" ]]; then
+        signer_all="$OUTSIGDIR_BASE/$SIGNER/all.SHA256SUMS"
+        if [[ -f "$signer_all" ]]; then
+            echo "Using $SIGNER's manifest as the base to compare against"
+            compare_all="$signer_all"
+        else
+            echo "Unable to find $SIGNER's manifest, using the first one found"
+        fi
+    else
+        echo "No SIGNER provided, using the first manifest found"
+    fi
 
     for current_manifest in "${all_all[@]}"; do
         verify "$compare_all" "$current_manifest"

diff --git a/contrib/guix/libexec/build.sh b/contrib/guix/libexec/build.sh
@@ -207,9 +207,9 @@ make -C depends --jobs="$JOBS" HOST="$HOST" \
                                    ${SDK_PATH+SDK_PATH="$SDK_PATH"} \
                                    x86_64_linux_CC=x86_64-linux-gnu-gcc \
                                    x86_64_linux_CXX=x86_64-linux-gnu-g++ \
-                                   x86_64_linux_AR=x86_64-linux-gnu-ar \
-                                   x86_64_linux_RANLIB=x86_64-linux-gnu-ranlib \
-                                   x86_64_linux_NM=x86_64-linux-gnu-nm \
+                                   x86_64_linux_AR=x86_64-linux-gnu-gcc-ar \
+                                   x86_64_linux_RANLIB=x86_64-linux-gnu-gcc-ranlib \
+                                   x86_64_linux_NM=x86_64-linux-gnu-gcc-nm \
                                    x86_64_linux_STRIP=x86_64-linux-gnu-strip \
                                    qt_config_opts_x86_64_linux='-platform linux-g++ -xplatform bitcoin-linux-g++' \
                                    FORCE_USE_SYSTEM_CLANG=1
-Original file line number
+Diff line change
@@ Expand Up / @@ -28,6 +28,7 @@ jobs: @@
                 perf
                 test
                 build
+                guix
                 ci
                 chore
                 revert
@@ Expand Down @@