From f6dfe42129d005662d91e1acb1c75eafe7b9b33e Mon Sep 17 00:00:00 2001 From: Oleg Pykhalov Date: Tue, 15 Sep 2020 01:10:55 +0300 Subject: services: docker: Fix service definition. This commit follows a404716d411cf7cd49ff02e3100f0bbf6622d6d5. * gnu/services/docker.scm (docker-configuration)[docker-cli]: New record field. (docker-service-type): Use this. * doc/guix.texi (Miscellaneous Services)[Docker Service]: Document this. --- doc/guix.texi | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index bfffb4ac2a..88128a4b3a 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -27990,7 +27990,10 @@ This is the data type representing the configuration of Docker and Containerd. @table @asis @item @code{package} (default: @code{docker}) -The Docker package to use. +The Docker daemon package to use. + +@item @code{package} (default: @code{docker-cli}) +The Docker client package to use. @item @code{containerd} (default: @var{containerd}) The Containerd package to use. -- cgit 1.4.1 From 125fc37e5f32afdbd1e5fca119c9eb41e7ad8ec1 Mon Sep 17 00:00:00 2001 From: Florian Pelz Date: Wed, 16 Sep 2020 15:18:08 +0200 Subject: doc: Clarify what propagated inputs are. Fixes . * doc/guix.texi (package Reference)[package-propagated-inputs]: Clarify. --- doc/guix.texi | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 88128a4b3a..f7e2204b53 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -59,7 +59,7 @@ Copyright @copyright{} 2018 Oleg Pykhalov@* Copyright @copyright{} 2018 Mike Gerwitz@* Copyright @copyright{} 2018 Pierre-Antoine Rouby@* Copyright @copyright{} 2018, 2019 Gábor Boskovits@* -Copyright @copyright{} 2018, 2019 Florian Pelz@* +Copyright @copyright{} 2018, 2019, 2020 Florian Pelz@* Copyright @copyright{} 2018 Laura Lazzati@* Copyright @copyright{} 2018 Alex Vong@* Copyright @copyright{} 2019 Josh Holland@* @@ -6370,21 +6370,22 @@ this area (@pxref{Invoking guix lint}). @anchor{package-propagated-inputs} Lastly, @code{propagated-inputs} is similar to @code{inputs}, but the -specified packages will be automatically installed alongside the package +specified packages will be automatically installed to profiles +(@pxref{Features, the role of profiles in Guix}) alongside the package they belong to (@pxref{package-cmd-propagated-inputs, @command{guix package}}, for information on how @command{guix package} deals with propagated inputs). -For example this is necessary when a C/C++ library needs headers of -another library to compile, or when a pkg-config file refers to another -one @i{via} its @code{Requires} field. +For example this is necessary when packaging a C/C++ library that needs +headers of another library to compile, or when a pkg-config file refers +to another one @i{via} its @code{Requires} field. Another example where @code{propagated-inputs} is useful is for languages that lack a facility to record the run-time search path akin to the @code{RUNPATH} of ELF files; this includes Guile, Python, Perl, and -more. To ensure that libraries written in those languages can find -library code they depend on at run time, run-time dependencies must be -listed in @code{propagated-inputs} rather than @code{inputs}. +more. When packaging libraries written in those languages, ensure they +can find library code they depend on at run time by listing run-time +dependencies in @code{propagated-inputs} rather than @code{inputs}. @item @code{outputs} (default: @code{'("out")}) The list of output names of the package. @xref{Packages with Multiple -- cgit 1.4.1 From aa9ba865deadf8b95c629daca6e57eb4f3edefbe Mon Sep 17 00:00:00 2001 From: Greg Hogan Date: Fri, 18 Sep 2020 14:29:05 -0400 Subject: doc: Fix broken hyperlinks in the contribution instructions. * doc/contributing.texi (Packaging Guidelines, Submitting Patches): Change URLs from relative to absolute links. --- doc/contributing.texi | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) (limited to 'doc') diff --git a/doc/contributing.texi b/doc/contributing.texi index b18ffa1e57..535cfc2a6f 100644 --- a/doc/contributing.texi +++ b/doc/contributing.texi @@ -353,7 +353,7 @@ Once your package builds correctly, please send us a patch (@pxref{Submitting Patches}). Well, if you need help, we will be happy to help you too. Once the patch is committed in the Guix repository, the new package automatically gets built on the supported platforms by -@url{@value{SUBSTITUTE-SERVER}, our continuous integration system}. +@url{@value{SUBSTITUTE-URL}, our continuous integration system}. @cindex substituter Users can obtain the new package definition simply by running @@ -1016,7 +1016,7 @@ changes). This branch is intended to be merged in @code{master} every 6 months or so. @end table -All these branches are @uref{@value{SUBSTITUTE-SERVER}, +All these branches are @uref{@value{SUBSTITUTE-URL}, tracked by our build farm} and merged into @code{master} once everything has been successfully built. This allows us to fix issues before they hit users, and to reduce the window during which pre-built -- cgit 1.4.1 From 9b65281de51bcb56714509524f5ae0731c9b96d0 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Mon, 14 Sep 2020 22:49:06 +0200 Subject: environment: '--link-profile' uses ~/.guix-profile for environment variables. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Before this patch, we had: $ guix environment -CP --ad-hoc coreutils [env]$ echo $PATH /gnu/store/…-profile/bin [env]$ echo $GUIX_ENVIRONMENT /gnu/store/…-profile After this patch: $ guix environment -CP --ad-hoc coreutils [env]$ echo $PATH /home/ludo/.guix-profile/bin [env]$ echo $GUIX_ENVIRONMENT /home/ludo/.guix-profile * guix/scripts/environment.scm (launch-environment/container): When LINK-PROFILE? is true, pass ~/.guix-profile as the second argument to 'launch-environment'. * tests/guix-environment-container.sh: Adjust test accordingly. * doc/guix.texi (Invoking guix environment): Update accordingly. --- doc/guix.texi | 5 +++-- guix/scripts/environment.scm | 6 +++++- tests/guix-environment-container.sh | 10 +++++++--- 3 files changed, 15 insertions(+), 6 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index f7e2204b53..949551a163 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -5420,8 +5420,9 @@ device. @item --link-profile @itemx -P For containers, link the environment profile to @file{~/.guix-profile} -within the container. This is equivalent to running the command -@samp{ln -s $GUIX_ENVIRONMENT ~/.guix-profile} within the container. +within the container and set @code{GUIX_ENVIRONMENT} to that. +This is equivalent to making @file{~/.guix-profile} a symlink to the +actual profile within the container. Linking will fail and abort the environment if the directory already exists, which will certainly be the case if @command{guix environment} was invoked in the user's home directory. diff --git a/guix/scripts/environment.scm b/guix/scripts/environment.scm index ad50281eb2..e2e481dd02 100644 --- a/guix/scripts/environment.scm +++ b/guix/scripts/environment.scm @@ -564,7 +564,11 @@ WHILE-LIST." (primitive-exit/status ;; A container's environment is already purified, so no need to ;; request it be purified again. - (launch-environment command profile manifest #:pure? #f))) + (launch-environment command + (if link-profile? + (string-append home-dir "/.guix-profile") + profile) + manifest #:pure? #f))) #:guest-uid uid #:guest-gid gid #:namespaces (if network? diff --git a/tests/guix-environment-container.sh b/tests/guix-environment-container.sh index 45264d4978..040f32cce9 100644 --- a/tests/guix-environment-container.sh +++ b/tests/guix-environment-container.sh @@ -127,11 +127,15 @@ grep -e "$NIX_STORE_DIR/.*-bash" $tmpdir/mounts # bootstrap bash rm $tmpdir/mounts -# Make sure 'GUIX_ENVIRONMENT' is linked to '~/.guix-profile' when requested +# Make sure 'GUIX_ENVIRONMENT' is set to '~/.guix-profile' when requested # within a container. ( - linktest='(exit (string=? (getenv "GUIX_ENVIRONMENT") -(readlink (string-append (getenv "HOME") "/.guix-profile"))))' + linktest=' +(exit (and (string=? (getenv "GUIX_ENVIRONMENT") + (string-append (getenv "HOME") "/.guix-profile")) + (string-prefix? "'"$NIX_STORE_DIR"'" + (readlink (string-append (getenv "HOME") + "/.guix-profile")))))' cd "$tmpdir" \ && guix environment --bootstrap --container --link-profile \ -- cgit 1.4.1 From 92e507c963470c6086a59ca8e1dd58ed4403649c Mon Sep 17 00:00:00 2001 From: Mathieu Othacehe Date: Thu, 24 Sep 2020 10:47:22 +0200 Subject: services: cuirass: Add SQL queries logging support. * gnu/services/cuirass.scm ()[queries-log-file]: New field. (cuirass-shepherd-service): Honor it. (cuirass-log-rotations): If defined, add the queries log file to the log rotation. --- doc/guix.texi | 4 ++++ gnu/services/cuirass.scm | 23 +++++++++++++++++++---- 2 files changed, 23 insertions(+), 4 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 949551a163..576845233a 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -23904,6 +23904,10 @@ Location of the log file. @item @code{web-log-file} (default: @code{"/var/log/cuirass-web.log"}) Location of the log file used by the web interface. +@item @code{queries-log-file} (default: @code{#f}) +Location of the SQL queries log file. By default, SQL queries logging is +disabled. + @item @code{cache-directory} (default: @code{"/var/cache/cuirass"}) Location of the repository cache. diff --git a/gnu/services/cuirass.scm b/gnu/services/cuirass.scm index 0f4f0f9948..44f40a632c 100644 --- a/gnu/services/cuirass.scm +++ b/gnu/services/cuirass.scm @@ -54,6 +54,8 @@ (default "/var/log/cuirass.log")) (web-log-file cuirass-configuration-web-log-file ;string (default "/var/log/cuirass-web.log")) + (queries-log-file cuirass-configuration-queries-log-file ;string + (default #f)) (cache-directory cuirass-configuration-cache-directory ;string (dir-name) (default "/var/cache/cuirass")) (ttl cuirass-configuration-ttl ;integer @@ -87,6 +89,7 @@ (cache-directory (cuirass-configuration-cache-directory config)) (web-log-file (cuirass-configuration-web-log-file config)) (log-file (cuirass-configuration-log-file config)) + (queries-log-file (cuirass-configuration-queries-log-file config)) (user (cuirass-configuration-user config)) (group (cuirass-configuration-group config)) (interval (cuirass-configuration-interval config)) @@ -111,6 +114,10 @@ "--database" #$database "--ttl" #$(string-append (number->string ttl) "s") "--interval" #$(number->string interval) + #$@(if queries-log-file + (list (string-append "--log-queries=" + queries-log-file)) + '()) #$@(if use-substitutes? '("--use-substitutes") '()) #$@(if one-shot? '("--one-shot") '()) #$@(if fallback? '("--fallback") '()) @@ -140,6 +147,10 @@ "--port" #$(number->string port) "--listen" #$host "--interval" #$(number->string interval) + #$@(if queries-log-file + (list (string-append "--log-queries=" + queries-log-file)) + '()) #$@(if use-substitutes? '("--use-substitutes") '()) #$@(if fallback? '("--fallback") '()) #$@extra-options) @@ -187,10 +198,14 @@ (define (cuirass-log-rotations config) "Return the list of log rotations that corresponds to CONFIG." - (list (log-rotation - (files (list (cuirass-configuration-log-file config))) - (frequency 'weekly) - (options '("rotate 40"))))) ;worth keeping + (let ((queries-log-file (cuirass-configuration-queries-log-file config))) + (list (log-rotation + (files `(,(cuirass-configuration-log-file config) + ,@(if queries-log-file + (list queries-log-file) + '()))) + (frequency 'weekly) + (options '("rotate 40")))))) ;worth keeping (define cuirass-service-type (service-type -- cgit 1.4.1 From 80e26d74294d40f311f61d13996f1ed1aa4d8edd Mon Sep 17 00:00:00 2001 From: Mathieu Othacehe Date: Thu, 24 Sep 2020 11:28:14 +0200 Subject: services: cuirass: Add web SQL queries logging support. * gnu/services/cuirass.scm ()[web-queries-log-file]: New field. (cuirass-shepherd-service): Honor it. (cuirass-log-rotations): If defined, add the web queries log file to the log rotation. --- doc/guix.texi | 4 ++++ gnu/services/cuirass.scm | 16 +++++++++++++--- 2 files changed, 17 insertions(+), 3 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 576845233a..6b2c749bc7 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -23908,6 +23908,10 @@ Location of the log file used by the web interface. Location of the SQL queries log file. By default, SQL queries logging is disabled. +@item @code{web-queries-log-file} (default: @code{#f}) +Location of the web SQL queries log file. By default, web SQL queries +logging is disabled. + @item @code{cache-directory} (default: @code{"/var/cache/cuirass"}) Location of the repository cache. diff --git a/gnu/services/cuirass.scm b/gnu/services/cuirass.scm index 44f40a632c..adf79af8ac 100644 --- a/gnu/services/cuirass.scm +++ b/gnu/services/cuirass.scm @@ -56,6 +56,9 @@ (default "/var/log/cuirass-web.log")) (queries-log-file cuirass-configuration-queries-log-file ;string (default #f)) + (web-queries-log-file + cuirass-configuration-web-queries-log-file ;string + (default #f)) (cache-directory cuirass-configuration-cache-directory ;string (dir-name) (default "/var/cache/cuirass")) (ttl cuirass-configuration-ttl ;integer @@ -90,6 +93,8 @@ (web-log-file (cuirass-configuration-web-log-file config)) (log-file (cuirass-configuration-log-file config)) (queries-log-file (cuirass-configuration-queries-log-file config)) + (web-queries-log-file + (cuirass-configuration-web-queries-log-file config)) (user (cuirass-configuration-user config)) (group (cuirass-configuration-group config)) (interval (cuirass-configuration-interval config)) @@ -147,9 +152,9 @@ "--port" #$(number->string port) "--listen" #$host "--interval" #$(number->string interval) - #$@(if queries-log-file + #$@(if web-queries-log-file (list (string-append "--log-queries=" - queries-log-file)) + web-queries-log-file)) '()) #$@(if use-substitutes? '("--use-substitutes") '()) #$@(if fallback? '("--fallback") '()) @@ -198,11 +203,16 @@ (define (cuirass-log-rotations config) "Return the list of log rotations that corresponds to CONFIG." - (let ((queries-log-file (cuirass-configuration-queries-log-file config))) + (let ((queries-log-file (cuirass-configuration-queries-log-file config)) + (web-queries-log-file + (cuirass-configuration-web-queries-log-file config))) (list (log-rotation (files `(,(cuirass-configuration-log-file config) ,@(if queries-log-file (list queries-log-file) + '()) + ,@(if web-queries-log-file + (list web-queries-log-file) '()))) (frequency 'weekly) (options '("rotate 40")))))) ;worth keeping -- cgit 1.4.1 From de96ed11efdfb450ca45952aceda656a78d981c4 Mon Sep 17 00:00:00 2001 From: Guillaume Le Vaillant Date: Thu, 24 Sep 2020 19:01:30 +0200 Subject: doc: Update documentation for asdf-build-system. * doc/guix.texi (Build Systems): Adapt the documentation of asd-build-system to the replacement of '#:asd-file' and '#:asd-system-name' by '#:asd-files' and '#:asd-systems'. Remove paragraph about one package per ASDF system. --- doc/guix.texi | 33 +++++++++++++++------------------ 1 file changed, 15 insertions(+), 18 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index a6e14ea177..22e1bfa089 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -68,7 +68,7 @@ Copyright @copyright{} 2019 Ivan Petkov@* Copyright @copyright{} 2019 Jakob L. Kreuze@* Copyright @copyright{} 2019 Kyle Andrews@* Copyright @copyright{} 2019 Alex Griffin@* -Copyright @copyright{} 2019 Guillaume Le Vaillant@* +Copyright @copyright{} 2019, 2020 Guillaume Le Vaillant@* Copyright @copyright{} 2020 Leo Prikler@* Copyright @copyright{} 2019, 2020 Simon Tournier@* Copyright @copyright{} 2020 Wiktor Żelazny@* @@ -6751,30 +6751,27 @@ Additionally, the corresponding source package should be labeled using the same convention as python packages (see @ref{Python Modules}), using the @code{cl-} prefix. -For binary packages, each system should be defined as a Guix package. -If one package @code{origin} contains several systems, package variants -can be created in order to build all the systems. Source packages, -which use @code{asdf-build-system/source}, may contain several systems. - In order to create executable programs and images, the build-side procedures @code{build-program} and @code{build-image} can be used. -They should be called in a build phase after the @code{create-symlinks} -phase, so that the system which was just built can be used within the -resulting image. @code{build-program} requires a list of Common Lisp -expressions to be passed as the @code{#:entry-program} argument. - -If the system is not defined within its own @file{.asd} file of the same -name, then the @code{#:asd-file} parameter should be used to specify -which file the system is defined in. Furthermore, if the package -defines a system for its tests in a separate file, it will be loaded -before the tests are run if it is specified by the +They should be called in a build phase after the +@code{create-asdf-configuration} phase, so that the system which was +just built can be used within the resulting image. @code{build-program} +requires a list of Common Lisp expressions to be passed as the +@code{#:entry-program} argument. + +By default, all the @file{.asd} files present in the sources are read to +find system definitions. The @code{#:asd-files} parameter can be used +to specify the list of @file{.asd} files to read. Furthermore, if the +package defines a system for its tests in a separate file, it will be +loaded before the tests are run if it is specified by the @code{#:test-asd-file} parameter. If it is not set, the files @code{-tests.asd}, @code{-test.asd}, @code{tests.asd}, and @code{test.asd} will be tried if they exist. If for some reason the package must be named in a different way than the -naming conventions suggest, the @code{#:asd-system-name} parameter can -be used to specify the name of the system. +naming conventions suggest, or if several systems must be compiled, the +@code{#:asd-systems} parameter can be used to specify the list of system +names. @end defvr -- cgit 1.4.1 From d48b17adb91d68acf6fb3f321c05102fcc8c39eb Mon Sep 17 00:00:00 2001 From: Marius Bakke Date: Thu, 24 Sep 2020 20:18:20 +0200 Subject: services: WPA Supplicant: Conditionally depend on D-Bus. Fixes . Reported by calcium . * gnu/services/networking.scm ()[requirement]: Remove 'dbus-system. (wpa-supplicant-shepherd-service)[requirement]: Add 'dbus-system when DBUS? is true. * doc/guix.texi (Networking Services)[wpa-supplicant-service-type]: Adjust accordingly. --- doc/guix.texi | 2 +- gnu/services/networking.scm | 6 ++++-- 2 files changed, 5 insertions(+), 3 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 6b2c749bc7..0fd56942e2 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -14603,7 +14603,7 @@ It takes the following parameters: @item @code{wpa-supplicant} (default: @code{wpa-supplicant}) The WPA Supplicant package to use. -@item @code{requirement} (default: @code{'(user-processes dbus-system loopback syslogd)} +@item @code{requirement} (default: @code{'(user-processes loopback syslogd)} List of services that should be started before WPA Supplicant starts. @item @code{dbus?} (default: @code{#t}) diff --git a/gnu/services/networking.scm b/gnu/services/networking.scm index e45b116218..64f54e787f 100644 --- a/gnu/services/networking.scm +++ b/gnu/services/networking.scm @@ -1324,7 +1324,7 @@ whatever the thing is supposed to do)."))) (wpa-supplicant wpa-supplicant-configuration-wpa-supplicant ; (default wpa-supplicant)) (requirement wpa-supplicant-configuration-requirement ;list of symbols - (default '(user-processes dbus-system loopback syslogd))) + (default '(user-processes loopback syslogd))) (pid-file wpa-supplicant-configuration-pid-file ;string (default "/var/run/wpa_supplicant.pid")) (dbus? wpa-supplicant-configuration-dbus? ;Boolean @@ -1343,7 +1343,9 @@ whatever the thing is supposed to do)."))) (list (shepherd-service (documentation "Run the WPA supplicant daemon") (provision '(wpa-supplicant)) - (requirement requirement) + (requirement (if dbus? + (cons 'dbus-system requirement) + requirement)) (start #~(make-forkexec-constructor (list (string-append #$wpa-supplicant "/sbin/wpa_supplicant") -- cgit 1.4.1 From fd67774d276e870c3f4f986a1a94dbd04aa81e23 Mon Sep 17 00:00:00 2001 From: André Batista Date: Thu, 24 Sep 2020 21:32:04 -0300 Subject: doc: Fix outdated info and add missing space. * doc/guix.texi (Messaging Services): Fix outdated info and add missing space. Signed-off-by: Mathieu Othacehe --- doc/guix.texi | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 0fd56942e2..a66ab82b31 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -80,6 +80,7 @@ Copyright @copyright{} 2020 Brice Waegeneire@* Copyright @copyright{} 2020 R Veera Kumar@* Copyright @copyright{} 2020 Pierre Langlois@* Copyright @copyright{} 2020 pinoaffe@* +Copyright @copyright{} 2020 André Batista@* Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or @@ -19526,7 +19527,8 @@ Mailutils Manual}, for details. @cindex jabber @cindex XMPP The @code{(gnu services messaging)} module provides Guix service -definitions for messaging services: currently only Prosody is supported. +definitions for messaging services. Currently it provides the following +services: @subsubheading Prosody Service @@ -19716,7 +19718,7 @@ can create such a file with: @end deftypevr @deftypevr {@code{ssl-configuration} parameter} maybe-string curve -Curve for Elliptic curve Diffie-Hellman. Prosody's default is +Curve for Elliptic curve Diffie-Hellman. Prosody's default is @samp{"secp384r1"}. @end deftypevr -- cgit 1.4.1 From 13a2272d1af1626fff1480ecba35156b52007dde Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Fri, 25 Sep 2020 23:13:19 +0200 Subject: doc: Explain how to connect to a childhurd. * doc/guix.texi (The Hurd in a Virtual Machine): Add instructions for VNC and SSH access. Mention childhurd secrets and /etc/childhurd. --- doc/guix.texi | 33 +++++++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index a66ab82b31..2dff2ad2a8 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -25424,16 +25424,41 @@ Return the name of @var{platform}---a string such as @code{"arm"}. @cindex childhurd Service @code{hurd-vm} provides support for running GNU/Hurd in a -virtual machine (VM), a so-called ``Childhurd''. The virtual machine is -a Shepherd service that can be referred to by the names @code{hurd-vm} -and @code{childhurd} and be controlled with commands such as: +virtual machine (VM), a so-called @dfn{childhurd}. This service is meant +to be used on GNU/Linux and the given GNU/Hurd operating system +configuration is cross-compiled. The virtual machine is a Shepherd +service that can be referred to by the names @code{hurd-vm} and +@code{childhurd} and be controlled with commands such as: @example herd start hurd-vm herd stop childhurd @end example -The given GNU/Hurd operating system configuration is cross-compiled. +When the service is running, you can view its console by connecting to +it with a VNC client, for example with: + +@example +guix environment --ad-hoc tigervnc-client -- \ + vncviewer localhost:5900 +@end example + +The default configuration (see @code{hurd-vm-configuration} below) +spawns a secure shell (SSH) server in your GNU/Hurd system, which QEMU +(the virtual machine emulator) redirects to port 10222 on the host. +Thus, you can connect over SSH to the childhurd with: + +@example +ssh root@@localhost -p 10022 +@end example + +The childhurd is volatile and stateless: it starts with a fresh root +file system every time you restart it. By default though, all the files +under @file{/etc/childhurd} on the host are copied as is to the root +file system of the childhurd when it boots. This allows you to +initialize ``secrets'' inside the VM: SSH host keys, authorized +substitute keys, and so on---see the explanation of @code{secret-root} +below. @defvr {Scheme Variable} hurd-vm-service-type This is the type of the Hurd in a Virtual Machine service. Its value -- cgit 1.4.1 From e71d7b5e236840ca31b05bcfb669fd0917254f26 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Fri, 25 Sep 2020 23:21:43 +0200 Subject: doc: Remove "--hda" from non-volatile childhurd example. Reported by Andreas Enge. * doc/guix.texi (Transparent Emulation with QEMU): Remove "--hda" in 'image' example. --- doc/guix.texi | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 2dff2ad2a8..9986df02cc 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -25566,7 +25566,7 @@ the @code{--snapshot} flag using something along these lines: (service hurd-vm-service-type (hurd-vm-configuration (image (const "/out/of/store/writable/hurd.img")) - (options '("--hda")))) + (options '()))) @end lisp @subsubheading Ganeti -- cgit 1.4.1 From a0d4aa2457d7e36012143ffe3fbc9dd4bc20ca4f Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Fri, 25 Sep 2020 23:25:17 +0200 Subject: doc: Use @var for meta-syntactic variables. * doc/guix.texi (Transparent Emulation with QEMU): Use @var instead of angle brackets for meta-syntactic variables. --- doc/guix.texi | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 9986df02cc..82241b010a 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -25519,15 +25519,17 @@ By default, it produces @lisp '("--device" "rtl8139,netdev=net0" "--netdev" "user,id=net0\ - ,hostfwd=tcp:127.0.0.1:-:1004\ - ,hostfwd=tcp:127.0.0.1:-:2222\ - ,hostfwd=tcp:127.0.0.1:-:5900") + ,hostfwd=tcp:127.0.0.1:@var{secrets-port}-:1004\ + ,hostfwd=tcp:127.0.0.1:@var{ssh-port}-:2222\ + ,hostfwd=tcp:127.0.0.1:@var{vnc-port}-:5900") @end lisp -with forwarded ports + +with forwarded ports: + @example -: @code{(+ 11004 (* 1000 @var{ID}))} -: @code{(+ 10022 (* 1000 @var{ID}))} -: @code{(+ 15900 (* 1000 @var{ID}))} +@var{ssh-port}: @code{(+ 11004 (* 1000 @var{ID}))} +@var{ssh-port}: @code{(+ 10022 (* 1000 @var{ID}))} +@var{vnc-port}: @code{(+ 15900 (* 1000 @var{ID}))} @end example @item @code{secret-root} (default: @file{/etc/childhurd}) -- cgit 1.4.1 From 740fd97ebeadebc02448cb0482084f359938b5fe Mon Sep 17 00:00:00 2001 From: Stefan Date: Sun, 20 Sep 2020 00:55:56 +0200 Subject: doc: Document new bootloader grub-efi-netboot-bootloader. * doc/guix.texi (Bootloader Configuration)[bootloader]: Add grub-efi-netboot-bootloader. [target]: Document TFTP root directory for grub-efi-netboot-bootloader. Signed-off-by: Danny Milosavljevic --- doc/guix.texi | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 58 insertions(+), 8 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 82241b010a..538e7cceab 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -28682,7 +28682,15 @@ The type of a bootloader configuration declaration. @cindex BIOS, bootloader The bootloader to use, as a @code{bootloader} object. For now @code{grub-bootloader}, @code{grub-efi-bootloader}, -@code{extlinux-bootloader} and @code{u-boot-bootloader} are supported. +@code{grub-efi-netboot-bootloader}, @code{extlinux-bootloader} and +@code{u-boot-bootloader} are supported. + +@cindex ARM, bootloaders +@cindex AArch64, bootloaders +Available bootloaders are described in @code{(gnu bootloader @dots{})} +modules. In particular, @code{(gnu bootloader u-boot)} contains definitions +of bootloaders for a wide range of ARM and AArch64 systems, using the +@uref{https://www.denx.de/wiki/U-Boot/, U-Boot bootloader}. @vindex grub-efi-bootloader @code{grub-efi-bootloader} allows to boot on modern systems using the @@ -28694,12 +28702,52 @@ when you boot it on your system. @code{grub-bootloader} allows you to boot in particular Intel-based machines in ``legacy'' BIOS mode. -@cindex ARM, bootloaders -@cindex AArch64, bootloaders -Available bootloaders are described in @code{(gnu bootloader @dots{})} -modules. In particular, @code{(gnu bootloader u-boot)} contains definitions -of bootloaders for a wide range of ARM and AArch64 systems, using the -@uref{https://www.denx.de/wiki/U-Boot/, U-Boot bootloader}. +@vindex grub-efi-netboot-bootloader +@code{grub-efi-netboot-bootloader} allows you to boot your system over network +through TFTP. In combination with an NFS root file system this allows you to +build a diskless Guix system. + +The installation of the @code{grub-efi-netboot-bootloader} generates the content +of the TFTP root directory at @code{target} +(@pxref{Bootloader Configuration, @code{target}}), to be served by a TFTP server. + You may want to mount your TFTP server directory onto @code{target} to move the +required files to the TFTP server automatically. + +If you plan to use an NFS root file system as well (actually if you mount the +store from an NFS share), then the TFTP server needs to serve the file +@file{/boot/grub/grub.cfg} and other files from the store (like GRUBs background +image, the kernel (@pxref{operating-system Reference, @code{kernel}}) and the +initrd (@pxref{operating-system Reference, @code{initrd}})), too. All these +files from the store will be accessed by GRUB through TFTP with their normal +store path, for example as +@file{tftp://tftp-server/gnu/store/…-initrd/initrd.cpio.gz}. + +Two symlinks are created to make this possible. The first symlink is +@code{target}@file{/efi/Guix/boot/grub/grub.cfg} pointing to +@file{../../../boot/grub/grub.cfg}, +where @code{target} may be @file{/boot}. In this case the link is not leaving +the served TFTP root directory, but otherwise it does. The second link is +@code{target}@file{/gnu/store} and points to @file{../gnu/store}. This link +is leaving the served TFTP root directory. + +The assumption behind all this is that you have an NFS server exporting the root +file system for your Guix system, and additionally a TFTP server exporting your +@code{target} directory—usually @file{/boot}—from that same root file system for +your Guix system. In this constellation the symlinks will work. + +For other constellations you will have to program your own bootloader installer, +which then takes care to make necessary files from the store accessible through +TFTP, for example by copying them into the TFTP root directory at @code{target}. + +It is important to note that symlinks pointing outside the TFTP root directory +may need to be allowed in the configuration of your TFTP server. Further the +store link exposes the whole store through TFTP. Both points need to be +considered carefully for security aspects. + +Beside the @code{grub-efi-netboot-bootloader}, the already mentioned TFTP and +NFS servers, you also need a properly configured DHCP server to make the booting +over netboot possible. For all this we can currently only recommend you to look +for instructions about @acronym{PXE, Preboot eXecution Environment}. @item @code{target} This is a string denoting the target onto which to install the @@ -28710,7 +28758,9 @@ The interpretation depends on the bootloader in question. For the bootloader @command{installer} command, such as @code{/dev/sda} or @code{(hd0)} (@pxref{Invoking grub-install,,, grub, GNU GRUB Manual}). For @code{grub-efi-bootloader}, it should be the mount point of the EFI file -system, usually @file{/boot/efi}. +system, usually @file{/boot/efi}. For @code{grub-efi-netboot-bootloader}, +@code{target} should be the mount point corresponding to the TFTP root +directory of your TFTP server. @item @code{menu-entries} (default: @code{()}) A possibly empty list of @code{menu-entry} objects (see below), denoting -- cgit 1.4.1 From f458cfbcc54ed87b1a87dd9e150ea276f17eab74 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Wed, 23 Sep 2020 22:29:17 +0200 Subject: guix build: Add '--without-tests'. * guix/scripts/build.scm (transform-package-tests): New procedure. (%transformations, %transformation-options) show-transformation-options-help): Add it. * tests/scripts-build.scm ("options->transformation, without-tests"): New test. * doc/guix.texi (Package Transformation Options): Document it. --- doc/guix.texi | 22 ++++++++++++++++++++++ guix/scripts/build.scm | 31 ++++++++++++++++++++++++++++--- tests/scripts-build.scm | 14 ++++++++++++++ 3 files changed, 64 insertions(+), 3 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 538e7cceab..8384eee6c3 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -9271,6 +9271,28 @@ guix build --with-branch=guile-sqlite3=master cuirass This is similar to @option{--with-branch}, except that it builds from @var{commit} rather than the tip of a branch. @var{commit} must be a valid Git commit SHA1 identifier or a tag. + +@cindex test suite, skipping +@item --without-tests=@var{package} +Build @var{package} without running its tests. This can be useful in +situations where you want to skip the lengthy test suite of a +intermediate package, or if a package's test suite fails in a +non-deterministic fashion. It should be used with care because running +the test suite is a good way to ensure a package is working as intended. + +Turning off tests leads to a different store item. Consequently, when +using this option, anything that depends on @var{package} must be +rebuilt, as in this example: + +@example +guix install --without-tests=python python-notebook +@end example + +The command above installs @code{python-notebook} on top of +@code{python} built without running its test suite. To do so, it also +rebuilds everything that depends on @code{python}, including +@code{python-notebook} itself. + @end table @node Additional Build Options diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm index 38e0516c95..f238e9b876 100644 --- a/guix/scripts/build.scm +++ b/guix/scripts/build.scm @@ -393,6 +393,25 @@ a checkout of the Git repository at the given URL." (rewrite obj) obj))) +(define (transform-package-tests specs) + "Return a procedure that, when passed a package, sets #:tests? #f in its +'arguments' field." + (define (package-without-tests p) + (package/inherit p + (arguments + (substitute-keyword-arguments (package-arguments p) + ((#:tests? _ #f) #f))))) + + (define rewrite + (package-input-rewriting/spec (map (lambda (spec) + (cons spec package-without-tests)) + specs))) + + (lambda (store obj) + (if (package? obj) + (rewrite obj) + obj))) + (define %transformations ;; Transformations that can be applied to things to build. The car is the ;; key used in the option alist, and the cdr is the transformation @@ -403,7 +422,8 @@ a checkout of the Git repository at the given URL." (with-graft . ,transform-package-inputs/graft) (with-branch . ,transform-package-source-branch) (with-commit . ,transform-package-source-commit) - (with-git-url . ,transform-package-source-git-url))) + (with-git-url . ,transform-package-source-git-url) + (without-tests . ,transform-package-tests))) (define %transformation-options ;; The command-line interface to the above transformations. @@ -423,7 +443,9 @@ a checkout of the Git repository at the given URL." (option '("with-commit") #t #f (parser 'with-commit)) (option '("with-git-url") #t #f - (parser 'with-git-url))))) + (parser 'with-git-url)) + (option '("without-tests") #t #f + (parser 'without-tests))))) (define (show-transformation-options-help) (display (G_ " @@ -443,7 +465,10 @@ a checkout of the Git repository at the given URL." build PACKAGE from COMMIT")) (display (G_ " --with-git-url=PACKAGE=URL - build PACKAGE from the repository at URL"))) + build PACKAGE from the repository at URL")) + (display (G_ " + --without-tests=PACKAGE + build PACKAGE without running its tests"))) (define (options->transformation opts) diff --git a/tests/scripts-build.scm b/tests/scripts-build.scm index 32876e956a..12114fc8f5 100644 --- a/tests/scripts-build.scm +++ b/tests/scripts-build.scm @@ -264,5 +264,19 @@ ((("x" dep3)) (map package-source (list dep1 dep3)))))))))))) +(test-assert "options->transformation, without-tests" + (let* ((dep (dummy-package "dep")) + (p (dummy-package "foo" + (inputs `(("dep" ,dep))))) + (t (options->transformation '((without-tests . "dep") + (without-tests . "tar"))))) + (with-store store + (let ((new (t store p))) + (match (bag-direct-inputs (package->bag new)) + ((("dep" dep) ("tar" tar) _ ...) + ;; TODO: Check whether TAR has #:tests? #f when transformations + ;; apply to implicit inputs. + (equal? (package-arguments dep) + '(#:tests? #f)))))))) (test-end) -- cgit 1.4.1 From ff39361c80dfc67a9afe35f315a774140d8cf99b Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Mon, 21 Sep 2020 17:44:29 +0200 Subject: packages: 'package-mapping' can recurse on implicit inputs. * guix/packages.scm (build-system-with-package-mapping): New procedure. (package-mapping): Add #:deep? and honor it. * tests/packages.scm ("package-mapping"): Compare the direct inputs of the bag of P0 and that of P1. ("package-mapping, deep"): New test. --- doc/guix.texi | 5 +++-- guix/packages.scm | 65 +++++++++++++++++++++++++++++++++++++++++------------- tests/packages.scm | 36 +++++++++++++++++++++++++++++- 3 files changed, 88 insertions(+), 18 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 8384eee6c3..054449d8d6 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -6296,10 +6296,11 @@ A more generic procedure to rewrite a package dependency graph is @code{package-mapping}: it supports arbitrary changes to nodes in the graph. -@deffn {Scheme Procedure} package-mapping @var{proc} [@var{cut?}] +@deffn {Scheme Procedure} package-mapping @var{proc} [@var{cut?}] [#:deep? #f] Return a procedure that, given a package, applies @var{proc} to all the packages depended on and returns the resulting package. The procedure stops recursion -when @var{cut?} returns true for a given package. +when @var{cut?} returns true for a given package. When @var{deep?} is true, @var{proc} is +applied to implicit inputs as well. @end deffn @menu diff --git a/guix/packages.scm b/guix/packages.scm index 6598bd3149..171fd048ef 100644 --- a/guix/packages.scm +++ b/guix/packages.scm @@ -968,10 +968,31 @@ packages they depend on, recursively." (vhash-consq package #t visited) (fold set-insert closure dependencies)))))))) -(define* (package-mapping proc #:optional (cut? (const #f))) +(define (build-system-with-package-mapping bs rewrite) + "Return a variant of BS, a build system, that rewrites a bag's inputs by +passing them through REWRITE, a procedure that takes an input tuplet and +returns a \"rewritten\" input tuplet." + (define lower + (build-system-lower bs)) + + (define (lower* . args) + (let ((lowered (apply lower args))) + (bag + (inherit lowered) + (build-inputs (map rewrite (bag-build-inputs lowered))) + (host-inputs (map rewrite (bag-host-inputs lowered))) + (target-inputs (map rewrite (bag-target-inputs lowered)))))) + + (build-system + (inherit bs) + (lower lower*))) + +(define* (package-mapping proc #:optional (cut? (const #f)) + #:key deep?) "Return a procedure that, given a package, applies PROC to all the packages depended on and returns the resulting package. The procedure stops recursion -when CUT? returns true for a given package." +when CUT? returns true for a given package. When DEEP? is true, PROC is +applied to implicit inputs as well." (define (rewrite input) (match input ((label (? package? package) outputs ...) @@ -980,21 +1001,35 @@ when CUT? returns true for a given package." (_ input))) + (define mapping-property + ;; Property indicating whether the package has already been processed. + (gensym " package-mapping-done")) + (define replace (mlambdaq (p) - ;; Return a variant of P with PROC applied to P and its explicit - ;; dependencies, recursively. Memoize the transformations. Failing to - ;; do that, we would build a huge object graph with lots of duplicates, - ;; which in turns prevents us from benefiting from memoization in - ;; 'package-derivation'. - (let ((p (proc p))) - (package - (inherit p) - (location (package-location p)) - (inputs (map rewrite (package-inputs p))) - (native-inputs (map rewrite (package-native-inputs p))) - (propagated-inputs (map rewrite (package-propagated-inputs p))) - (replacement (and=> (package-replacement p) proc)))))) + ;; If P is the result of a previous call, return it. + (if (assq-ref (package-properties p) mapping-property) + p + + ;; Return a variant of P with PROC applied to P and its explicit + ;; dependencies, recursively. Memoize the transformations. Failing + ;; to do that, we would build a huge object graph with lots of + ;; duplicates, which in turns prevents us from benefiting from + ;; memoization in 'package-derivation'. + (let ((p (proc p))) + (package + (inherit p) + (location (package-location p)) + (build-system (if deep? + (build-system-with-package-mapping + (package-build-system p) rewrite) + (package-build-system p))) + (inputs (map rewrite (package-inputs p))) + (native-inputs (map rewrite (package-native-inputs p))) + (propagated-inputs (map rewrite (package-propagated-inputs p))) + (replacement (and=> (package-replacement p) proc)) + (properties `((,mapping-property . #t) + ,@(package-properties p)))))))) replace) diff --git a/tests/packages.scm b/tests/packages.scm index cbd0503733..f33332a461 100644 --- a/tests/packages.scm +++ b/tests/packages.scm @@ -1172,15 +1172,24 @@ (let* ((dep (dummy-package "chbouib" (native-inputs `(("x" ,grep))))) (p0 (dummy-package "example" + (source 77) (inputs `(("foo" ,coreutils) ("bar" ,grep) ("baz" ,dep))))) (transform (lambda (p) (package (inherit p) (source 42)))) (rewrite (package-mapping transform)) - (p1 (rewrite p0))) + (p1 (rewrite p0)) + (bag0 (package->bag p0)) + (bag1 (package->bag p1))) (and (eq? p1 (rewrite p0)) (eqv? 42 (package-source p1)) + + ;; Implicit inputs should be left unchanged (skip "source", "foo", + ;; "bar", and "baz" in this comparison). + (equal? (drop (bag-direct-inputs bag0) 4) + (drop (bag-direct-inputs bag1) 4)) + (match (package-inputs p1) ((("foo" dep1) ("bar" dep2) ("baz" dep3)) (and (eq? dep1 (rewrite coreutils)) ;memoization @@ -1194,6 +1203,31 @@ (and (eq? dep (rewrite grep)) (package-source dep)))))))))) +(test-equal "package-mapping, deep" + '(42) + (let* ((p0 (dummy-package "example" + (inputs `(("foo" ,coreutils) + ("bar" ,grep))))) + (transform (lambda (p) + (package (inherit p) (source 42)))) + (rewrite (package-mapping transform #:deep? #t)) + (p1 (rewrite p0)) + (bag (package->bag p1))) + (and (eq? p1 (rewrite p0)) + (match (bag-direct-inputs bag) + ((("source" 42) ("foo" dep1) ("bar" dep2) rest ..1) + (and (eq? dep1 (rewrite coreutils)) ;memoization + (eq? dep2 (rewrite grep)) + (= 42 (package-source dep1)) + (= 42 (package-source dep2)) + + ;; Check that implicit inputs of P0 also got rewritten. + (delete-duplicates + (map (match-lambda + ((_ package . _) + (package-source package))) + rest)))))))) + (test-assert "package-input-rewriting" (let* ((dep (dummy-package "chbouib" (native-inputs `(("x" ,grep))))) -- cgit 1.4.1 From 2bf6f962b91123b0474c0f7123cd17efe7f09a66 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Wed, 23 Sep 2020 10:29:09 +0200 Subject: packages: 'package-input-rewriting/spec' can rewrite implicit dependencies. With this change, '--with-input', '--with-graft', etc. also apply to implicit dependencies. Thus, it's now possible to do: guix build python-itsdangerous --with-input=python-wrapper=python@2 or: guix build hello --with-graft=glibc=glibc@2.29 Additionally, before, implicit inputs were not rewritten, which could lead to duplicates in the output of 'bag-transitive-inputs' (packages that are not 'eq?' but lead to the same derivation). This in turn would lead to unnecessary rebuilds when using '--with-input' & co. This change fixes it by ensuring even implicit inputs are rewritten. Fixes . * guix/packages.scm (package-input-rewriting/spec): Add #:deep? defaulting to #true, and pass it to 'package-mapping'. [replacement-property]: New variable. [rewrite]: Check that property and set it on the result of PROC. [cut?]: New procedure. * tests/packages.scm ("package-input-rewriting/spec"): Ensure implicit inputs were unchanged. ("package-input-rewriting/spec, partial match"): Pass #:deep? #f. ("package-input-rewriting/spec, deep") ("package-input-rewriting/spec, no duplicates"): New tests. (package/inherit): Move before use. * tests/guix-build.sh: Add tests. * tests/scripts-build.scm ("options->transformation, with-graft"): Compare dependencies by package name or derivation file name. * doc/guix.texi (Defining Packages): Adjust accordingly. --- doc/guix.texi | 13 +++++----- guix/packages.scm | 55 ++++++++++++++++++++++++++--------------- tests/guix-build.sh | 11 +++++++++ tests/packages.scm | 66 ++++++++++++++++++++++++++++++++++++++++++++++--- tests/scripts-build.scm | 12 ++++++--- 5 files changed, 125 insertions(+), 32 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 054449d8d6..e72e1ec130 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -6272,12 +6272,13 @@ This is exactly what the @option{--with-input} command-line option does The following variant of @code{package-input-rewriting} can match packages to be replaced by name rather than by identity. -@deffn {Scheme Procedure} package-input-rewriting/spec @var{replacements} -Return a procedure that, given a package, applies the given @var{replacements} to -all the package graph (excluding implicit inputs). @var{replacements} is a list of -spec/procedures pair; each spec is a package specification such as @code{"gcc"} or -@code{"guile@@2"}, and each procedure takes a matching package and returns a -replacement for that package. +@deffn {Scheme Procedure} package-input-rewriting/spec @var{replacements} [#:deep? #t] +Return a procedure that, given a package, applies the given +@var{replacements} to all the package graph, including implicit inputs +unless @var{deep?} is false. @var{replacements} is a list of +spec/procedures pair; each spec is a package specification such as +@code{"gcc"} or @code{"guile@@2"}, and each procedure takes a matching +package and returns a replacement for that package. @end deffn The example above could be rewritten this way: diff --git a/guix/packages.scm b/guix/packages.scm index 171fd048ef..f696945e30 100644 --- a/guix/packages.scm +++ b/guix/packages.scm @@ -422,6 +422,16 @@ name of its URI." package) 16))))) +(define-syntax-rule (package/inherit p overrides ...) + "Like (package (inherit P) OVERRIDES ...), except that the same +transformation is done to the package replacement, if any. P must be a bare +identifier, and will be bound to either P or its replacement when evaluating +OVERRIDES." + (let loop ((p p)) + (package (inherit p) + overrides ... + (replacement (and=> (package-replacement p) loop))))) + (define (package-upstream-name package) "Return the upstream name of PACKAGE, which could be different from the name it has in Guix." @@ -1051,12 +1061,12 @@ package and returns its new name after rewrite." (package-mapping rewrite (cut assq <> replacements))) -(define (package-input-rewriting/spec replacements) +(define* (package-input-rewriting/spec replacements #:key (deep? #t)) "Return a procedure that, given a package, applies the given REPLACEMENTS to -all the package graph (excluding implicit inputs). REPLACEMENTS is a list of -spec/procedures pair; each spec is a package specification such as \"gcc\" or -\"guile@2\", and each procedure takes a matching package and returns a -replacement for that package." +all the package graph, including implicit inputs unless DEEP? is false. +REPLACEMENTS is a list of spec/procedures pair; each spec is a package +specification such as \"gcc\" or \"guile@2\", and each procedure takes a +matching package and returns a replacement for that package." (define table (fold (lambda (replacement table) (match replacement @@ -1081,22 +1091,27 @@ replacement for that package." (package-name package) table)) - (define (rewrite package) - (match (find-replacement package) - (#f package) - (proc (proc package)))) - - (package-mapping rewrite find-replacement)) + (define replacement-property + (gensym " package-replacement")) -(define-syntax-rule (package/inherit p overrides ...) - "Like (package (inherit P) OVERRIDES ...), except that the same -transformation is done to the package replacement, if any. P must be a bare -identifier, and will be bound to either P or its replacement when evaluating -OVERRIDES." - (let loop ((p p)) - (package (inherit p) - overrides ... - (replacement (and=> (package-replacement p) loop))))) + (define (rewrite p) + (if (assq-ref (package-properties p) replacement-property) + p + (match (find-replacement p) + (#f p) + (proc + (let ((new (proc p))) + ;; Mark NEW as already processed. + (package/inherit new + (properties `((,replacement-property . #t) + ,@(package-properties new))))))))) + + (define (cut? p) + (or (assq-ref (package-properties p) replacement-property) + (find-replacement p))) + + (package-mapping rewrite cut? + #:deep? deep?)) ;;; diff --git a/tests/guix-build.sh b/tests/guix-build.sh index 6c08857358..ec2f736ccb 100644 --- a/tests/guix-build.sh +++ b/tests/guix-build.sh @@ -259,6 +259,17 @@ drv1=`guix build guile -d` drv2=`guix build guile --with-input=gimp=ruby -d` test "$drv1" = "$drv2" +# See . +drv1=`guix build glib -d` +drv2=`guix build glib -d --with-input=libreoffice=inkscape` +test "$drv1" = "$drv2" + +# Rewriting implicit inputs. +drv1=`guix build hello -d` +drv2=`guix build hello -d --with-input=gcc=gcc-toolchain` +test "$drv1" != "$drv2" +guix gc -R "$drv2" | grep `guix build -d gcc-toolchain` + if guix build guile --with-input=libunistring=something-really-silly then false; else true; fi diff --git a/tests/packages.scm b/tests/packages.scm index f33332a461..6fa4ad2f1b 100644 --- a/tests/packages.scm +++ b/tests/packages.scm @@ -38,6 +38,7 @@ #:use-module (guix build-system) #:use-module (guix build-system trivial) #:use-module (guix build-system gnu) + #:use-module (guix build-system python) #:use-module (guix memoization) #:use-module (guix profiles) #:use-module (guix scripts package) @@ -45,6 +46,7 @@ #:use-module (gnu packages base) #:use-module (gnu packages guile) #:use-module (gnu packages bootstrap) + #:use-module (gnu packages python) #:use-module (gnu packages version-control) #:use-module (gnu packages xml) #:use-module (srfi srfi-1) @@ -1262,7 +1264,8 @@ ("baz" ,dep))))) (rewrite (package-input-rewriting/spec `(("coreutils" . ,(const sed)) - ("grep" . ,(const findutils))))) + ("grep" . ,(const findutils))) + #:deep? #f)) (p1 (rewrite p0)) (p2 (rewrite p0))) (and (not (eq? p1 p0)) @@ -1279,7 +1282,11 @@ (match (package-native-inputs dep3) ((("x" dep)) (string=? (package-full-name dep) - (package-full-name findutils)))))))))) + (package-full-name findutils))))))) + + ;; Make sure implicit inputs were left unchanged. + (equal? (drop (bag-direct-inputs (package->bag p1)) 3) + (drop (bag-direct-inputs (package->bag p0)) 3))))) (test-assert "package-input-rewriting/spec, partial match" (let* ((dep (dummy-package "chbouib" @@ -1290,7 +1297,8 @@ ("bar" ,dep))))) (rewrite (package-input-rewriting/spec `(("chbouib@123" . ,(const sed)) ;not matched - ("grep" . ,(const findutils))))) + ("grep" . ,(const findutils))) + #:deep? #f)) (p1 (rewrite p0))) (and (not (eq? p1 p0)) (string=? "example" (package-name p1)) @@ -1304,6 +1312,58 @@ (string=? (package-full-name dep) (package-full-name findutils)))))))))) +(test-assert "package-input-rewriting/spec, deep" + (let* ((dep (dummy-package "chbouib")) + (p0 (dummy-package "example" + (build-system gnu-build-system) + (inputs `(("dep" ,dep))))) + (rewrite (package-input-rewriting/spec + `(("tar" . ,(const sed)) + ("gzip" . ,(const findutils))))) + (p1 (rewrite p0)) + (p2 (rewrite p0))) + (and (not (eq? p1 p0)) + (eq? p1 p2) ;memoization + (string=? "example" (package-name p1)) + (match (package-inputs p1) + ((("dep" dep1)) + (and (string=? (package-full-name dep1) + (package-full-name dep)) + (eq? dep1 (rewrite dep))))) ;memoization + + ;; Make sure implicit inputs were replaced. + (match (bag-direct-inputs (package->bag p1)) + ((("dep" dep1) ("tar" tar) ("gzip" gzip) _ ...) + (and (eq? dep1 (rewrite dep)) + (string=? (package-full-name tar) + (package-full-name sed)) + (string=? (package-full-name gzip) + (package-full-name findutils)))))))) + +(test-assert "package-input-rewriting/spec, no duplicates" + ;; Ensure that deep input rewriting does not forget implicit inputs. Doing + ;; so could lead to duplicates in a package's inputs: in the example below, + ;; P0's transitive inputs would contain one rewritten "python" and one + ;; original "python". These two "python" packages are thus not 'eq?' but + ;; they lower to the same derivation. See , + ;; which can be reproduced by passing #:deep? #f. + (let* ((dep0 (dummy-package "dep0" + (build-system trivial-build-system) + (propagated-inputs `(("python" ,python))))) + (p0 (dummy-package "chbouib" + (build-system python-build-system) + (arguments `(#:python ,python)) + (inputs `(("dep0" ,dep0))))) + (rewrite (package-input-rewriting/spec '() #:deep? #t)) + (p1 (rewrite p0)) + (bag1 (package->bag p1)) + (pythons (filter-map (match-lambda + (("python" python) python) + (_ #f)) + (bag-transitive-inputs bag1)))) + (match (delete-duplicates pythons eq?) + ((p) (eq? p (rewrite python)))))) + (test-equal "package-patched-vulnerabilities" '(("CVE-2015-1234") ("CVE-2016-1234" "CVE-2018-4567") diff --git a/tests/scripts-build.scm b/tests/scripts-build.scm index 12114fc8f5..5f91360953 100644 --- a/tests/scripts-build.scm +++ b/tests/scripts-build.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2016, 2017, 2019 Ludovic Courtès +;;; Copyright © 2016, 2017, 2019, 2020 Ludovic Courtès ;;; ;;; This file is part of GNU Guix. ;;; @@ -19,6 +19,7 @@ (define-module (test-scripts-build) #:use-module (guix tests) #:use-module (guix store) + #:use-module (guix derivations) #:use-module (guix packages) #:use-module (guix git-download) #:use-module (guix scripts build) @@ -163,11 +164,16 @@ ((("foo" dep1) ("bar" dep2)) (and (string=? (package-full-name dep1) (package-full-name grep)) - (eq? (package-replacement dep1) findutils) + (string=? (package-full-name (package-replacement dep1)) + (package-full-name findutils)) (string=? (package-name dep2) "chbouib") (match (package-native-inputs dep2) ((("x" dep)) - (eq? (package-replacement dep) findutils))))))))))) + (with-store store + (string=? (derivation-file-name + (package-derivation store findutils)) + (derivation-file-name + (package-derivation store dep)))))))))))))) (test-equal "options->transformation, with-branch" (git-checkout (url "https://example.org") -- cgit 1.4.1 From 8819551c8d2a12cd4e84e09b51e434d05a012c9d Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Wed, 23 Sep 2020 14:56:38 +0200 Subject: packages: 'package-input-rewriting' has a #:deep? parameter. * guix/packages.scm (package-input-rewriting): Add #:deep? and pass it to 'package-mapping'. [replacement-property]: New variable. [rewrite]: Check it. [cut?]: New procedure. * tests/packages.scm ("package-input-rewriting"): Pass #:deep? #f and ensure implicit inputs were not rewritten. Avoid 'eq?' comparisons. ("package-input-rewriting, deep"): New test. * gnu/packages/guile.scm (package-for-guile-2.0, package-for-guile-3.0): Pass #:deep? #f. --- doc/guix.texi | 10 +++++----- gnu/packages/guile.scm | 6 ++++-- guix/packages.scm | 35 +++++++++++++++++++++++++---------- tests/packages.scm | 20 ++++++++++++++++++-- 4 files changed, 52 insertions(+), 19 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index e72e1ec130..0805e2d508 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -6238,12 +6238,12 @@ transformation is @dfn{input rewriting}, whereby the dependency tree of a package is rewritten by replacing specific inputs by others: @deffn {Scheme Procedure} package-input-rewriting @var{replacements} @ - [@var{rewrite-name}] + [@var{rewrite-name}] [#:deep? #t] Return a procedure that, when passed a package, replaces its direct and -indirect dependencies (but not its implicit inputs) according to -@var{replacements}. @var{replacements} is a list of package pairs; the -first element of each pair is the package to replace, and the second one -is the replacement. +indirect dependencies, including implicit inputs when @var{deep?} is +true, according to @var{replacements}. @var{replacements} is a list of +package pairs; the first element of each pair is the package to replace, +and the second one is the replacement. Optionally, @var{rewrite-name} is a one-argument procedure that takes the name of a package and returns its new name after rewrite. diff --git a/gnu/packages/guile.scm b/gnu/packages/guile.scm index c59daeebe2..280053bf06 100644 --- a/gnu/packages/guile.scm +++ b/gnu/packages/guile.scm @@ -420,11 +420,13 @@ GNU@tie{}Guile. Use the @code{(ice-9 readline)} module and call its ;; A procedure that rewrites the dependency tree of the given package to use ;; GUILE-2.0 instead of GUILE-3.0. (package-input-rewriting `((,guile-3.0 . ,guile-2.0)) - (guile-variant-package-name "guile2.0"))) + (guile-variant-package-name "guile2.0") + #:deep? #f)) (define package-for-guile-2.2 (package-input-rewriting `((,guile-3.0 . ,guile-2.2)) - (guile-variant-package-name "guile2.2"))) + (guile-variant-package-name "guile2.2") + #:deep? #f)) (define-syntax define-deprecated-guile3.0-package (lambda (s) diff --git a/guix/packages.scm b/guix/packages.scm index 0d0d7492b6..4f2bb432be 100644 --- a/guix/packages.scm +++ b/guix/packages.scm @@ -1044,22 +1044,37 @@ applied to implicit inputs as well." replace) (define* (package-input-rewriting replacements - #:optional (rewrite-name identity)) + #:optional (rewrite-name identity) + #:key (deep? #t)) "Return a procedure that, when passed a package, replaces its direct and -indirect dependencies (but not its implicit inputs) according to REPLACEMENTS. -REPLACEMENTS is a list of package pairs; the first element of each pair is the -package to replace, and the second one is the replacement. +indirect dependencies, including implicit inputs when DEEP? is true, according +to REPLACEMENTS. REPLACEMENTS is a list of package pairs; the first element +of each pair is the package to replace, and the second one is the replacement. Optionally, REWRITE-NAME is a one-argument procedure that takes the name of a package and returns its new name after rewrite." + (define replacement-property + ;; Property to tag right-hand sides in REPLACEMENTS. + (gensym " package-replacement")) + (define (rewrite p) - (match (assq-ref replacements p) - (#f (package - (inherit p) - (name (rewrite-name (package-name p))))) - (new new))) + (if (assq-ref (package-properties p) replacement-property) + p + (match (assq-ref replacements p) + (#f (package/inherit p + (name (rewrite-name (package-name p))))) + (new (if deep? + (package/inherit new + (properties `((,replacement-property . #t) + ,@(package-properties new)))) + new))))) - (package-mapping rewrite (cut assq <> replacements))) + (define (cut? p) + (or (assq-ref (package-properties p) replacement-property) + (assq-ref replacements p))) + + (package-mapping rewrite cut? + #:deep? deep?)) (define* (package-input-rewriting/spec replacements #:key (deep? #t)) "Return a procedure that, given a package, applies the given REPLACEMENTS to diff --git a/tests/packages.scm b/tests/packages.scm index e31dea6f72..af8941c2e2 100644 --- a/tests/packages.scm +++ b/tests/packages.scm @@ -1239,7 +1239,8 @@ ("baz" ,dep))))) (rewrite (package-input-rewriting `((,coreutils . ,sed) (,grep . ,findutils)) - (cut string-append "r-" <>))) + (cut string-append "r-" <>) + #:deep? #f)) (p1 (rewrite p0)) (p2 (rewrite p0))) (and (not (eq? p1 p0)) @@ -1253,7 +1254,22 @@ (eq? dep3 (rewrite dep)) ;memoization (match (package-native-inputs dep3) ((("x" dep)) - (eq? dep findutils))))))))) + (eq? dep findutils)))))) + + ;; Make sure implicit inputs were left unchanged. + (equal? (drop (bag-direct-inputs (package->bag p1)) 3) + (drop (bag-direct-inputs (package->bag p0)) 3))))) + +(test-eq "package-input-rewriting, deep" + (derivation-file-name (package-derivation %store sed)) + (let* ((p0 (dummy-package "chbouib" + (build-system python-build-system) + (arguments `(#:python ,python)))) + (rewrite (package-input-rewriting `((,python . ,sed)))) + (p1 (rewrite p0))) + (match (bag-direct-inputs (package->bag p1)) + ((("python" python) _ ...) + (derivation-file-name (package-derivation %store python)))))) (test-assert "package-input-rewriting/spec" (let* ((dep (dummy-package "chbouib" -- cgit 1.4.1 From fa18db48ae9e45874967cc90e7e8430cac996d23 Mon Sep 17 00:00:00 2001 From: zimoun Date: Fri, 25 Sep 2020 22:00:54 +0200 Subject: doc: Update the master menu. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * doc/guix.texi: Update the master menu. Signed-off-by: Ludovic Courtès --- doc/guix.texi | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 0805e2d508..dbf0c517a7 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -179,6 +179,7 @@ Installation * Setting Up the Daemon:: Preparing the build daemon's environment. * Invoking guix-daemon:: Running the build daemon. * Application Setup:: Application-specific setup. +* Upgrading Guix:: Upgrading Guix and its build daemon. Setting Up the Daemon @@ -198,8 +199,6 @@ System Installation * Installing Guix in a VM:: Guix System playground. * Building the Installation Image:: How this comes to be. -Getting Started - Manual Installation * Keyboard Layout and Networking and Partitioning:: Initial setup. @@ -233,6 +232,7 @@ Development * Invoking guix environment:: Setting up development environments. * Invoking guix pack:: Creating software bundles. * The GCC toolchain:: Working with languages supported by GCC. +* Invoking guix git authenticate:: Authenticating Git repositories. Programming Interface @@ -301,6 +301,7 @@ Services * Scheduled Job Execution:: The mcron service. * Log Rotation:: The rottlog service. * Networking Services:: Network setup, SSH daemon, etc. +* Unattended Upgrades:: Automated system upgrades. * X Window:: Graphical display. * Printing Services:: Local and remote printer support. * Desktop Services:: D-Bus and desktop services. @@ -311,6 +312,7 @@ Services * Telephony Services:: Telephony services. * Monitoring Services:: Monitoring services. * Kerberos Services:: Kerberos services. +* LDAP Services:: LDAP services. * Web Services:: Web servers. * Certificate Services:: TLS certificates via Let's Encrypt. * DNS Services:: DNS daemons. @@ -325,7 +327,7 @@ Services * PAM Mount Service:: Service to mount volumes when logging in. * Guix Services:: Services relating specifically to Guix. * Linux Services:: Services tied to the Linux kernel. -* Hurd Services:: Services specific to a Hurd System. +* Hurd Services:: Services specific for a Hurd System. * Miscellaneous Services:: Other services. Defining Services @@ -335,6 +337,11 @@ Defining Services * Service Reference:: API reference. * Shepherd Services:: A particular type of service. +Bootstrapping + +* Reduced Binary Seed Bootstrap:: A Bootstrap worthy of GNU. +* Preparing to Use the Bootstrap Binaries:: Building that what matters most. + @end detailmenu @end menu -- cgit 1.4.1 From 12871cc8db701a89f7f64303e1f16a8537a12561 Mon Sep 17 00:00:00 2001 From: zimoun Date: Fri, 25 Sep 2020 22:00:55 +0200 Subject: doc: Promote "Channels" as chapter and reorder. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The sectioning becomes: 1. Specifying Additional Channels (was 3.) 2. Using a Custom Guix Channel (was 2.) 3. Replicating Guix (was 9.) 4. Channel Authentication (was 1.) 5. Primary URL (was 7.) 6. Creating a Channel (reworded) 7. Package Modules in a Sub-directory (was 5.) 8. Declaring Channel Dependencies (was 4.) 9. Specifying Channel Authorizations (was 6.) 10. Writing Channel News (was 8.) * doc/guix.texi (Channels): Move section to chapter. Reorder the chapter. Minor tweaks to keep uniformity. Update the master menu. Signed-off-by: Ludovic Courtès --- doc/guix.texi | 1587 +++++++++++++++++++++++++++++---------------------------- 1 file changed, 817 insertions(+), 770 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index dbf0c517a7..cc8214c257 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -147,6 +147,7 @@ Project}. * System Installation:: Installing the whole operating system. * Getting Started:: Your first steps. * Package Management:: Package installation, upgrade, etc. +* Channels:: Customizing the package collection. * Development:: Guix-aided software development. * Programming Interface:: Using Guix in Scheme. * Utilities:: Package management commands. @@ -212,7 +213,6 @@ Package Management * Packages with Multiple Outputs:: Single source package, multiple outputs. * Invoking guix gc:: Running the garbage collector. * Invoking guix pull:: Fetching the latest Guix and distribution. -* Channels:: Customizing the package collection. * Invoking guix time-machine:: Running an older revision of Guix. * Inferiors:: Interacting with another revision of Guix. * Invoking guix describe:: Display information about your Guix revision. @@ -227,6 +227,19 @@ Substitutes * Substitution Failure:: What happens when substitution fails. * On Trusting Binaries:: How can you trust that binary blob? +Channels + +* Specifying Additional Channels:: Extending the package collection. +* Using a Custom Guix Channel:: Using a customized Guix. +* Replicating Guix:: Running the @emph{exact same} Guix. +* Channel Authentication:: How Guix verifies what it fetches. +* Primary URL:: Distinguishing mirror to original. +* Creating a Channel:: How to write your custom channel. +* Package Modules in a Sub-directory:: Specifying the channel's package modules location. +* Declaring Channel Dependencies:: How to depend on other channels. +* Specifying Channel Authorizations:: Defining channel authors authorizations. +* Writing Channel News:: Communicating information to channel's users. + Development * Invoking guix environment:: Setting up development environments. @@ -2812,7 +2825,6 @@ guix install emacs-guix * Packages with Multiple Outputs:: Single source package, multiple outputs. * Invoking guix gc:: Running the garbage collector. * Invoking guix pull:: Fetching the latest Guix and distribution. -* Channels:: Customizing the package collection. * Invoking guix time-machine:: Running an older revision of Guix. * Inferiors:: Interacting with another revision of Guix. * Invoking guix describe:: Display information about your Guix revision. @@ -4189,937 +4201,972 @@ information. In addition, @command{guix pull} supports all the common build options (@pxref{Common Build Options}). -@node Channels -@section Channels - -@cindex channels -@cindex @file{channels.scm}, configuration file -@cindex configuration file for channels -@cindex @command{guix pull}, configuration file -@cindex configuration of @command{guix pull} -Guix and its package collection are updated by running @command{guix pull} -(@pxref{Invoking guix pull}). By default @command{guix pull} downloads and -deploys Guix itself from the official GNU@tie{}Guix repository. This can be -customized by defining @dfn{channels} in the -@file{~/.config/guix/channels.scm} file. A channel specifies a URL and branch -of a Git repository to be deployed, and @command{guix pull} can be instructed -to pull from one or more channels. In other words, channels can be used -to @emph{customize} and to @emph{extend} Guix, as we will see below. -Before that, some security considerations. +@node Invoking guix time-machine +@section Invoking @command{guix time-machine} -@subsection Channel Authentication +@cindex @command{guix time-machine} +@cindex pinning, channels +@cindex replicating Guix +@cindex reproducibility, of Guix -@anchor{channel-authentication} -@cindex authentication, of channel code -The @command{guix pull} and @command{guix time-machine} commands -@dfn{authenticate} the code retrieved from channels: they make sure each -commit that is fetched is signed by an authorized developer. The goal -is to protect from unauthorized modifications to the channel that would -lead users to run malicious code. +The @command{guix time-machine} command provides access to other +revisions of Guix, for example to install older versions of packages, +or to reproduce a computation in an identical environment. The revision +of Guix to be used is defined by a commit or by a channel +description file created by @command{guix describe} +(@pxref{Invoking guix describe}). -As a user, you must provide a @dfn{channel introduction} in your -channels file so that Guix knows how to authenticate its first commit. -A channel specification, including its introduction, looks something -along these lines: +The general syntax is: -@lisp -(channel - (name 'my-channel) - (url "https://example.org/my-channel.git") - (introduction - (make-channel-introduction - "6f0d8cc0d88abb59c324b2990bfee2876016bb86" - (openpgp-fingerprint - "CABB A931 C0FF EEC6 900D 0CFB 090B 1199 3D9A EBB5")))) -@end lisp +@example +guix time-machine @var{options}@dots{} -- @var{command} @var {arg}@dots{} +@end example -The specification above shows the name and URL of the channel. The call -to @code{make-channel-introduction} above specifies that authentication -of this channel starts at commit @code{6f0d8cc@dots{}}, which is signed -by the OpenPGP key with fingerprint @code{CABB A931@dots{}}. +where @var{command} and @var{arg}@dots{} are passed unmodified to the +@command{guix} command of the specified revision. The @var{options} that define +this revision are the same as for @command{guix pull} (@pxref{Invoking guix pull}): -For the main channel, called @code{guix}, you automatically get that -information from your Guix installation. For other channels, include -the channel introduction provided by the channel authors in your -@file{channels.scm} file. Make sure you retrieve the channel -introduction from a trusted source since that is the root of your trust. +@table @code +@item --url=@var{url} +@itemx --commit=@var{commit} +@itemx --branch=@var{branch} +Use the @code{guix} channel from the specified @var{url}, at the +given @var{commit} (a valid Git commit ID represented as a hexadecimal +string), or @var{branch}. -If you're curious about the authentication mechanics, read on! +@item --channels=@var{file} +@itemx -C @var{file} +Read the list of channels from @var{file}. @var{file} must contain +Scheme code that evaluates to a list of channel objects. +@xref{Channels} for more information. +@end table -@subsection Using a Custom Guix Channel +As for @command{guix pull}, the absence of any options means that the +the latest commit on the master branch will be used. The command -The channel called @code{guix} specifies where Guix itself---its command-line -tools as well as its package collection---should be downloaded. For instance, -suppose you want to update from your own copy of the Guix repository at -@code{example.org}, and specifically the @code{super-hacks} branch, you can -write in @code{~/.config/guix/channels.scm} this specification: +@example +guix time-machine -- build hello +@end example -@lisp -;; Tell 'guix pull' to use my own repo. -(list (channel - (name 'guix) - (url "https://example.org/my-guix.git") - (branch "super-hacks"))) -@end lisp +will thus build the package @code{hello} as defined in the master branch, +which is in general a newer revision of Guix than you have installed. +Time travel works in both directions! -@noindent -From there on, @command{guix pull} will fetch code from the @code{super-hacks} -branch of the repository at @code{example.org}. +Note that @command{guix time-machine} can trigger builds of channels and +their dependencies, and these are controlled by the standard build +options (@pxref{Common Build Options}). -@subsection Specifying Additional Channels +@node Inferiors +@section Inferiors -@cindex extending the package collection (channels) -@cindex personal packages (channels) -@cindex channels, for personal packages -You can also specify @emph{additional channels} to pull from. Let's say you -have a bunch of custom package variants or personal packages that you think -would make little sense to contribute to the Guix project, but would like to -have these packages transparently available to you at the command line. You -would first write modules containing those package definitions (@pxref{Package -Modules}), maintain them in a Git repository, and then you and anyone else can -use it as an additional channel to get packages from. Neat, no? +@c TODO: Remove this once we're more confident about API stability. +@quotation Note +The functionality described here is a ``technology preview'' as of version +@value{VERSION}. As such, the interface is subject to change. +@end quotation -@c What follows stems from discussions at -@c as well as -@c earlier discussions on guix-devel@gnu.org. -@quotation Warning -Before you, dear user, shout---``woow this is @emph{soooo coool}!''---and -publish your personal channel to the world, we would like to share a few words -of caution: +@cindex inferiors +@cindex composition of Guix revisions +Sometimes you might need to mix packages from the revision of Guix you're +currently running with packages available in a different revision of Guix. +Guix @dfn{inferiors} allow you to achieve that by composing different Guix +revisions in arbitrary ways. -@itemize -@item -Before publishing a channel, please consider contributing your package -definitions to Guix proper (@pxref{Contributing}). Guix as a project is open -to free software of all sorts, and packages in Guix proper are readily -available to all Guix users and benefit from the project's quality assurance -process. +@cindex inferior packages +Technically, an ``inferior'' is essentially a separate Guix process connected +to your main Guix process through a REPL (@pxref{Invoking guix repl}). The +@code{(guix inferior)} module allows you to create inferiors and to +communicate with them. It also provides a high-level interface to browse and +manipulate the packages that an inferior provides---@dfn{inferior packages}. -@item -When you maintain package definitions outside Guix, we, Guix developers, -consider that @emph{the compatibility burden is on you}. Remember that -package modules and package definitions are just Scheme code that uses various -programming interfaces (APIs). We want to remain free to change these APIs to -keep improving Guix, possibly in ways that break your channel. We never -change APIs gratuitously, but we will @emph{not} commit to freezing APIs -either. +When combined with channels (@pxref{Channels}), inferiors provide a simple way +to interact with a separate revision of Guix. For example, let's assume you +want to install in your profile the current @code{guile} package, along with +the @code{guile-json} as it existed in an older revision of Guix---perhaps +because the newer @code{guile-json} has an incompatible API and you want to +run your code against the old API@. To do that, you could write a manifest for +use by @code{guix package --manifest} (@pxref{Invoking guix package}); in that +manifest, you would create an inferior for that old Guix revision you care +about, and you would look up the @code{guile-json} package in the inferior: -@item -Corollary: if you're using an external channel and that channel breaks, please -@emph{report the issue to the channel authors}, not to the Guix project. -@end itemize +@lisp +(use-modules (guix inferior) (guix channels) + (srfi srfi-1)) ;for 'first' -You've been warned! Having said this, we believe external channels are a -practical way to exert your freedom to augment Guix' package collection and to -share your improvements, which are basic tenets of -@uref{https://www.gnu.org/philosophy/free-sw.html, free software}. Please -email us at @email{guix-devel@@gnu.org} if you'd like to discuss this. -@end quotation +(define channels + ;; This is the old revision from which we want to + ;; extract guile-json. + (list (channel + (name 'guix) + (url "https://git.savannah.gnu.org/git/guix.git") + (commit + "65956ad3526ba09e1f7a40722c96c6ef7c0936fe")))) -To use a channel, write @code{~/.config/guix/channels.scm} to instruct -@command{guix pull} to pull from it @emph{in addition} to the default Guix -channel(s): +(define inferior + ;; An inferior representing the above revision. + (inferior-for-channels channels)) -@vindex %default-channels -@lisp -;; Add my personal packages to those Guix provides. -(cons (channel - (name 'my-personal-packages) - (url "https://example.org/personal-packages.git")) - %default-channels) +;; Now create a manifest with the current "guile" package +;; and the old "guile-json" package. +(packages->manifest + (list (first (lookup-inferior-packages inferior "guile-json")) + (specification->package "guile"))) @end lisp -@noindent -Note that the snippet above is (as always!)@: Scheme code; we use @code{cons} to -add a channel the list of channels that the variable @code{%default-channels} -is bound to (@pxref{Pairs, @code{cons} and lists,, guile, GNU Guile Reference -Manual}). With this file in place, @command{guix pull} builds not only Guix -but also the package modules from your own repository. The result in -@file{~/.config/guix/current} is the union of Guix with your own package -modules: +On its first run, @command{guix package --manifest} might have to build the +channel you specified before it can create the inferior; subsequent runs will +be much faster because the Guix revision will be cached. -@example -$ guix pull --list-generations -@dots{} -Generation 19 Aug 27 2018 16:20:48 - guix d894ab8 - repository URL: https://git.savannah.gnu.org/git/guix.git - branch: master - commit: d894ab8e9bfabcefa6c49d9ba2e834dd5a73a300 - my-personal-packages dd3df5e - repository URL: https://example.org/personal-packages.git - branch: master - commit: dd3df5e2c8818760a8fc0bd699e55d3b69fef2bb - 11 new packages: my-gimp, my-emacs-with-cool-features, @dots{} - 4 packages upgraded: emacs-racket-mode@@0.0.2-2.1b78827, @dots{} -@end example +The @code{(guix inferior)} module provides the following procedures to open an +inferior: -@noindent -The output of @command{guix pull} above shows that Generation@tie{}19 includes -both Guix and packages from the @code{my-personal-packages} channel. Among -the new and upgraded packages that are listed, some like @code{my-gimp} and -@code{my-emacs-with-cool-features} might come from -@code{my-personal-packages}, while others come from the Guix default channel. +@deffn {Scheme Procedure} inferior-for-channels @var{channels} @ + [#:cache-directory] [#:ttl] +Return an inferior for @var{channels}, a list of channels. Use the cache at +@var{cache-directory}, where entries can be reclaimed after @var{ttl} seconds. +This procedure opens a new connection to the build daemon. -To create a channel, create a Git repository containing your own package -modules and make it available. The repository can contain anything, but a -useful channel will contain Guile modules that export packages. Once you -start using a channel, Guix will behave as if the root directory of that -channel's Git repository has been added to the Guile load path (@pxref{Load -Paths,,, guile, GNU Guile Reference Manual}). For example, if your channel -contains a file at @file{my-packages/my-tools.scm} that defines a Guile -module, then the module will be available under the name @code{(my-packages -my-tools)}, and you will be able to use it like any other module -(@pxref{Modules,,, guile, GNU Guile Reference Manual}). +As a side effect, this procedure may build or substitute binaries for +@var{channels}, which can take time. +@end deffn -@cindex dependencies, channels -@cindex meta-data, channels -@subsection Declaring Channel Dependencies +@deffn {Scheme Procedure} open-inferior @var{directory} @ + [#:command "bin/guix"] +Open the inferior Guix in @var{directory}, running +@code{@var{directory}/@var{command} repl} or equivalent. Return @code{#f} if +the inferior could not be launched. +@end deffn -Channel authors may decide to augment a package collection provided by other -channels. They can declare their channel to be dependent on other channels in -a meta-data file @file{.guix-channel}, which is to be placed in the root of -the channel repository. +@cindex inferior packages +The procedures listed below allow you to obtain and manipulate inferior +packages. -The meta-data file should contain a simple S-expression like this: +@deffn {Scheme Procedure} inferior-packages @var{inferior} +Return the list of packages known to @var{inferior}. +@end deffn -@lisp -(channel - (version 0) - (dependencies - (channel - (name some-collection) - (url "https://example.org/first-collection.git") - - ;; The 'introduction' bit below is optional: you would - ;; provide it for dependencies that can be authenticated. - (introduction - (channel-introduction - (version 0) - (commit "a8883b58dc82e167c96506cf05095f37c2c2c6cd") - (signer "CABB A931 C0FF EEC6 900D 0CFB 090B 1199 3D9A EBB5")))) - (channel - (name some-other-collection) - (url "https://example.org/second-collection.git") - (branch "testing")))) -@end lisp +@deffn {Scheme Procedure} lookup-inferior-packages @var{inferior} @var{name} @ + [@var{version}] +Return the sorted list of inferior packages matching @var{name} in +@var{inferior}, with highest version numbers first. If @var{version} is true, +return only packages with a version number prefixed by @var{version}. +@end deffn -In the above example this channel is declared to depend on two other channels, -which will both be fetched automatically. The modules provided by the channel -will be compiled in an environment where the modules of all these declared -channels are available. +@deffn {Scheme Procedure} inferior-package? @var{obj} +Return true if @var{obj} is an inferior package. +@end deffn -For the sake of reliability and maintainability, you should avoid dependencies -on channels that you don't control, and you should aim to keep the number of -dependencies to a minimum. +@deffn {Scheme Procedure} inferior-package-name @var{package} +@deffnx {Scheme Procedure} inferior-package-version @var{package} +@deffnx {Scheme Procedure} inferior-package-synopsis @var{package} +@deffnx {Scheme Procedure} inferior-package-description @var{package} +@deffnx {Scheme Procedure} inferior-package-home-page @var{package} +@deffnx {Scheme Procedure} inferior-package-location @var{package} +@deffnx {Scheme Procedure} inferior-package-inputs @var{package} +@deffnx {Scheme Procedure} inferior-package-native-inputs @var{package} +@deffnx {Scheme Procedure} inferior-package-propagated-inputs @var{package} +@deffnx {Scheme Procedure} inferior-package-transitive-propagated-inputs @var{package} +@deffnx {Scheme Procedure} inferior-package-native-search-paths @var{package} +@deffnx {Scheme Procedure} inferior-package-transitive-native-search-paths @var{package} +@deffnx {Scheme Procedure} inferior-package-search-paths @var{package} +These procedures are the counterpart of package record accessors +(@pxref{package Reference}). Most of them work by querying the inferior +@var{package} comes from, so the inferior must still be live when you call +these procedures. +@end deffn -@cindex subdirectory, channels -@subsection Package Modules in a Sub-directory +Inferior packages can be used transparently like any other package or +file-like object in G-expressions (@pxref{G-Expressions}). They are also +transparently handled by the @code{packages->manifest} procedure, which is +commonly use in manifests (@pxref{Invoking guix package, the +@option{--manifest} option of @command{guix package}}). Thus you can insert +an inferior package pretty much anywhere you would insert a regular package: +in manifests, in the @code{packages} field of your @code{operating-system} +declaration, and so on. -As a channel author, you may want to keep your channel modules in a -sub-directory. If your modules are in the sub-directory @file{guix}, you must -add a meta-data file @file{.guix-channel} that contains: +@node Invoking guix describe +@section Invoking @command{guix describe} -@lisp -(channel - (version 0) - (directory "guix")) -@end lisp +@cindex reproducibility +@cindex replicating Guix +Often you may want to answer questions like: ``Which revision of Guix am I +using?'' or ``Which channels am I using?'' This is useful information in many +situations: if you want to @emph{replicate} an environment on a different +machine or user account, if you want to report a bug or to determine what +change in the channels you are using caused it, or if you want to record your +system state for reproducibility purposes. The @command{guix describe} +command answers these questions. -@cindex channel authorizations -@subsection Specifying Channel Authorizations +When run from a @command{guix pull}ed @command{guix}, @command{guix describe} +displays the channel(s) that it was built from, including their repository URL +and commit IDs (@pxref{Channels}): -@anchor{channel-authorizations} -As we saw above, Guix ensures the source code it pulls from channels -comes from authorized developers. As a channel author, you need to -specify the list of authorized developers in the -@file{.guix-authorizations} file in the channel's Git repository. The -authentication rule is simple: each commit must be signed by a key -listed in the @file{.guix-authorizations} file of its parent -commit(s)@footnote{Git commits form a @dfn{directed acyclic graph} -(DAG). Each commit can have zero or more parents; ``regular'' commits -have one parent and merge commits have two parent commits. Read -@uref{https://eagain.net/articles/git-for-computer-scientists/, @i{Git -for Computer Scientists}} for a great overview.} The -@file{.guix-authorizations} file looks like this: +@example +$ guix describe +Generation 10 Sep 03 2018 17:32:44 (current) + guix e0fa68c + repository URL: https://git.savannah.gnu.org/git/guix.git + branch: master + commit: e0fa68c7718fffd33d81af415279d6ddb518f727 +@end example -@lisp -;; Example '.guix-authorizations' file. +If you're familiar with the Git version control system, this is similar in +spirit to @command{git describe}; the output is also similar to that of +@command{guix pull --list-generations}, but limited to the current generation +(@pxref{Invoking guix pull, the @option{--list-generations} option}). Because +the Git commit ID shown above unambiguously refers to a snapshot of Guix, this +information is all it takes to describe the revision of Guix you're using, and +also to replicate it. -(authorizations - (version 0) ;current file format version +To make it easier to replicate Guix, @command{guix describe} can also be asked +to return a list of channels instead of the human-readable description above: - (("AD17 A21E F8AE D8F1 CC02 DBD9 F8AE D8F1 765C 61E3" - (name "alice")) - ("2A39 3FFF 68F4 EF7A 3D29 12AF 68F4 EF7A 22FB B2D5" - (name "bob")) - ("CABB A931 C0FF EEC6 900D 0CFB 090B 1199 3D9A EBB5" - (name "charlie")))) -@end lisp +@example +$ guix describe -f channels +(list (channel + (name 'guix) + (url "https://git.savannah.gnu.org/git/guix.git") + (commit + "e0fa68c7718fffd33d81af415279d6ddb518f727") + (introduction + (make-channel-introduction + "9edb3f66fd807b096b48283debdcddccfea34bad" + (openpgp-fingerprint + "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA"))))) +@end example -Each fingerprint is followed by optional key/value pairs, as in the -example above. Currently these key/value pairs are ignored. +@noindent +You can save this to a file and feed it to @command{guix pull -C} on some +other machine or at a later point in time, which will instantiate @emph{this +exact Guix revision} (@pxref{Invoking guix pull, the @option{-C} option}). +From there on, since you're able to deploy the same revision of Guix, you can +just as well @emph{replicate a complete software environment}. We humbly +think that this is @emph{awesome}, and we hope you'll like it too! -This authentication rule creates a chicken-and-egg issue: how do we -authenticate the first commit? Related to that: how do we deal with -channels whose repository history contains unsigned commits and lack -@file{.guix-authorizations}? And how do we fork existing channels? +The details of the options supported by @command{guix describe} are as +follows: -@cindex channel introduction -Channel introductions answer these questions by describing the first -commit of a channel that should be authenticated. The first time a -channel is fetched with @command{guix pull} or @command{guix -time-machine}, the command looks up the introductory commit and verifies -that it is signed by the specified OpenPGP key. From then on, it -authenticates commits according to the rule above. +@table @code +@item --format=@var{format} +@itemx -f @var{format} +Produce output in the specified @var{format}, one of: -Additionally, your channel must provide all the OpenPGP keys that were -ever mentioned in @file{.guix-authorizations}, stored as @file{.key} -files, which can be either binary or ``ASCII-armored''. By default, -those @file{.key} files are searched for in the branch named -@code{keyring} but you can specify a different branch name in -@code{.guix-channel} like so: +@table @code +@item human +produce human-readable output; +@item channels +produce a list of channel specifications that can be passed to @command{guix +pull -C} or installed as @file{~/.config/guix/channels.scm} (@pxref{Invoking +guix pull}); +@item channels-sans-intro +like @code{channels}, but omit the @code{introduction} field; use it to +produce a channel specification suitable for Guix version 1.1.0 or +earlier---the @code{introduction} field has to do with channel +authentication (@pxref{Channels, Channel Authentication}) and is not +supported by these older versions; +@item json +@cindex JSON +produce a list of channel specifications in JSON format; +@item recutils +produce a list of channel specifications in Recutils format. +@end table -@lisp -(channel - (version 0) - (keyring-reference "my-keyring-branch")) -@end lisp +@item --list-formats +Display available formats for @option{--format} option. -To summarize, as the author of a channel, there are three things you have -to do to allow users to authenticate your code: +@item --profile=@var{profile} +@itemx -p @var{profile} +Display information about @var{profile}. +@end table -@enumerate -@item -Export the OpenPGP keys of past and present committers with @command{gpg ---export} and store them in @file{.key} files, by default in a branch -named @code{keyring} (we recommend making it an @dfn{orphan branch}). +@node Invoking guix archive +@section Invoking @command{guix archive} -@item -Introduce an initial @file{.guix-authorizations} in the channel's -repository. Do that in a signed commit (@pxref{Commit Access}, for -information on how to sign Git commits.) +@cindex @command{guix archive} +@cindex archive +The @command{guix archive} command allows users to @dfn{export} files +from the store into a single archive, and to later @dfn{import} them on +a machine that runs Guix. +In particular, it allows store files to be transferred from one machine +to the store on another machine. -@item -Advertise the channel introduction, for instance on your channel's web -page. The channel introduction, as we saw above, is the commit/key -pair---i.e., the commit that introduced @file{.guix-authorizations}, and -the fingerprint of the OpenPGP used to sign it. -@end enumerate +@quotation Note +If you're looking for a way to produce archives in a format suitable for +tools other than Guix, @pxref{Invoking guix pack}. +@end quotation -Before pushing to your public Git repository, you can run @command{guix -git-authenticate} to verify that you did sign all the commits you are -about to push with an authorized key: +@cindex exporting store items +To export store files as an archive to standard output, run: @example -guix git authenticate @var{commit} @var{signer} +guix archive --export @var{options} @var{specifications}... @end example -@noindent -where @var{commit} and @var{signer} are your channel introduction. -@xref{Invoking guix git authenticate}, for details. +@var{specifications} may be either store file names or package +specifications, as for @command{guix package} (@pxref{Invoking guix +package}). For instance, the following command creates an archive +containing the @code{gui} output of the @code{git} package and the main +output of @code{emacs}: -Publishing a signed channel requires discipline: any mistake, such as an -unsigned commit or a commit signed by an unauthorized key, will prevent -users from pulling from your channel---well, that's the whole point of -authentication! Pay attention to merges in particular: merge commits -are considered authentic if and only if they are signed by a key present -in the @file{.guix-authorizations} file of @emph{both} branches. +@example +guix archive --export git:gui /gnu/store/...-emacs-24.3 > great.nar +@end example -@cindex primary URL, channels -@subsection Primary URL +If the specified packages are not built yet, @command{guix archive} +automatically builds them. The build process may be controlled with the +common build options (@pxref{Common Build Options}). -Channel authors can indicate the primary URL of their channel's Git -repository in the @file{.guix-channel} file, like so: - -@lisp -(channel - (version 0) - (url "https://example.org/guix.git")) -@end lisp - -This allows @command{guix pull} to determine whether it is pulling code -from a mirror of the channel; when that is the case, it warns the user -that the mirror might be stale and displays the primary URL. That way, -users cannot be tricked into fetching code from a stale mirror that does -not receive security updates. +To transfer the @code{emacs} package to a machine connected over SSH, +one would run: -This feature only makes sense for authenticated repositories, such as -the official @code{guix} channel, for which @command{guix pull} ensures -the code it fetches is authentic. +@example +guix archive --export -r emacs | ssh the-machine guix archive --import +@end example -@cindex news, for channels -@subsection Writing Channel News +@noindent +Similarly, a complete user profile may be transferred from one machine +to another like this: -Channel authors may occasionally want to communicate to their users -information about important changes in the channel. You'd send them all -an email, but that's not convenient. +@example +guix archive --export -r $(readlink -f ~/.guix-profile) | \ + ssh the-machine guix archive --import +@end example -Instead, channels can provide a @dfn{news file}; when the channel users -run @command{guix pull}, that news file is automatically read and -@command{guix pull --news} can display the announcements that correspond -to the new commits that have been pulled, if any. +@noindent +However, note that, in both examples, all of @code{emacs} and the +profile as well as all of their dependencies are transferred (due to +@option{-r}), regardless of what is already available in the store on +the target machine. The @option{--missing} option can help figure out +which items are missing from the target store. The @command{guix copy} +command simplifies and optimizes this whole process, so this is probably +what you should use in this case (@pxref{Invoking guix copy}). -To do that, channel authors must first declare the name of the news file -in their @file{.guix-channel} file: +@cindex nar, archive format +@cindex normalized archive (nar) +@cindex nar bundle, archive format +Each store item is written in the @dfn{normalized archive} or @dfn{nar} +format (described below), and the output of @command{guix archive +--export} (and input of @command{guix archive --import}) is a @dfn{nar +bundle}. -@lisp -(channel - (version 0) - (news-file "etc/news.txt")) -@end lisp +The nar format is +comparable in spirit to `tar', but with differences +that make it more appropriate for our purposes. First, rather than +recording all Unix metadata for each file, the nar format only mentions +the file type (regular, directory, or symbolic link); Unix permissions +and owner/group are dismissed. Second, the order in which directory +entries are stored always follows the order of file names according to +the C locale collation order. This makes archive production fully +deterministic. -The news file itself, @file{etc/news.txt} in this example, must look -something like this: +That nar bundle format is essentially the concatenation of zero or more +nars along with metadata for each store item it contains: its file name, +references, corresponding derivation, and a digital signature. -@lisp -(channel-news - (version 0) - (entry (tag "the-bug-fix") - (title (en "Fixed terrible bug") - (fr "Oh la la")) - (body (en "@@emph@{Good news@}! It's fixed!") - (eo "Certe ĝi pli bone funkcias nun!"))) - (entry (commit "bdcabe815cd28144a2d2b4bc3c5057b051fa9906") - (title (en "Added a great package") - (ca "Què vol dir guix?")) - (body (en "Don't miss the @@code@{hello@} package!")))) -@end lisp +When exporting, the daemon digitally signs the contents of the archive, +and that digital signature is appended. When importing, the daemon +verifies the signature and rejects the import in case of an invalid +signature or if the signing key is not authorized. +@c FIXME: Add xref to daemon doc about signatures. -While the news file is using the Scheme syntax, avoid naming it with a -@file{.scm} extension or else it will get picked up when building the -channel and yield an error since it is not a valid module. -Alternatively, you can move the channel module to a subdirectory and -store the news file in another directory. +The main options are: -The file consists of a list of @dfn{news entries}. Each entry is -associated with a commit or tag: it describes changes made in this -commit, possibly in preceding commits as well. Users see entries only -the first time they obtain the commit the entry refers to. +@table @code +@item --export +Export the specified store files or packages (see below). Write the +resulting archive to the standard output. -The @code{title} field should be a one-line summary while @code{body} -can be arbitrarily long, and both can contain Texinfo markup -(@pxref{Overview,,, texinfo, GNU Texinfo}). Both the title and body are -a list of language tag/message tuples, which allows @command{guix pull} -to display news in the language that corresponds to the user's locale. +Dependencies are @emph{not} included in the output, unless +@option{--recursive} is passed. -If you want to translate news using a gettext-based workflow, you can -extract translatable strings with @command{xgettext} (@pxref{xgettext -Invocation,,, gettext, GNU Gettext Utilities}). For example, assuming -you write news entries in English first, the command below creates a PO -file containing the strings to translate: +@item -r +@itemx --recursive +When combined with @option{--export}, this instructs @command{guix archive} +to include dependencies of the given items in the archive. Thus, the +resulting archive is self-contained: it contains the closure of the +exported store items. -@example -xgettext -o news.po -l scheme -ken etc/news.txt -@end example +@item --import +Read an archive from the standard input, and import the files listed +therein into the store. Abort if the archive has an invalid digital +signature, or if it is signed by a public key not among the authorized +keys (see @option{--authorize} below). -To sum up, yes, you could use your channel as a blog. But beware, this -is @emph{not quite} what your users might expect. +@item --missing +Read a list of store file names from the standard input, one per line, +and write on the standard output the subset of these files missing from +the store. -@subsection Replicating Guix +@item --generate-key[=@var{parameters}] +@cindex signing, archives +Generate a new key pair for the daemon. This is a prerequisite before +archives can be exported with @option{--export}. Note that this +operation usually takes time, because it needs to gather enough entropy +to generate the key pair. -@cindex pinning, channels -@cindex replicating Guix -@cindex reproducibility, of Guix -The @command{guix pull --list-generations} output above shows precisely which -commits were used to build this instance of Guix. We can thus replicate it, -say, on another machine, by providing a channel specification in -@file{~/.config/guix/channels.scm} that is ``pinned'' to these commits: +The generated key pair is typically stored under @file{/etc/guix}, in +@file{signing-key.pub} (public key) and @file{signing-key.sec} (private +key, which must be kept secret). When @var{parameters} is omitted, +an ECDSA key using the Ed25519 curve is generated, or, for Libgcrypt +versions before 1.6.0, it is a 4096-bit RSA key. +Alternatively, @var{parameters} can specify +@code{genkey} parameters suitable for Libgcrypt (@pxref{General +public-key related Functions, @code{gcry_pk_genkey},, gcrypt, The +Libgcrypt Reference Manual}). -@lisp -;; Deploy specific commits of my channels of interest. -(list (channel - (name 'guix) - (url "https://git.savannah.gnu.org/git/guix.git") - (commit "6298c3ffd9654d3231a6f25390b056483e8f407c")) - (channel - (name 'my-personal-packages) - (url "https://example.org/personal-packages.git") - (commit "dd3df5e2c8818760a8fc0bd699e55d3b69fef2bb"))) -@end lisp +@item --authorize +@cindex authorizing, archives +Authorize imports signed by the public key passed on standard input. +The public key must be in ``s-expression advanced format''---i.e., the +same format as the @file{signing-key.pub} file. -The @command{guix describe --format=channels} command can even generate this -list of channels directly (@pxref{Invoking guix describe}). The resulting -file can be used with the -C options of @command{guix pull} -(@pxref{Invoking guix pull}) or @command{guix time-machine} -(@pxref{Invoking guix time-machine}). +The list of authorized keys is kept in the human-editable file +@file{/etc/guix/acl}. The file contains +@url{https://people.csail.mit.edu/rivest/Sexp.txt, ``advanced-format +s-expressions''} and is structured as an access-control list in the +@url{https://theworld.com/~cme/spki.txt, Simple Public-Key Infrastructure +(SPKI)}. -At this point the two machines run the @emph{exact same Guix}, with access to -the @emph{exact same packages}. The output of @command{guix build gimp} on -one machine will be exactly the same, bit for bit, as the output of the same -command on the other machine. It also means both machines have access to all -the source code of Guix and, transitively, to all the source code of every -package it defines. +@item --extract=@var{directory} +@itemx -x @var{directory} +Read a single-item archive as served by substitute servers +(@pxref{Substitutes}) and extract it to @var{directory}. This is a +low-level operation needed in only very narrow use cases; see below. -This gives you super powers, allowing you to track the provenance of binary -artifacts with very fine grain, and to reproduce software environments at -will---some sort of ``meta reproducibility'' capabilities, if you will. -@xref{Inferiors}, for another way to take advantage of these super powers. +For example, the following command extracts the substitute for Emacs +served by @code{@value{SUBSTITUTE-SERVER}} to @file{/tmp/emacs}: -@node Invoking guix time-machine -@section Invoking @command{guix time-machine} +@example +$ wget -O - \ + https://@value{SUBSTITUTE-SERVER}/nar/gzip/@dots{}-emacs-24.5 \ + | gunzip | guix archive -x /tmp/emacs +@end example -@cindex @command{guix time-machine} -@cindex pinning, channels -@cindex replicating Guix -@cindex reproducibility, of Guix +Single-item archives are different from multiple-item archives produced +by @command{guix archive --export}; they contain a single store item, +and they do @emph{not} embed a signature. Thus this operation does +@emph{no} signature verification and its output should be considered +unsafe. -The @command{guix time-machine} command provides access to other -revisions of Guix, for example to install older versions of packages, -or to reproduce a computation in an identical environment. The revision -of Guix to be used is defined by a commit or by a channel -description file created by @command{guix describe} -(@pxref{Invoking guix describe}). +The primary purpose of this operation is to facilitate inspection of +archive contents coming from possibly untrusted substitute servers +(@pxref{Invoking guix challenge}). -The general syntax is: +@item --list +@itemx -t +Read a single-item archive as served by substitute servers +(@pxref{Substitutes}) and print the list of files it contains, as in +this example: @example -guix time-machine @var{options}@dots{} -- @var{command} @var {arg}@dots{} +$ wget -O - \ + https://@value{SUBSTITUTE-SERVER}/nar/lzip/@dots{}-emacs-26.3 \ + | lzip -d | guix archive -t @end example -where @var{command} and @var{arg}@dots{} are passed unmodified to the -@command{guix} command of the specified revision. The @var{options} that define -this revision are the same as for @command{guix pull} (@pxref{Invoking guix pull}): - -@table @code -@item --url=@var{url} -@itemx --commit=@var{commit} -@itemx --branch=@var{branch} -Use the @code{guix} channel from the specified @var{url}, at the -given @var{commit} (a valid Git commit ID represented as a hexadecimal -string), or @var{branch}. - -@item --channels=@var{file} -@itemx -C @var{file} -Read the list of channels from @var{file}. @var{file} must contain -Scheme code that evaluates to a list of channel objects. -@xref{Channels} for more information. @end table -As for @command{guix pull}, the absence of any options means that the -the latest commit on the master branch will be used. The command +@c ********************************************************************* +@node Channels +@chapter Channels + +@cindex channels +@cindex @file{channels.scm}, configuration file +@cindex configuration file for channels +@cindex @command{guix pull}, configuration file +@cindex configuration of @command{guix pull} +Guix and its package collection are updated by running @command{guix pull} +(@pxref{Invoking guix pull}). By default @command{guix pull} downloads and +deploys Guix itself from the official GNU@tie{}Guix repository. This can be +customized by defining @dfn{channels} in the +@file{~/.config/guix/channels.scm} file. A channel specifies a URL and branch +of a Git repository to be deployed, and @command{guix pull} can be instructed +to pull from one or more channels. In other words, channels can be used +to @emph{customize} and to @emph{extend} Guix, as we will see below. +Guix is able to take into account security concerns and deal with authenticated +updates. + +@menu +* Specifying Additional Channels:: Extending the package collection. +* Using a Custom Guix Channel:: Using a customized Guix. +* Replicating Guix:: Running the @emph{exact same} Guix. +* Channel Authentication:: How Guix verifies what it fetches. +* Primary URL:: Distinguishing mirror to original. +* Creating a Channel:: How to write your custom channel. +* Package Modules in a Sub-directory:: Specifying the channel's package modules location. +* Declaring Channel Dependencies:: How to depend on other channels. +* Specifying Channel Authorizations:: Defining channel authors authorizations. +* Writing Channel News:: Communicating information to channel's users. +@end menu + +@node Specifying Additional Channels +@section Specifying Additional Channels + +@cindex extending the package collection (channels) +@cindex variant packages (channels) +You can specify @emph{additional channels} to pull from. To use a channel, write +@code{~/.config/guix/channels.scm} to instruct @command{guix pull} to pull from it +@emph{in addition} to the default Guix channel(s): + +@vindex %default-channels +@lisp +;; Add variant packages to those Guix provides. +(cons (channel + (name 'variant-packages) + (url "https://example.org/variant-packages.git")) + %default-channels) +@end lisp + +@noindent +Note that the snippet above is (as always!)@: Scheme code; we use @code{cons} to +add a channel the list of channels that the variable @code{%default-channels} +is bound to (@pxref{Pairs, @code{cons} and lists,, guile, GNU Guile Reference +Manual}). With this file in place, @command{guix pull} builds not only Guix +but also the package modules from your own repository. The result in +@file{~/.config/guix/current} is the union of Guix with your own package +modules: @example -guix time-machine -- build hello +$ guix pull --list-generations +@dots{} +Generation 19 Aug 27 2018 16:20:48 + guix d894ab8 + repository URL: https://git.savannah.gnu.org/git/guix.git + branch: master + commit: d894ab8e9bfabcefa6c49d9ba2e834dd5a73a300 + variant-packages dd3df5e + repository URL: https://example.org/variant-packages.git + branch: master + commit: dd3df5e2c8818760a8fc0bd699e55d3b69fef2bb + 11 new packages: variant-gimp, variant-emacs-with-cool-features, @dots{} + 4 packages upgraded: emacs-racket-mode@@0.0.2-2.1b78827, @dots{} @end example -will thus build the package @code{hello} as defined in the master branch, -which is in general a newer revision of Guix than you have installed. -Time travel works in both directions! +@noindent +The output of @command{guix pull} above shows that Generation@tie{}19 includes +both Guix and packages from the @code{variant-personal-packages} channel. Among +the new and upgraded packages that are listed, some like @code{variant-gimp} and +@code{variant-emacs-with-cool-features} might come from +@code{variant-packages}, while others come from the Guix default channel. -Note that @command{guix time-machine} can trigger builds of channels and -their dependencies, and these are controlled by the standard build -options (@pxref{Common Build Options}). +@node Using a Custom Guix Channel +@section Using a Custom Guix Channel -@node Inferiors -@section Inferiors +The channel called @code{guix} specifies where Guix itself---its command-line +tools as well as its package collection---should be downloaded. For instance, +suppose you want to update from another copy of the Guix repository at +@code{example.org}, and specifically the @code{super-hacks} branch, you can +write in @code{~/.config/guix/channels.scm} this specification: -@c TODO: Remove this once we're more confident about API stability. -@quotation Note -The functionality described here is a ``technology preview'' as of version -@value{VERSION}. As such, the interface is subject to change. -@end quotation +@lisp +;; Tell 'guix pull' to use another repo. +(list (channel + (name 'guix) + (url "https://example.org/another-guix.git") + (branch "super-hacks"))) +@end lisp -@cindex inferiors -@cindex composition of Guix revisions -Sometimes you might need to mix packages from the revision of Guix you're -currently running with packages available in a different revision of Guix. -Guix @dfn{inferiors} allow you to achieve that by composing different Guix -revisions in arbitrary ways. +@noindent +From there on, @command{guix pull} will fetch code from the @code{super-hacks} +branch of the repository at @code{example.org}. The authentication concern is +addressed below ((@pxref{Channel Authentication}). -@cindex inferior packages -Technically, an ``inferior'' is essentially a separate Guix process connected -to your main Guix process through a REPL (@pxref{Invoking guix repl}). The -@code{(guix inferior)} module allows you to create inferiors and to -communicate with them. It also provides a high-level interface to browse and -manipulate the packages that an inferior provides---@dfn{inferior packages}. +@node Replicating Guix +@section Replicating Guix -When combined with channels (@pxref{Channels}), inferiors provide a simple way -to interact with a separate revision of Guix. For example, let's assume you -want to install in your profile the current @code{guile} package, along with -the @code{guile-json} as it existed in an older revision of Guix---perhaps -because the newer @code{guile-json} has an incompatible API and you want to -run your code against the old API@. To do that, you could write a manifest for -use by @code{guix package --manifest} (@pxref{Invoking guix package}); in that -manifest, you would create an inferior for that old Guix revision you care -about, and you would look up the @code{guile-json} package in the inferior: +@cindex pinning, channels +@cindex replicating Guix +@cindex reproducibility, of Guix +The @command{guix pull --list-generations} output above shows precisely which +commits were used to build this instance of Guix. We can thus replicate it, +say, on another machine, by providing a channel specification in +@file{~/.config/guix/channels.scm} that is ``pinned'' to these commits: @lisp -(use-modules (guix inferior) (guix channels) - (srfi srfi-1)) ;for 'first' +;; Deploy specific commits of my channels of interest. +(list (channel + (name 'guix) + (url "https://git.savannah.gnu.org/git/guix.git") + (commit "6298c3ffd9654d3231a6f25390b056483e8f407c")) + (channel + (name 'variant-packages) + (url "https://example.org/variant-packages.git") + (commit "dd3df5e2c8818760a8fc0bd699e55d3b69fef2bb"))) +@end lisp -(define channels - ;; This is the old revision from which we want to - ;; extract guile-json. - (list (channel - (name 'guix) - (url "https://git.savannah.gnu.org/git/guix.git") - (commit - "65956ad3526ba09e1f7a40722c96c6ef7c0936fe")))) +The @command{guix describe --format=channels} command can even generate this +list of channels directly (@pxref{Invoking guix describe}). The resulting +file can be used with the -C options of @command{guix pull} +(@pxref{Invoking guix pull}) or @command{guix time-machine} +(@pxref{Invoking guix time-machine}). -(define inferior - ;; An inferior representing the above revision. - (inferior-for-channels channels)) +At this point the two machines run the @emph{exact same Guix}, with access to +the @emph{exact same packages}. The output of @command{guix build gimp} on +one machine will be exactly the same, bit for bit, as the output of the same +command on the other machine. It also means both machines have access to all +the source code of Guix and, transitively, to all the source code of every +package it defines. -;; Now create a manifest with the current "guile" package -;; and the old "guile-json" package. -(packages->manifest - (list (first (lookup-inferior-packages inferior "guile-json")) - (specification->package "guile"))) +This gives you super powers, allowing you to track the provenance of binary +artifacts with very fine grain, and to reproduce software environments at +will---some sort of ``meta reproducibility'' capabilities, if you will. +@xref{Inferiors}, for another way to take advantage of these super powers. + +@node Channel Authentication +@section Channel Authentication + +@anchor{channel-authentication} +@cindex authentication, of channel code +The @command{guix pull} and @command{guix time-machine} commands +@dfn{authenticate} the code retrieved from channels: they make sure each +commit that is fetched is signed by an authorized developer. The goal +is to protect from unauthorized modifications to the channel that would +lead users to run malicious code. + +As a user, you must provide a @dfn{channel introduction} in your +channels file so that Guix knows how to authenticate its first commit. +A channel specification, including its introduction, looks something +along these lines: + +@lisp +(channel + (name 'some-channel) + (url "https://example.org/some-channel.git") + (introduction + (make-channel-introduction + "6f0d8cc0d88abb59c324b2990bfee2876016bb86" + (openpgp-fingerprint + "CABB A931 C0FF EEC6 900D 0CFB 090B 1199 3D9A EBB5")))) @end lisp -On its first run, @command{guix package --manifest} might have to build the -channel you specified before it can create the inferior; subsequent runs will -be much faster because the Guix revision will be cached. +The specification above shows the name and URL of the channel. The call +to @code{make-channel-introduction} above specifies that authentication +of this channel starts at commit @code{6f0d8cc@dots{}}, which is signed +by the OpenPGP key with fingerprint @code{CABB A931@dots{}}. -The @code{(guix inferior)} module provides the following procedures to open an -inferior: +For the main channel, called @code{guix}, you automatically get that +information from your Guix installation. For other channels, include +the channel introduction provided by the channel authors in your +@file{channels.scm} file. Make sure you retrieve the channel +introduction from a trusted source since that is the root of your trust. -@deffn {Scheme Procedure} inferior-for-channels @var{channels} @ - [#:cache-directory] [#:ttl] -Return an inferior for @var{channels}, a list of channels. Use the cache at -@var{cache-directory}, where entries can be reclaimed after @var{ttl} seconds. -This procedure opens a new connection to the build daemon. +If you're curious about the authentication mechanics, read on! -As a side effect, this procedure may build or substitute binaries for -@var{channels}, which can take time. -@end deffn +@cindex primary URL, channels +@node Primary URL +@section Primary URL -@deffn {Scheme Procedure} open-inferior @var{directory} @ - [#:command "bin/guix"] -Open the inferior Guix in @var{directory}, running -@code{@var{directory}/@var{command} repl} or equivalent. Return @code{#f} if -the inferior could not be launched. -@end deffn +Channel authors can indicate the primary URL of their channel's Git +repository in the @file{.guix-channel} file, like so: -@cindex inferior packages -The procedures listed below allow you to obtain and manipulate inferior -packages. +@lisp +(channel + (version 0) + (url "https://example.org/guix.git")) +@end lisp -@deffn {Scheme Procedure} inferior-packages @var{inferior} -Return the list of packages known to @var{inferior}. -@end deffn +This allows @command{guix pull} to determine whether it is pulling code +from a mirror of the channel; when that is the case, it warns the user +that the mirror might be stale and displays the primary URL. That way, +users cannot be tricked into fetching code from a stale mirror that does +not receive security updates. -@deffn {Scheme Procedure} lookup-inferior-packages @var{inferior} @var{name} @ - [@var{version}] -Return the sorted list of inferior packages matching @var{name} in -@var{inferior}, with highest version numbers first. If @var{version} is true, -return only packages with a version number prefixed by @var{version}. -@end deffn +This feature only makes sense for authenticated repositories, such as +the official @code{guix} channel, for which @command{guix pull} ensures +the code it fetches is authentic. -@deffn {Scheme Procedure} inferior-package? @var{obj} -Return true if @var{obj} is an inferior package. -@end deffn +@cindex personal packages (channels) +@cindex channels, for personal packages +@node Creating a Channel +@section Creating a Channel -@deffn {Scheme Procedure} inferior-package-name @var{package} -@deffnx {Scheme Procedure} inferior-package-version @var{package} -@deffnx {Scheme Procedure} inferior-package-synopsis @var{package} -@deffnx {Scheme Procedure} inferior-package-description @var{package} -@deffnx {Scheme Procedure} inferior-package-home-page @var{package} -@deffnx {Scheme Procedure} inferior-package-location @var{package} -@deffnx {Scheme Procedure} inferior-package-inputs @var{package} -@deffnx {Scheme Procedure} inferior-package-native-inputs @var{package} -@deffnx {Scheme Procedure} inferior-package-propagated-inputs @var{package} -@deffnx {Scheme Procedure} inferior-package-transitive-propagated-inputs @var{package} -@deffnx {Scheme Procedure} inferior-package-native-search-paths @var{package} -@deffnx {Scheme Procedure} inferior-package-transitive-native-search-paths @var{package} -@deffnx {Scheme Procedure} inferior-package-search-paths @var{package} -These procedures are the counterpart of package record accessors -(@pxref{package Reference}). Most of them work by querying the inferior -@var{package} comes from, so the inferior must still be live when you call -these procedures. -@end deffn +Let's say you have a bunch of custom package variants or personal packages +that you think would make little sense to contribute to the Guix project, but +would like to have these packages transparently available to you at the +command line. You would first write modules containing those package +definitions (@pxref{Package Modules}), maintain them in a Git repository, and +then you and anyone else can use it as an additional channel to get packages +from. Neat, no? -Inferior packages can be used transparently like any other package or -file-like object in G-expressions (@pxref{G-Expressions}). They are also -transparently handled by the @code{packages->manifest} procedure, which is -commonly use in manifests (@pxref{Invoking guix package, the -@option{--manifest} option of @command{guix package}}). Thus you can insert -an inferior package pretty much anywhere you would insert a regular package: -in manifests, in the @code{packages} field of your @code{operating-system} -declaration, and so on. +@c What follows stems from discussions at +@c as well as +@c earlier discussions on guix-devel@gnu.org. +@quotation Warning +Before you, dear user, shout---``woow this is @emph{soooo coool}!''---and +publish your personal channel to the world, we would like to share a few words +of caution: -@node Invoking guix describe -@section Invoking @command{guix describe} +@itemize +@item +Before publishing a channel, please consider contributing your package +definitions to Guix proper (@pxref{Contributing}). Guix as a project is open +to free software of all sorts, and packages in Guix proper are readily +available to all Guix users and benefit from the project's quality assurance +process. -@cindex reproducibility -@cindex replicating Guix -Often you may want to answer questions like: ``Which revision of Guix am I -using?'' or ``Which channels am I using?'' This is useful information in many -situations: if you want to @emph{replicate} an environment on a different -machine or user account, if you want to report a bug or to determine what -change in the channels you are using caused it, or if you want to record your -system state for reproducibility purposes. The @command{guix describe} -command answers these questions. +@item +When you maintain package definitions outside Guix, we, Guix developers, +consider that @emph{the compatibility burden is on you}. Remember that +package modules and package definitions are just Scheme code that uses various +programming interfaces (APIs). We want to remain free to change these APIs to +keep improving Guix, possibly in ways that break your channel. We never +change APIs gratuitously, but we will @emph{not} commit to freezing APIs +either. -When run from a @command{guix pull}ed @command{guix}, @command{guix describe} -displays the channel(s) that it was built from, including their repository URL -and commit IDs (@pxref{Channels}): +@item +Corollary: if you're using an external channel and that channel breaks, please +@emph{report the issue to the channel authors}, not to the Guix project. +@end itemize -@example -$ guix describe -Generation 10 Sep 03 2018 17:32:44 (current) - guix e0fa68c - repository URL: https://git.savannah.gnu.org/git/guix.git - branch: master - commit: e0fa68c7718fffd33d81af415279d6ddb518f727 -@end example +You've been warned! Having said this, we believe external channels are a +practical way to exert your freedom to augment Guix' package collection and to +share your improvements, which are basic tenets of +@uref{https://www.gnu.org/philosophy/free-sw.html, free software}. Please +email us at @email{guix-devel@@gnu.org} if you'd like to discuss this. +@end quotation -If you're familiar with the Git version control system, this is similar in -spirit to @command{git describe}; the output is also similar to that of -@command{guix pull --list-generations}, but limited to the current generation -(@pxref{Invoking guix pull, the @option{--list-generations} option}). Because -the Git commit ID shown above unambiguously refers to a snapshot of Guix, this -information is all it takes to describe the revision of Guix you're using, and -also to replicate it. +To create a channel, create a Git repository containing your own package +modules and make it available. The repository can contain anything, but a +useful channel will contain Guile modules that export packages. Once you +start using a channel, Guix will behave as if the root directory of that +channel's Git repository has been added to the Guile load path (@pxref{Load +Paths,,, guile, GNU Guile Reference Manual}). For example, if your channel +contains a file at @file{my-packages/my-tools.scm} that defines a Guile +module, then the module will be available under the name @code{(my-packages +my-tools)}, and you will be able to use it like any other module +(@pxref{Modules,,, guile, GNU Guile Reference Manual}). -To make it easier to replicate Guix, @command{guix describe} can also be asked -to return a list of channels instead of the human-readable description above: +As a channel author, consider bundling authentication material with your +channel so that users can authenticate it. @xref{Channel +Authentication}, and @ref{Specifying Channel Authorizations}, for info +on how to do it. -@example -$ guix describe -f channels -(list (channel - (name 'guix) - (url "https://git.savannah.gnu.org/git/guix.git") - (commit - "e0fa68c7718fffd33d81af415279d6ddb518f727") - (introduction - (make-channel-introduction - "9edb3f66fd807b096b48283debdcddccfea34bad" - (openpgp-fingerprint - "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA"))))) -@end example -@noindent -You can save this to a file and feed it to @command{guix pull -C} on some -other machine or at a later point in time, which will instantiate @emph{this -exact Guix revision} (@pxref{Invoking guix pull, the @option{-C} option}). -From there on, since you're able to deploy the same revision of Guix, you can -just as well @emph{replicate a complete software environment}. We humbly -think that this is @emph{awesome}, and we hope you'll like it too! +@cindex subdirectory, channels +@node Package Modules in a Sub-directory +@section Package Modules in a Sub-directory -The details of the options supported by @command{guix describe} are as -follows: +As a channel author, you may want to keep your channel modules in a +sub-directory. If your modules are in the sub-directory @file{guix}, you must +add a meta-data file @file{.guix-channel} that contains: -@table @code -@item --format=@var{format} -@itemx -f @var{format} -Produce output in the specified @var{format}, one of: +@lisp +(channel + (version 0) + (directory "guix")) +@end lisp -@table @code -@item human -produce human-readable output; -@item channels -produce a list of channel specifications that can be passed to @command{guix -pull -C} or installed as @file{~/.config/guix/channels.scm} (@pxref{Invoking -guix pull}); -@item channels-sans-intro -like @code{channels}, but omit the @code{introduction} field; use it to -produce a channel specification suitable for Guix version 1.1.0 or -earlier---the @code{introduction} field has to do with channel -authentication (@pxref{Channels, Channel Authentication}) and is not -supported by these older versions; -@item json -@cindex JSON -produce a list of channel specifications in JSON format; -@item recutils -produce a list of channel specifications in Recutils format. -@end table +@cindex dependencies, channels +@cindex meta-data, channels +@node Declaring Channel Dependencies +@section Declaring Channel Dependencies -@item --list-formats -Display available formats for @option{--format} option. +Channel authors may decide to augment a package collection provided by other +channels. They can declare their channel to be dependent on other channels in +a meta-data file @file{.guix-channel}, which is to be placed in the root of +the channel repository. -@item --profile=@var{profile} -@itemx -p @var{profile} -Display information about @var{profile}. -@end table +The meta-data file should contain a simple S-expression like this: -@node Invoking guix archive -@section Invoking @command{guix archive} +@lisp +(channel + (version 0) + (dependencies + (channel + (name 'some-collection) + (url "https://example.org/first-collection.git") -@cindex @command{guix archive} -@cindex archive -The @command{guix archive} command allows users to @dfn{export} files -from the store into a single archive, and to later @dfn{import} them on -a machine that runs Guix. -In particular, it allows store files to be transferred from one machine -to the store on another machine. + ;; The 'introduction' bit below is optional: you would + ;; provide it for dependencies that can be authenticated. + (introduction + (channel-introduction + (version 0) + (commit "a8883b58dc82e167c96506cf05095f37c2c2c6cd") + (signer "CABB A931 C0FF EEC6 900D 0CFB 090B 1199 3D9A EBB5")))) + (channel + (name 'some-other-collection) + (url "https://example.org/second-collection.git") + (branch "testing")))) +@end lisp -@quotation Note -If you're looking for a way to produce archives in a format suitable for -tools other than Guix, @pxref{Invoking guix pack}. -@end quotation +In the above example this channel is declared to depend on two other channels, +which will both be fetched automatically. The modules provided by the channel +will be compiled in an environment where the modules of all these declared +channels are available. -@cindex exporting store items -To export store files as an archive to standard output, run: +For the sake of reliability and maintainability, you should avoid dependencies +on channels that you don't control, and you should aim to keep the number of +dependencies to a minimum. -@example -guix archive --export @var{options} @var{specifications}... -@end example +@cindex channel authorizations +@node Specifying Channel Authorizations +@section Specifying Channel Authorizations -@var{specifications} may be either store file names or package -specifications, as for @command{guix package} (@pxref{Invoking guix -package}). For instance, the following command creates an archive -containing the @code{gui} output of the @code{git} package and the main -output of @code{emacs}: +@anchor{channel-authorizations} +As we saw above, Guix ensures the source code it pulls from channels +comes from authorized developers. As a channel author, you need to +specify the list of authorized developers in the +@file{.guix-authorizations} file in the channel's Git repository. The +authentication rule is simple: each commit must be signed by a key +listed in the @file{.guix-authorizations} file of its parent +commit(s)@footnote{Git commits form a @dfn{directed acyclic graph} +(DAG). Each commit can have zero or more parents; ``regular'' commits +have one parent and merge commits have two parent commits. Read +@uref{https://eagain.net/articles/git-for-computer-scientists/, @i{Git +for Computer Scientists}} for a great overview.} The +@file{.guix-authorizations} file looks like this: -@example -guix archive --export git:gui /gnu/store/...-emacs-24.3 > great.nar -@end example +@lisp +;; Example '.guix-authorizations' file. -If the specified packages are not built yet, @command{guix archive} -automatically builds them. The build process may be controlled with the -common build options (@pxref{Common Build Options}). +(authorizations + (version 0) ;current file format version -To transfer the @code{emacs} package to a machine connected over SSH, -one would run: + (("AD17 A21E F8AE D8F1 CC02 DBD9 F8AE D8F1 765C 61E3" + (name "alice")) + ("2A39 3FFF 68F4 EF7A 3D29 12AF 68F4 EF7A 22FB B2D5" + (name "bob")) + ("CABB A931 C0FF EEC6 900D 0CFB 090B 1199 3D9A EBB5" + (name "charlie")))) +@end lisp -@example -guix archive --export -r emacs | ssh the-machine guix archive --import -@end example +Each fingerprint is followed by optional key/value pairs, as in the +example above. Currently these key/value pairs are ignored. -@noindent -Similarly, a complete user profile may be transferred from one machine -to another like this: +This authentication rule creates a chicken-and-egg issue: how do we +authenticate the first commit? Related to that: how do we deal with +channels whose repository history contains unsigned commits and lack +@file{.guix-authorizations}? And how do we fork existing channels? -@example -guix archive --export -r $(readlink -f ~/.guix-profile) | \ - ssh the-machine guix archive --import -@end example +@cindex channel introduction +Channel introductions answer these questions by describing the first +commit of a channel that should be authenticated. The first time a +channel is fetched with @command{guix pull} or @command{guix +time-machine}, the command looks up the introductory commit and verifies +that it is signed by the specified OpenPGP key. From then on, it +authenticates commits according to the rule above. -@noindent -However, note that, in both examples, all of @code{emacs} and the -profile as well as all of their dependencies are transferred (due to -@option{-r}), regardless of what is already available in the store on -the target machine. The @option{--missing} option can help figure out -which items are missing from the target store. The @command{guix copy} -command simplifies and optimizes this whole process, so this is probably -what you should use in this case (@pxref{Invoking guix copy}). +Additionally, your channel must provide all the OpenPGP keys that were +ever mentioned in @file{.guix-authorizations}, stored as @file{.key} +files, which can be either binary or ``ASCII-armored''. By default, +those @file{.key} files are searched for in the branch named +@code{keyring} but you can specify a different branch name in +@code{.guix-channel} like so: -@cindex nar, archive format -@cindex normalized archive (nar) -@cindex nar bundle, archive format -Each store item is written in the @dfn{normalized archive} or @dfn{nar} -format (described below), and the output of @command{guix archive ---export} (and input of @command{guix archive --import}) is a @dfn{nar -bundle}. +@lisp +(channel + (version 0) + (keyring-reference "my-keyring-branch")) +@end lisp -The nar format is -comparable in spirit to `tar', but with differences -that make it more appropriate for our purposes. First, rather than -recording all Unix metadata for each file, the nar format only mentions -the file type (regular, directory, or symbolic link); Unix permissions -and owner/group are dismissed. Second, the order in which directory -entries are stored always follows the order of file names according to -the C locale collation order. This makes archive production fully -deterministic. +To summarize, as the author of a channel, there are three things you have +to do to allow users to authenticate your code: -That nar bundle format is essentially the concatenation of zero or more -nars along with metadata for each store item it contains: its file name, -references, corresponding derivation, and a digital signature. +@enumerate +@item +Export the OpenPGP keys of past and present committers with @command{gpg +--export} and store them in @file{.key} files, by default in a branch +named @code{keyring} (we recommend making it an @dfn{orphan branch}). -When exporting, the daemon digitally signs the contents of the archive, -and that digital signature is appended. When importing, the daemon -verifies the signature and rejects the import in case of an invalid -signature or if the signing key is not authorized. -@c FIXME: Add xref to daemon doc about signatures. +@item +Introduce an initial @file{.guix-authorizations} in the channel's +repository. Do that in a signed commit (@pxref{Commit Access}, for +information on how to sign Git commits.) -The main options are: +@item +Advertise the channel introduction, for instance on your channel's web +page. The channel introduction, as we saw above, is the commit/key +pair---i.e., the commit that introduced @file{.guix-authorizations}, and +the fingerprint of the OpenPGP used to sign it. +@end enumerate -@table @code -@item --export -Export the specified store files or packages (see below). Write the -resulting archive to the standard output. +Before pushing to your public Git repository, you can run @command{guix +git-authenticate} to verify that you did sign all the commits you are +about to push with an authorized key: -Dependencies are @emph{not} included in the output, unless -@option{--recursive} is passed. +@example +guix git authenticate @var{commit} @var{signer} +@end example -@item -r -@itemx --recursive -When combined with @option{--export}, this instructs @command{guix archive} -to include dependencies of the given items in the archive. Thus, the -resulting archive is self-contained: it contains the closure of the -exported store items. +@noindent +where @var{commit} and @var{signer} are your channel introduction. +@xref{Invoking guix git authenticate}, for details. -@item --import -Read an archive from the standard input, and import the files listed -therein into the store. Abort if the archive has an invalid digital -signature, or if it is signed by a public key not among the authorized -keys (see @option{--authorize} below). +Publishing a signed channel requires discipline: any mistake, such as an +unsigned commit or a commit signed by an unauthorized key, will prevent +users from pulling from your channel---well, that's the whole point of +authentication! Pay attention to merges in particular: merge commits +are considered authentic if and only if they are signed by a key present +in the @file{.guix-authorizations} file of @emph{both} branches. -@item --missing -Read a list of store file names from the standard input, one per line, -and write on the standard output the subset of these files missing from -the store. +@cindex news, for channels +@node Writing Channel News +@section Writing Channel News -@item --generate-key[=@var{parameters}] -@cindex signing, archives -Generate a new key pair for the daemon. This is a prerequisite before -archives can be exported with @option{--export}. Note that this -operation usually takes time, because it needs to gather enough entropy -to generate the key pair. +Channel authors may occasionally want to communicate to their users +information about important changes in the channel. You'd send them all +an email, but that's not convenient. -The generated key pair is typically stored under @file{/etc/guix}, in -@file{signing-key.pub} (public key) and @file{signing-key.sec} (private -key, which must be kept secret). When @var{parameters} is omitted, -an ECDSA key using the Ed25519 curve is generated, or, for Libgcrypt -versions before 1.6.0, it is a 4096-bit RSA key. -Alternatively, @var{parameters} can specify -@code{genkey} parameters suitable for Libgcrypt (@pxref{General -public-key related Functions, @code{gcry_pk_genkey},, gcrypt, The -Libgcrypt Reference Manual}). +Instead, channels can provide a @dfn{news file}; when the channel users +run @command{guix pull}, that news file is automatically read and +@command{guix pull --news} can display the announcements that correspond +to the new commits that have been pulled, if any. -@item --authorize -@cindex authorizing, archives -Authorize imports signed by the public key passed on standard input. -The public key must be in ``s-expression advanced format''---i.e., the -same format as the @file{signing-key.pub} file. +To do that, channel authors must first declare the name of the news file +in their @file{.guix-channel} file: -The list of authorized keys is kept in the human-editable file -@file{/etc/guix/acl}. The file contains -@url{https://people.csail.mit.edu/rivest/Sexp.txt, ``advanced-format -s-expressions''} and is structured as an access-control list in the -@url{https://theworld.com/~cme/spki.txt, Simple Public-Key Infrastructure -(SPKI)}. +@lisp +(channel + (version 0) + (news-file "etc/news.txt")) +@end lisp -@item --extract=@var{directory} -@itemx -x @var{directory} -Read a single-item archive as served by substitute servers -(@pxref{Substitutes}) and extract it to @var{directory}. This is a -low-level operation needed in only very narrow use cases; see below. +The news file itself, @file{etc/news.txt} in this example, must look +something like this: -For example, the following command extracts the substitute for Emacs -served by @code{@value{SUBSTITUTE-SERVER}} to @file{/tmp/emacs}: +@lisp +(channel-news + (version 0) + (entry (tag "the-bug-fix") + (title (en "Fixed terrible bug") + (fr "Oh la la")) + (body (en "@@emph@{Good news@}! It's fixed!") + (eo "Certe ĝi pli bone funkcias nun!"))) + (entry (commit "bdcabe815cd28144a2d2b4bc3c5057b051fa9906") + (title (en "Added a great package") + (ca "Què vol dir guix?")) + (body (en "Don't miss the @@code@{hello@} package!")))) +@end lisp -@example -$ wget -O - \ - https://@value{SUBSTITUTE-SERVER}/nar/gzip/@dots{}-emacs-24.5 \ - | gunzip | guix archive -x /tmp/emacs -@end example +While the news file is using the Scheme syntax, avoid naming it with a +@file{.scm} extension or else it will get picked up when building the +channel and yield an error since it is not a valid module. +Alternatively, you can move the channel module to a subdirectory and +store the news file in another directory. -Single-item archives are different from multiple-item archives produced -by @command{guix archive --export}; they contain a single store item, -and they do @emph{not} embed a signature. Thus this operation does -@emph{no} signature verification and its output should be considered -unsafe. +The file consists of a list of @dfn{news entries}. Each entry is +associated with a commit or tag: it describes changes made in this +commit, possibly in preceding commits as well. Users see entries only +the first time they obtain the commit the entry refers to. -The primary purpose of this operation is to facilitate inspection of -archive contents coming from possibly untrusted substitute servers -(@pxref{Invoking guix challenge}). +The @code{title} field should be a one-line summary while @code{body} +can be arbitrarily long, and both can contain Texinfo markup +(@pxref{Overview,,, texinfo, GNU Texinfo}). Both the title and body are +a list of language tag/message tuples, which allows @command{guix pull} +to display news in the language that corresponds to the user's locale. -@item --list -@itemx -t -Read a single-item archive as served by substitute servers -(@pxref{Substitutes}) and print the list of files it contains, as in -this example: +If you want to translate news using a gettext-based workflow, you can +extract translatable strings with @command{xgettext} (@pxref{xgettext +Invocation,,, gettext, GNU Gettext Utilities}). For example, assuming +you write news entries in English first, the command below creates a PO +file containing the strings to translate: @example -$ wget -O - \ - https://@value{SUBSTITUTE-SERVER}/nar/lzip/@dots{}-emacs-26.3 \ - | lzip -d | guix archive -t +xgettext -o news.po -l scheme -ken etc/news.txt @end example -@end table +To sum up, yes, you could use your channel as a blog. But beware, this +is @emph{not quite} what your users might expect. @c ********************************************************************* -- cgit 1.4.1 From 3b7bd8cc904dc7e550135aa81be05c6c64a84f07 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Sun, 27 Sep 2020 22:39:49 +0200 Subject: doc: Move "Primary URL" after "Specifying Channel Authorizations". * doc/guix.texi (Primary URL): Move after "Specifying Channel Authorizations" since the audience is channel authors who've already taken care of authorizations. --- doc/guix.texi | 50 +++++++++++++++++++++++++------------------------- 1 file changed, 25 insertions(+), 25 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index cc8214c257..d20b33cd9a 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -233,11 +233,11 @@ Channels * Using a Custom Guix Channel:: Using a customized Guix. * Replicating Guix:: Running the @emph{exact same} Guix. * Channel Authentication:: How Guix verifies what it fetches. -* Primary URL:: Distinguishing mirror to original. * Creating a Channel:: How to write your custom channel. * Package Modules in a Sub-directory:: Specifying the channel's package modules location. * Declaring Channel Dependencies:: How to depend on other channels. * Specifying Channel Authorizations:: Defining channel authors authorizations. +* Primary URL:: Distinguishing mirror to original. * Writing Channel News:: Communicating information to channel's users. Development @@ -4691,11 +4691,11 @@ updates. * Using a Custom Guix Channel:: Using a customized Guix. * Replicating Guix:: Running the @emph{exact same} Guix. * Channel Authentication:: How Guix verifies what it fetches. -* Primary URL:: Distinguishing mirror to original. * Creating a Channel:: How to write your custom channel. * Package Modules in a Sub-directory:: Specifying the channel's package modules location. * Declaring Channel Dependencies:: How to depend on other channels. * Specifying Channel Authorizations:: Defining channel authors authorizations. +* Primary URL:: Distinguishing mirror to original. * Writing Channel News:: Communicating information to channel's users. @end menu @@ -4852,29 +4852,6 @@ introduction from a trusted source since that is the root of your trust. If you're curious about the authentication mechanics, read on! -@cindex primary URL, channels -@node Primary URL -@section Primary URL - -Channel authors can indicate the primary URL of their channel's Git -repository in the @file{.guix-channel} file, like so: - -@lisp -(channel - (version 0) - (url "https://example.org/guix.git")) -@end lisp - -This allows @command{guix pull} to determine whether it is pulling code -from a mirror of the channel; when that is the case, it warns the user -that the mirror might be stale and displays the primary URL. That way, -users cannot be tricked into fetching code from a stale mirror that does -not receive security updates. - -This feature only makes sense for authenticated repositories, such as -the official @code{guix} channel, for which @command{guix pull} ensures -the code it fetches is authentic. - @cindex personal packages (channels) @cindex channels, for personal packages @node Creating a Channel @@ -5099,6 +5076,29 @@ authentication! Pay attention to merges in particular: merge commits are considered authentic if and only if they are signed by a key present in the @file{.guix-authorizations} file of @emph{both} branches. +@cindex primary URL, channels +@node Primary URL +@section Primary URL + +Channel authors can indicate the primary URL of their channel's Git +repository in the @file{.guix-channel} file, like so: + +@lisp +(channel + (version 0) + (url "https://example.org/guix.git")) +@end lisp + +This allows @command{guix pull} to determine whether it is pulling code +from a mirror of the channel; when that is the case, it warns the user +that the mirror might be stale and displays the primary URL. That way, +users cannot be tricked into fetching code from a stale mirror that does +not receive security updates. + +This feature only makes sense for authenticated repositories, such as +the official @code{guix} channel, for which @command{guix pull} ensures +the code it fetches is authentic. + @cindex news, for channels @node Writing Channel News @section Writing Channel News -- cgit 1.4.1 From 8e2d5a0c0d0542f220fff16d92c6b5267ebbcc1f Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Sun, 27 Sep 2020 22:42:40 +0200 Subject: doc: Move channel @cindex within their nodes. * doc/guix.texi (Channels): Move @cindex at the beginning of the node they refer to from the end of the previous node. --- doc/guix.texi | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index d20b33cd9a..81f46cc18d 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -4852,11 +4852,11 @@ introduction from a trusted source since that is the root of your trust. If you're curious about the authentication mechanics, read on! -@cindex personal packages (channels) -@cindex channels, for personal packages @node Creating a Channel @section Creating a Channel +@cindex personal packages (channels) +@cindex channels, for personal packages Let's say you have a bunch of custom package variants or personal packages that you think would make little sense to contribute to the Guix project, but would like to have these packages transparently available to you at the @@ -4919,10 +4919,10 @@ Authentication}, and @ref{Specifying Channel Authorizations}, for info on how to do it. -@cindex subdirectory, channels @node Package Modules in a Sub-directory @section Package Modules in a Sub-directory +@cindex subdirectory, channels As a channel author, you may want to keep your channel modules in a sub-directory. If your modules are in the sub-directory @file{guix}, you must add a meta-data file @file{.guix-channel} that contains: @@ -4933,11 +4933,11 @@ add a meta-data file @file{.guix-channel} that contains: (directory "guix")) @end lisp -@cindex dependencies, channels -@cindex meta-data, channels @node Declaring Channel Dependencies @section Declaring Channel Dependencies +@cindex dependencies, channels +@cindex meta-data, channels Channel authors may decide to augment a package collection provided by other channels. They can declare their channel to be dependent on other channels in a meta-data file @file{.guix-channel}, which is to be placed in the root of @@ -4975,10 +4975,10 @@ For the sake of reliability and maintainability, you should avoid dependencies on channels that you don't control, and you should aim to keep the number of dependencies to a minimum. -@cindex channel authorizations @node Specifying Channel Authorizations @section Specifying Channel Authorizations +@cindex channel authorizations @anchor{channel-authorizations} As we saw above, Guix ensures the source code it pulls from channels comes from authorized developers. As a channel author, you need to @@ -5076,10 +5076,10 @@ authentication! Pay attention to merges in particular: merge commits are considered authentic if and only if they are signed by a key present in the @file{.guix-authorizations} file of @emph{both} branches. -@cindex primary URL, channels @node Primary URL @section Primary URL +@cindex primary URL, channels Channel authors can indicate the primary URL of their channel's Git repository in the @file{.guix-channel} file, like so: @@ -5099,10 +5099,10 @@ This feature only makes sense for authenticated repositories, such as the official @code{guix} channel, for which @command{guix pull} ensures the code it fetches is authentic. -@cindex news, for channels @node Writing Channel News @section Writing Channel News +@cindex news, for channels Channel authors may occasionally want to communicate to their users information about important changes in the channel. You'd send them all an email, but that's not convenient. -- cgit 1.4.1 From 09ffacd455e6e93651536321339e18b64a752233 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Mon, 28 Sep 2020 11:32:35 +0200 Subject: doc: Clarify how to choose a user account shell. * doc/guix.texi (User Accounts): Add example with custom shell and custom home directory. Add examples of shell gexps. --- doc/guix.texi | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 81f46cc18d..c600d577ac 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -12830,8 +12830,19 @@ User accounts and groups are entirely managed through the "audio" ;sound card "video" ;video devices such as webcams "cdrom")) ;the good ol' CD-ROM - (comment "Bob's sister") - (home-directory "/home/alice")) + (comment "Bob's sister")) +@end lisp + +Here's a user account that uses a different shell and a custom home +directory (the default would be @file{"/home/bob"}): + +@lisp +(user-account + (name "bob") + (group "users") + (comment "Alice's bro") + (shell (file-append zsh "/bin/zsh")) + (home-directory "/home/robert")) @end lisp When booting or upon completion of @command{guix system reconfigure}, @@ -12876,7 +12887,19 @@ if it does not exist yet. @item @code{shell} (default: Bash) This is a G-expression denoting the file name of a program to be used as -the shell (@pxref{G-Expressions}). +the shell (@pxref{G-Expressions}). For example, you would refer to the +Bash executable like this: + +@lisp +(file-append bash "/bin/bash") +@end lisp + +@noindent +... and to the Zsh executable like that: + +@lisp +(file-append zsh "/bin/zsh") +@end lisp @item @code{system?} (default: @code{#f}) This Boolean value indicates whether the account is a ``system'' -- cgit 1.4.1 From 4c698cd51209a0102477478b026ea04bd7e45908 Mon Sep 17 00:00:00 2001 From: Tobias Geerinckx-Rice Date: Mon, 28 Sep 2020 15:55:55 +0200 Subject: services: Add elogind ‘handle-lid-switch-external-power’. MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * gnu/services/desktop.scm : Add an handle-lid-switch-external-power field, mapping to the HandleLidSwitchExternalPower logind.conf setting. * doc/guix.texi (Desktop Services): ‘Document’ it. --- doc/guix.texi | 2 + gnu/services/desktop.scm | 119 ++++++++++++++++++++++++----------------------- 2 files changed, 63 insertions(+), 58 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index c600d577ac..452453241c 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -17262,6 +17262,8 @@ their default values are: @code{suspend} @item handle-lid-switch-docked @code{ignore} +@item handle-lid-switch-external-power +@code{ignore} @item power-key-ignore-inhibited? @code{#f} @item suspend-key-ignore-inhibited? diff --git a/gnu/services/desktop.scm b/gnu/services/desktop.scm index 1dcf71d359..96bacf1cff 100644 --- a/gnu/services/desktop.scm +++ b/gnu/services/desktop.scm @@ -596,64 +596,66 @@ include the @command{udisksctl} command, part of UDisks, and GNOME Disks." (define-record-type* elogind-configuration make-elogind-configuration elogind-configuration? - (elogind elogind-package - (default elogind)) - (kill-user-processes? elogind-kill-user-processes? - (default #f)) - (kill-only-users elogind-kill-only-users - (default '())) - (kill-exclude-users elogind-kill-exclude-users - (default '("root"))) - (inhibit-delay-max-seconds elogind-inhibit-delay-max-seconds - (default 5)) - (handle-power-key elogind-handle-power-key - (default 'poweroff)) - (handle-suspend-key elogind-handle-suspend-key - (default 'suspend)) - (handle-hibernate-key elogind-handle-hibernate-key - ;; (default 'hibernate) - ;; XXX Ignore it for now, since we don't - ;; yet handle resume-from-hibernation in - ;; our initrd. - (default 'ignore)) - (handle-lid-switch elogind-handle-lid-switch - (default 'suspend)) - (handle-lid-switch-docked elogind-handle-lid-switch-docked - (default 'ignore)) - (power-key-ignore-inhibited? elogind-power-key-ignore-inhibited? - (default #f)) - (suspend-key-ignore-inhibited? elogind-suspend-key-ignore-inhibited? - (default #f)) - (hibernate-key-ignore-inhibited? elogind-hibernate-key-ignore-inhibited? - (default #f)) - (lid-switch-ignore-inhibited? elogind-lid-switch-ignore-inhibited? - (default #t)) - (holdoff-timeout-seconds elogind-holdoff-timeout-seconds - (default 30)) - (idle-action elogind-idle-action - (default 'ignore)) - (idle-action-seconds elogind-idle-action-seconds - (default (* 30 60))) - (runtime-directory-size-percent elogind-runtime-directory-size-percent - (default 10)) - (runtime-directory-size elogind-runtime-directory-size - (default #f)) - (remove-ipc? elogind-remove-ipc? - (default #t)) - - (suspend-state elogind-suspend-state - (default '("mem" "standby" "freeze"))) - (suspend-mode elogind-suspend-mode - (default '())) - (hibernate-state elogind-hibernate-state - (default '("disk"))) - (hibernate-mode elogind-hibernate-mode - (default '("platform" "shutdown"))) - (hybrid-sleep-state elogind-hybrid-sleep-state - (default '("disk"))) - (hybrid-sleep-mode elogind-hybrid-sleep-mode - (default - '("suspend" "platform" "shutdown")))) + (elogind elogind-package + (default elogind)) + (kill-user-processes? elogind-kill-user-processes? + (default #f)) + (kill-only-users elogind-kill-only-users + (default '())) + (kill-exclude-users elogind-kill-exclude-users + (default '("root"))) + (inhibit-delay-max-seconds elogind-inhibit-delay-max-seconds + (default 5)) + (handle-power-key elogind-handle-power-key + (default 'poweroff)) + (handle-suspend-key elogind-handle-suspend-key + (default 'suspend)) + (handle-hibernate-key elogind-handle-hibernate-key + ;; (default 'hibernate) + ;; XXX Ignore it for now, since we don't + ;; yet handle resume-from-hibernation in + ;; our initrd. + (default 'ignore)) + (handle-lid-switch elogind-handle-lid-switch + (default 'suspend)) + (handle-lid-switch-docked elogind-handle-lid-switch-docked + (default 'ignore)) + (handle-lid-switch-external-power elogind-handle-lid-switch-external-power + (default 'ignore)) + (power-key-ignore-inhibited? elogind-power-key-ignore-inhibited? + (default #f)) + (suspend-key-ignore-inhibited? elogind-suspend-key-ignore-inhibited? + (default #f)) + (hibernate-key-ignore-inhibited? elogind-hibernate-key-ignore-inhibited? + (default #f)) + (lid-switch-ignore-inhibited? elogind-lid-switch-ignore-inhibited? + (default #t)) + (holdoff-timeout-seconds elogind-holdoff-timeout-seconds + (default 30)) + (idle-action elogind-idle-action + (default 'ignore)) + (idle-action-seconds elogind-idle-action-seconds + (default (* 30 60))) + (runtime-directory-size-percent elogind-runtime-directory-size-percent + (default 10)) + (runtime-directory-size elogind-runtime-directory-size + (default #f)) + (remove-ipc? elogind-remove-ipc? + (default #t)) + + (suspend-state elogind-suspend-state + (default '("mem" "standby" "freeze"))) + (suspend-mode elogind-suspend-mode + (default '())) + (hibernate-state elogind-hibernate-state + (default '("disk"))) + (hibernate-mode elogind-hibernate-mode + (default '("platform" "shutdown"))) + (hybrid-sleep-state elogind-hybrid-sleep-state + (default '("disk"))) + (hybrid-sleep-mode elogind-hybrid-sleep-mode + (default + '("suspend" "platform" "shutdown")))) (define (elogind-configuration-file config) (define (yesno x) @@ -705,6 +707,7 @@ include the @command{udisksctl} command, part of UDisks, and GNOME Disks." ("HandleHibernateKey" (handle-action elogind-handle-hibernate-key)) ("HandleLidSwitch" (handle-action elogind-handle-lid-switch)) ("HandleLidSwitchDocked" (handle-action elogind-handle-lid-switch-docked)) + ("HandleLidSwitchExternalPower" (handle-action elogind-handle-lid-switch-external-power)) ("PowerKeyIgnoreInhibited" (yesno elogind-power-key-ignore-inhibited?)) ("SuspendKeyIgnoreInhibited" (yesno elogind-suspend-key-ignore-inhibited?)) ("HibernateKeyIgnoreInhibited" (yesno elogind-hibernate-key-ignore-inhibited?)) -- cgit 1.4.1 From d367a7f3d00de20d5c6a88123297f878b3044fee Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Sun, 27 Sep 2020 14:55:32 +0200 Subject: services: guix: Generate key pair if needed during activation. * gnu/services/base.scm (guix-activation): Invoke "guix archive --generate-key". * doc/guix.texi (Invoking guix archive) (Invoking guix deploy): Mention that 'guix-service-type' takes care of generating the key pair. --- doc/guix.texi | 11 +++++++---- gnu/services/base.scm | 13 +++++++++---- 2 files changed, 16 insertions(+), 8 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 452453241c..e6f04e1413 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -4600,9 +4600,11 @@ the store. @item --generate-key[=@var{parameters}] @cindex signing, archives Generate a new key pair for the daemon. This is a prerequisite before -archives can be exported with @option{--export}. Note that this -operation usually takes time, because it needs to gather enough entropy -to generate the key pair. +archives can be exported with @option{--export}. This +operation is usually instantaneous but it can take time if the system's +entropy pool needs to be refilled. On Guix System, +@code{guix-service-type} takes care of generating this key pair the +first boot. The generated key pair is typically stored under @file{/etc/guix}, in @file{signing-key.pub} (public key) and @file{signing-key.sec} (private @@ -29684,7 +29686,8 @@ a Virtual Private Server (VPS) provider. In such a case, a different Do note that you first need to generate a key pair on the coordinator machine to allow the daemon to export signed archives of files from the store -(@pxref{Invoking guix archive}). +(@pxref{Invoking guix archive}), though this step is automatic on Guix +System: @example # guix archive --generate-key diff --git a/gnu/services/base.scm b/gnu/services/base.scm index bef4eef241..04bc991356 100644 --- a/gnu/services/base.scm +++ b/gnu/services/base.scm @@ -1653,10 +1653,15 @@ proxy of 'guix-daemon'...~%") ;; otherwise call 'chown' here, but the problem is that on a COW overlayfs, ;; chown leads to an entire copy of the tree, which is a bad idea. - ;; Optionally authorize substitute server keys. - (if authorize-key? - (substitute-key-authorization keys guix) - #~#f)))) + ;; Generate a key pair and optionally authorize substitute server keys. + #~(begin + (unless (file-exists? "/etc/guix/signing-key.pub") + (system* #$(file-append guix "/bin/guix") "archive" + "--generate-key")) + + #$(if authorize-key? + (substitute-key-authorization keys guix) + #~#f))))) (define* (references-file item #:optional (name "references")) "Return a file that contains the list of references of ITEM." -- cgit 1.4.1 From 37283f9f3e0130e262e0e4e0dc4b260471bc485b Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Sun, 27 Sep 2020 15:14:36 +0200 Subject: services: hurd-vm: Initialize the guest's SSH/Guix keys at activation time. * gnu/services/virtualization.scm (initialize-hurd-vm-substitutes) (hurd-vm-activation): New procedures. (hurd-vm-service-type)[extensions]: Add ACTIVATION-SERVICE-TYPE extension. * doc/guix.texi (Transparent Emulation with QEMU): Mention GNU/Hurd. (The Hurd in a Virtual Machine): Explain which files are automatically installed and mention offloading. --- doc/guix.texi | 33 ++++++++++++++++++-- gnu/services/virtualization.scm | 67 ++++++++++++++++++++++++++++++++++++++++- 2 files changed, 96 insertions(+), 4 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index e6f04e1413..ff2e582347 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -25445,6 +25445,8 @@ emulation of program binaries built for different architectures---e.g., it allows you to transparently execute an ARMv7 program on an x86_64 machine. It achieves this by combining the @uref{https://www.qemu.org, QEMU} emulator and the @code{binfmt_misc} feature of the kernel Linux. +This feature only allows you to emulate GNU/Linux on a different +architecture, but see below for GNU/Hurd support. @defvr {Scheme Variable} qemu-binfmt-service-type This is the type of the QEMU/binfmt service for transparent emulation. @@ -25647,10 +25649,11 @@ If the @file{/etc/childhurd} directory does not exist, the @code{secret-service} running in the Childhurd will be sent an empty list of secrets. -Typical use to populate @file{"/etc/childhurd"} with a tree of -non-volatile secrets, like so +By default, the service automatically populates @file{/etc/childhurd} +with the following non-volatile secrets, unless they already exist: @example +/etc/childhurd/etc/guix/acl /etc/childhurd/etc/guix/signing-key.pub /etc/childhurd/etc/guix/signing-key.sec /etc/childhurd/etc/ssh/ssh_host_ed25519_key @@ -25659,8 +25662,32 @@ non-volatile secrets, like so /etc/childhurd/etc/ssh/ssh_host_ecdsa_key.pub @end example -to be sent to the Childhurd, including permissions. +These files are automatically sent to the guest Hurd VM when it boots, +including permissions. +@cindex childhurd, offloading +@cindex Hurd, offloading +Having these files in place means that only a couple of things are +missing to allow the host to offload @code{i586-gnu} builds to the +childhurd: + +@enumerate +@item +Authorizing the childhurd's key on the host so that the host accepts +build results coming from the childhurd, which can be done like so: + +@example +guix archive --authorize < \ + /etc/childhurd/etc/guix/signing-key.pub +@end example + +@item +Adding the childhurd to @file{/etc/guix/machines.scm} (@pxref{Daemon +Offload Setup}). +@end enumerate + +We're working towards making that happen automatically---get in touch +with us at @email{guix-devel@@gnu.org} to discuss it! @end table @end deftp diff --git a/gnu/services/virtualization.scm b/gnu/services/virtualization.scm index b84203ad18..c639fa3741 100644 --- a/gnu/services/virtualization.scm +++ b/gnu/services/virtualization.scm @@ -23,6 +23,7 @@ #:use-module (gnu bootloader grub) #:use-module (gnu image) #:use-module (gnu packages admin) + #:use-module (gnu packages package-management) #:use-module (gnu packages ssh) #:use-module (gnu packages virtualization) #:use-module (gnu services base) @@ -992,13 +993,77 @@ is added to the OS specified in CONFIG." (shell (file-append shadow "/sbin/nologin")) (system? #t)))) +(define (initialize-hurd-vm-substitutes) + "Initialize the Hurd VM's key pair and ACL and store it on the host." + (define run + (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils) + (ice-9 match)) + + (define host-key + "/etc/guix/signing-key.pub") + + (define host-acl + "/etc/guix/acl") + + (match (command-line) + ((_ guest-config-directory) + (setenv "GUIX_CONFIGURATION_DIRECTORY" + guest-config-directory) + (invoke #+(file-append guix "/bin/guix") "archive" + "--generate-key") + + (when (file-exists? host-acl) + ;; Copy the host ACL. + (copy-file host-acl + (string-append guest-config-directory + "/acl"))) + + (when (file-exists? host-key) + ;; Add the host key to the childhurd's ACL. + (let ((key (open-fdes host-key O_RDONLY))) + (close-fdes 0) + (dup2 key 0) + (execl #+(file-append guix "/bin/guix") + "guix" "archive" "--authorize")))))))) + + (program-file "initialize-hurd-vm-substitutes" run)) + +(define (hurd-vm-activation config) + "Return a gexp to activate the Hurd VM according to CONFIG." + (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils)) + + (define secret-directory + #$(hurd-vm-configuration-secret-root config)) + + (define ssh-directory + (string-append secret-directory "/etc/ssh")) + + (define guix-directory + (string-append secret-directory "/etc/guix")) + + (unless (file-exists? ssh-directory) + ;; Generate SSH host keys under SSH-DIRECTORY. + (mkdir-p ssh-directory) + (invoke #$(file-append openssh "/bin/ssh-keygen") + "-A" "-f" secret-directory)) + + (unless (file-exists? guix-directory) + (invoke #$(initialize-hurd-vm-substitutes) + guix-directory))))) + (define hurd-vm-service-type (service-type (name 'hurd-vm) (extensions (list (service-extension shepherd-root-service-type hurd-vm-shepherd-service) (service-extension account-service-type - (const %hurd-vm-accounts)))) + (const %hurd-vm-accounts)) + (service-extension activation-service-type + hurd-vm-activation))) (default-value (hurd-vm-configuration)) (description "Provide a virtual machine (VM) running GNU/Hurd, also known as a -- cgit 1.4.1 From 313f492657f1d0863c641fa5ee7f5b7028e27c94 Mon Sep 17 00:00:00 2001 From: Mathieu Othacehe Date: Fri, 31 Jul 2020 16:49:29 +0200 Subject: scripts: system: Add support for image-type. * guix/scripts/system.scm (list-image-types): New procedure, (%options): add "image-type" and "list-image-types" options, remove "file-system-type" option, (show-help): adapt accordingly, (%default-options): also adapt, and set the default "image-type" to "raw", (perform-action): add image-type argument and remove file-system-type argument, (process-action): adapt perform-action call, (system-derivation-for-action): remove base-image argument, add image-type argument, and use it to create the image passed to "system-image". * tests/guix-system.sh: Adapt accordingly and add a test for "--list-image-types" command. * doc/guix.texi (Building the Installation Image, Invoking guix system): Adapt accordingly. Signed-off-by: Mathieu Othacehe --- Makefile.am | 5 ++-- doc/guix.texi | 43 ++++++++++++++++-------------- guix/scripts/system.scm | 70 +++++++++++++++++++++++++++++++------------------ tests/guix-system.sh | 9 ++++--- 4 files changed, 75 insertions(+), 52 deletions(-) (limited to 'doc') diff --git a/Makefile.am b/Makefile.am index 8e91e1e558..9c3ff4420f 100644 --- a/Makefile.am +++ b/Makefile.am @@ -833,9 +833,8 @@ release: dist-with-updated-version -v1 --no-grafts --fallback for system in $(GUIX_SYSTEM_SUPPORTED_SYSTEMS) ; do \ image=`$(top_builddir)/pre-inst-env \ - guix system disk-image \ - --file-system-type=iso9660 \ - --label="GUIX_$${system}_$(VERSION)" \ + guix system disk-image -t iso9660 \ + --label="GUIX_$${system}_$(VERSION)" \ --system=$$system --fallback \ gnu/system/install.scm` ; \ if [ ! -f "$$image" ] ; then \ diff --git a/doc/guix.texi b/doc/guix.texi index ff2e582347..e8458ad8d8 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -40,7 +40,7 @@ Copyright @copyright{} 2016, 2017, 2018, 2019, 2020 Julien Lepiller@* Copyright @copyright{} 2016 Alex ter Weele@* Copyright @copyright{} 2016, 2017, 2018, 2019 Christopher Baines@* Copyright @copyright{} 2017, 2018, 2019 Clément Lassieur@* -Copyright @copyright{} 2017, 2018 Mathieu Othacehe@* +Copyright @copyright{} 2017, 2018, 2020 Mathieu Othacehe@* Copyright @copyright{} 2017 Federico Beffa@* Copyright @copyright{} 2017, 2018 Carlo Zancanaro@* Copyright @copyright{} 2017 Thomas Danckaert@* @@ -2568,8 +2568,7 @@ The installation image described above was built using the @command{guix system} command, specifically: @example -guix system disk-image --file-system-type=iso9660 \ - gnu/system/install.scm +guix system disk-image -t iso9660 gnu/system/install.scm @end example Have a look at @file{gnu/system/install.scm} in the source tree, @@ -29375,24 +29374,28 @@ a value. Docker images are built to contain exactly what they need, so the @option{--image-size} option is ignored in the case of @code{docker-image}. -You can specify the root file system type by using the -@option{--file-system-type} option. It defaults to @code{ext4}. When its -value is @code{iso9660}, the @option{--label} option can be used to specify -a volume ID with @code{disk-image}. +The @code{disk-image} command can produce various image types. The +image type can be selected using the @command{--image-type} option. It +defaults to @code{raw}. When its value is @code{iso9660}, the +@option{--label} option can be used to specify a volume ID with +@code{disk-image}. -When using @code{vm-image}, the returned image is in qcow2 format, which -the QEMU emulator can efficiently use. @xref{Running Guix in a VM}, -for more information on how to run the image in a virtual machine. - -When using @code{disk-image}, a raw disk image is produced; it can be -copied as is to a USB stick, for instance. Assuming @code{/dev/sdc} is -the device corresponding to a USB stick, one can copy the image to it -using the following command: +When using the @code{raw} image type, a raw disk image is produced; it +can be copied as is to a USB stick, for instance. Assuming +@code{/dev/sdc} is the device corresponding to a USB stick, one can copy +the image to it using the following command: @example # dd if=$(guix system disk-image my-os.scm) of=/dev/sdc status=progress @end example +The @code{--list-image-types} command lists all the available image +types. + +When using @code{vm-image}, the returned image is in qcow2 format, which +the QEMU emulator can efficiently use. @xref{Running Guix in a VM}, +for more information on how to run the image in a virtual machine. + When using @code{docker-image}, a Docker image is produced. Guix builds the image from scratch, not from a pre-existing Docker base image. As a result, it contains @emph{exactly} what you define in the operating @@ -29494,17 +29497,17 @@ information, one can rebuild the image to make sure it really contains what it pretends to contain; or they could use that to derive a variant of the image. -@item --file-system-type=@var{type} +@item --image-type=@var{type} @itemx -t @var{type} -For the @code{disk-image} action, create a file system of the given -@var{type} on the image. +For the @code{disk-image} action, create an image with given @var{type}. -When this option is omitted, @command{guix system} uses @code{ext4}. +When this option is omitted, @command{guix system} uses the @code{raw} +image type. @cindex ISO-9660 format @cindex CD image format @cindex DVD image format -@option{--file-system-type=iso9660} produces an ISO-9660 image, suitable +@option{--image-type=iso9660} produces an ISO-9660 image, suitable for burning on CDs and DVDs. @item --image-size=@var{size} diff --git a/guix/scripts/system.scm b/guix/scripts/system.scm index bd5f84fc5b..7b3eacf2e1 100644 --- a/guix/scripts/system.scm +++ b/guix/scripts/system.scm @@ -666,8 +666,8 @@ checking this by themselves in their 'check' procedure." ;;; Action. ;;; -(define* (system-derivation-for-action os base-image action - #:key image-size file-system-type +(define* (system-derivation-for-action os action + #:key image-size image-type full-boot? container-shared-network? mappings label) "Return as a monadic value the derivation for OS according to ACTION." @@ -690,12 +690,15 @@ checking this by themselves in their 'check' procedure." (* 70 (expt 2 20))) #:mappings mappings)) ((disk-image) - (lower-object - (system-image - (image - (inherit (if label (image-with-label base-image label) base-image)) - (size image-size) - (operating-system os))))) + (let ((base-image (os->image os #:type image-type))) + (lower-object + (system-image + (image + (inherit (if label + (image-with-label base-image label) + base-image)) + (size image-size) + (operating-system os)))))) ((docker-image) (system-docker-image os #:shared-network? container-shared-network?)))) @@ -748,18 +751,19 @@ and TARGET arguments." install-bootloader? dry-run? derivations-only? use-substitutes? bootloader-target target - image-size file-system-type full-boot? label - container-shared-network? + image-size image-type + full-boot? label container-shared-network? (mappings '()) (gc-root #f)) "Perform ACTION for OS. INSTALL-BOOTLOADER? specifies whether to install bootloader; BOOTLOADER-TAGET is the target for the bootloader; TARGET is the target root directory; IMAGE-SIZE is the size of the image to be built, for -the 'vm-image' and 'disk-image' actions. The root file system is created as a -FILE-SYSTEM-TYPE file system. FULL-BOOT? is used for the 'vm' action; it -determines whether to boot directly to the kernel or to the bootloader. -CONTAINER-SHARED-NETWORK? determines if the container will use a separate -network namespace. +the 'vm-image' and 'disk-image' actions. IMAGE-TYPE is the type of image to +be built. + +FULL-BOOT? is used for the 'vm' action; it determines whether to +boot directly to the kernel or to the bootloader. CONTAINER-SHARED-NETWORK? +determines if the container will use a separate network namespace. When DERIVATIONS-ONLY? is true, print the derivation file name(s) without building anything. @@ -799,11 +803,9 @@ static checks." (check-initrd-modules os))) (mlet* %store-monad - ((target* (current-target-system)) - (image -> (find-image file-system-type target*)) - (sys (system-derivation-for-action os image action + ((sys (system-derivation-for-action os action #:label label - #:file-system-type file-system-type + #:image-type image-type #:image-size image-size #:full-boot? full-boot? #:container-shared-network? container-shared-network? @@ -886,6 +888,17 @@ Run 'herd status' to view the list of services on your system.\n")))))) #:node-type (shepherd-service-node-type shepherds) #:reverse-edges? #t))) + +;;; +;;; Images. +;;; + +(define (list-image-types) + "Print the available image types." + (display (G_ "The available image types are:\n")) + (newline) + (format #t "~{ - ~a ~%~}" (map image-type-name (force %image-types)))) + ;;; ;;; Options. @@ -945,9 +958,9 @@ Some ACTIONS support additional ARGS.\n")) apply STRATEGY (one of nothing-special, backtrace, or debug) when an error occurs while reading FILE")) (display (G_ " - --file-system-type=TYPE - for 'disk-image', produce a root file system of TYPE - (one of 'ext4', 'iso9660')")) + --list-image-types list available image types")) + (display (G_ " + -t, --image-type=TYPE for 'disk-image', produce an image of TYPE")) (display (G_ " --image-size=SIZE for 'vm-image', produce an image of SIZE")) (display (G_ " @@ -1008,10 +1021,14 @@ Some ACTIONS support additional ARGS.\n")) (lambda (opt name arg result) (alist-cons 'on-error (string->symbol arg) result))) - (option '(#\t "file-system-type") #t #f + (option '(#\t "image-type") #t #f (lambda (opt name arg result) - (alist-cons 'file-system-type arg + (alist-cons 'image-type (string->symbol arg) result))) + (option '("list-image-types") #f #f + (lambda (opt name arg result) + (list-image-types) + (exit 0))) (option '("image-size") #t #f (lambda (opt name arg result) (alist-cons 'image-size (size->number arg) @@ -1080,7 +1097,7 @@ Some ACTIONS support additional ARGS.\n")) (debug . 0) (verbosity . #f) ;default (validate-reconfigure . ,ensure-forward-reconfigure) - (file-system-type . "ext4") + (image-type . raw) (image-size . guess) (install-bootloader? . #t) (label . #f))) @@ -1177,7 +1194,8 @@ resulting from command-line parsing." (assoc-ref opts 'skip-safety-checks?) #:validate-reconfigure (assoc-ref opts 'validate-reconfigure) - #:file-system-type (assoc-ref opts 'file-system-type) + #:image-type (lookup-image-type-by-name + (assoc-ref opts 'image-type)) #:image-size (assoc-ref opts 'image-size) #:full-boot? (assoc-ref opts 'full-boot?) #:container-shared-network? diff --git a/tests/guix-system.sh b/tests/guix-system.sh index 0e22686a34..667e084fcf 100644 --- a/tests/guix-system.sh +++ b/tests/guix-system.sh @@ -261,8 +261,8 @@ guix system vm "$tmpfile" -d | grep '\.drv$' drv1="`guix system vm "$tmpfile" -d`" drv2="`guix system vm "$tmpfile" -d`" test "$drv1" = "$drv2" -drv1="`guix system disk-image --file-system-type=iso9660 "$tmpfile" -d`" -drv2="`guix system disk-image --file-system-type=iso9660 "$tmpfile" -d`" +drv1="`guix system disk-image -t iso9660 "$tmpfile" -d`" +drv2="`guix system disk-image -t iso9660 "$tmpfile" -d`" test "$drv1" = "$drv2" make_user_config "group-that-does-not-exist" "users" @@ -320,5 +320,8 @@ guix system -n vm gnu/system/examples/vm-image.tmpl guix system -n vm-image gnu/system/examples/vm-image.tmpl # This invocation was taken care of in the loop above: # guix system -n disk-image gnu/system/examples/bare-bones.tmpl -guix system -n disk-image --file-system-type=iso9660 gnu/system/examples/bare-bones.tmpl +guix system -n disk-image -t iso9660 gnu/system/examples/bare-bones.tmpl guix system -n docker-image gnu/system/examples/docker-image.tmpl + +# Verify that at least the raw image type is available. +guix system --list-image-types | grep "raw" -- cgit 1.4.1 From da6aec32cf13cf06cfda909c45f7f35625c4e3fa Mon Sep 17 00:00:00 2001 From: Oleg Pykhalov Date: Mon, 28 Sep 2020 23:52:51 +0300 Subject: gnu: Add webssh service. * gnu/services/ssh.scm: (): New record type. (%webssh-configuration-nginx, webssh-service-type): New variables. (webssh-account, webssh-activation, webssh-shepherd-service): New procedures. * doc/guix.texi: Document this. --- doc/guix.texi | 75 +++++++++++++++++++++++++++++ gnu/services/ssh.scm | 131 ++++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 205 insertions(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index e8458ad8d8..79c79b6a96 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -15547,6 +15547,81 @@ may cause undefined behaviour. @end table @end deftp +@cindex WebSSH +@deffn {Scheme Variable} webssh-service-type +This is the type for the @uref{https://webssh.huashengdun.org/, WebSSH} +program that runs a web SSH client. WebSSH can be run manually from the +command-line by passing arguments to the binary @command{wssh} from the +package @code{webssh}, but it can also be run as a Guix service. This +latter use case is documented here. + +For example, to specify a service running WebSSH on loopback interface +on port @code{8888} with reject policy with a list of allowed to +connection hosts, and NGINX as a reverse-proxy to this service listening +for HTTPS connection, add this call to the operating system's +@code{services} field: + +@lisp +(service webssh-service-type + (webssh-configuration (address "127.0.0.1") + (port 8888) + (policy 'reject) + (known-hosts '("localhost ecdsa-sha2-nistp256 AAAA…" + "127.0.0.1 ecdsa-sha2-nistp256 AAAA…")))) + +(service nginx-service-type + (nginx-configuration + (server-blocks + (list + (nginx-server-configuration + (inherit %webssh-configuration-nginx) + (server-name '("webssh.example.com")) + (listen '("443 ssl")) + (ssl-certificate (letsencrypt-certificate "webssh.example.com")) + (ssl-certificate-key (letsencrypt-key "webssh.example.com")) + (locations + (cons (nginx-location-configuration + (uri "/.well-known") + (body '("root /var/www;"))) + (nginx-server-configuration-locations %webssh-configuration-nginx)))))))) +@end lisp +@end deffn + +@deftp {Data Type} webssh-configuration +Data type representing the configuration for @code{webssh-service}. + +@table @asis +@item @code{package} (default: @var{webssh}) +@code{webssh} package to use. + +@item @code{user-name} (default: @var{"webssh"}) +User name or user ID that file transfers to and from that module should take +place. + +@item @code{group-name} (default: @var{"webssh"}) +Group name or group ID that will be used when accessing the module. + +@item @code{address} (default: @var{#f}) +IP address on which @command{webssh} listens for incoming connections. + +@item @code{port} (default: @var{8888}) +TCP port on which @command{webssh} listens for incoming connections. + +@item @code{policy} (default: @var{#f}) +Connection policy. @var{reject} policy requires to specify @var{known-hosts}. + +@item @code{known-hosts} (default: @var{'()}) +List of hosts which allowed for SSH connection from @command{webssh}. + +@item @code{log-file} (default: @file{"/var/log/webssh.log"}) +Name of the file where @command{rsync} writes its log file. + +@item @code{log-level} (default: @var{#f}) +Logging level. + +@end table +@end deftp + @defvr {Scheme Variable} %facebook-host-aliases This variable contains a string for use in @file{/etc/hosts} (@pxref{Host Names,,, libc, The GNU C Library Reference Manual}). Each diff --git a/gnu/services/ssh.scm b/gnu/services/ssh.scm index ced21c0742..1891db0487 100644 --- a/gnu/services/ssh.scm +++ b/gnu/services/ssh.scm @@ -5,6 +5,7 @@ ;;; Copyright © 2017 Clément Lassieur ;;; Copyright © 2019 Ricardo Wurmus ;;; Copyright © 2020 pinoaffe +;;; Copyright © 2020 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -26,6 +27,7 @@ #:use-module (gnu packages admin) #:use-module (gnu services) #:use-module (gnu services shepherd) + #:use-module (gnu services web) #:use-module (gnu system pam) #:use-module (gnu system shadow) #:use-module (guix gexp) @@ -50,7 +52,12 @@ autossh-configuration autossh-configuration? - autossh-service-type)) + autossh-service-type + + webssh-configuration + webssh-configuration? + webssh-service-type + %webssh-configuration-nginx)) ;;; Commentary: ;;; @@ -732,4 +739,126 @@ object." autossh-service-activation))) (default-value (autossh-configuration)))) + +;;; +;;; WebSSH +;;; + +(define-record-type* + webssh-configuration make-webssh-configuration + webssh-configuration? + (package webssh-configuration-package ;package + (default webssh)) + (user-name webssh-configuration-user-name ;string + (default "webssh")) + (group-name webssh-configuration-group-name ;string + (default "webssh")) + (policy webssh-configuration-policy ;symbol + (default #f)) + (known-hosts webssh-configuration-known-hosts ;list of strings + (default #f)) + (port webssh-configuration-port ;number + (default #f)) + (address webssh-configuration-address ;string + (default #f)) + (log-file webssh-configuration-log-file ;string + (default "/var/log/webssh.log")) + (log-level webssh-configuration-log-level ;symbol + (default #f))) + +(define %webssh-configuration-nginx + (nginx-server-configuration + (listen '("80")) + (locations + (list (nginx-location-configuration + (uri "/") + (body '("proxy_pass http://127.0.0.1:8888;" + "proxy_http_version 1.1;" + "proxy_read_timeout 300;" + "proxy_set_header Upgrade $http_upgrade;" + "proxy_set_header Connection \"upgrade\";" + "proxy_set_header Host $http_host;" + "proxy_set_header X-Real-IP $remote_addr;" + "proxy_set_header X-Real-PORT $remote_port;"))))))) + +(define webssh-account + ;; Return the user accounts and user groups for CONFIG. + (match-lambda + (($ _ user-name group-name _ _ _ _ _ _) + (list (user-group + (name group-name)) + (user-account + (name user-name) + (group group-name) + (comment "webssh privilege separation user") + (home-directory (string-append "/var/run/" user-name)) + (shell #~(string-append #$shadow "/sbin/nologin"))))))) + +(define webssh-activation + ;; Return the activation GEXP for CONFIG. + (match-lambda + (($ _ user-name group-name policy known-hosts _ _ + log-file _) + (with-imported-modules '((guix build utils)) + #~(begin + (let* ((home-dir (string-append "/var/run/" #$user-name)) + (ssh-dir (string-append home-dir "/.ssh")) + (known-hosts-file (string-append ssh-dir "/known_hosts"))) + (call-with-output-file #$log-file (const #t)) + (mkdir-p ssh-dir) + (case '#$policy + ((reject) + (if '#$known-hosts + (call-with-output-file known-hosts-file + (lambda (port) + (for-each (lambda (host) (display host port) (newline port)) + '#$known-hosts))) + (display-hint (G_ "webssh: reject policy requires `known-hosts'."))))) + (for-each (lambda (file) + (chown file + (passwd:uid (getpw #$user-name)) + (group:gid (getpw #$group-name)))) + (list #$log-file ssh-dir known-hosts-file)) + (chmod ssh-dir #o700))))))) + +(define webssh-shepherd-service + (match-lambda + (($ package user-name group-name policy _ port + address log-file log-level) + (list (shepherd-service + (provision '(webssh)) + (documentation "Run webssh daemon.") + (start #~(make-forkexec-constructor + `(,(string-append #$webssh "/bin/wssh") + ,(string-append "--log-file-prefix=" #$log-file) + ,@(case '#$log-level + ((debug) '("--logging=debug")) + (else '())) + ,@(case '#$policy + ((reject) '("--policy=reject")) + (else '())) + ,@(if #$port + (list (string-append "--port=" (number->string #$port))) + '()) + ,@(if #$address + (list (string-append "--address=" #$address)) + '())) + #:user #$user-name + #:group #$group-name)) + (stop #~(make-kill-destructor))))))) + +(define webssh-service-type + (service-type + (name 'webssh) + (extensions + (list (service-extension shepherd-root-service-type + webssh-shepherd-service) + (service-extension account-service-type + webssh-account) + (service-extension activation-service-type + webssh-activation))) + (default-value (webssh-configuration)) + (description + "Run the webssh."))) + ;;; ssh.scm ends here -- cgit 1.4.1 From 3c01fcc1bb9c086f487d9694cb91a57d7abd0880 Mon Sep 17 00:00:00 2001 From: Florian Pelz Date: Fri, 2 Oct 2020 21:59:38 +0200 Subject: doc: Explain why '--without-tests' may fail with modified 'check' phase. * doc/guix.texi (Package Transformation Options): Explain. --- doc/guix.texi | 6 ++++++ 1 file changed, 6 insertions(+) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 79c79b6a96..da48c8a72d 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -9350,6 +9350,12 @@ The command above installs @code{python-notebook} on top of rebuilds everything that depends on @code{python}, including @code{python-notebook} itself. +Internally, @option{--without-tests} relies on changing the +@code{#:tests?} option of a package's @code{check} phase (@pxref{Build +Systems}). Note that some packages use a customized @code{check} phase +that does not respect a @code{#:tests? #f} setting. Therefore, +@option{--without-tests} has no effect on these packages. + @end table @node Additional Build Options -- cgit 1.4.1 From 8e1907a72430aa989125b053573ef0897c480697 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Fri, 25 Sep 2020 17:16:34 +0200 Subject: guix package: Re-apply package transformation when upgrading. * guix/scripts/package.scm (transaction-upgrade-entry)[upgrade]: Add 'transform' parameter. Pass PKG through it. Use 'manifest-entry-with-transformations'. Call 'options->transformation' to get the transformation procedure. * tests/guix-package.sh: Add 'guix package -u' test. * tests/packages.scm ("transaction-upgrade-entry, transformation options preserved"): New test. * doc/guix.texi (Invoking guix package): Mention that transformations are preserved across upgrades. (Package Transformation Options): Likewise. --- doc/guix.texi | 27 +++++++++++++++++++++++++++ guix/scripts/package.scm | 20 +++++++++++++++----- tests/guix-package.sh | 15 +++++++++++++++ tests/packages.scm | 23 +++++++++++++++++++++++ 4 files changed, 80 insertions(+), 5 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index da48c8a72d..a6260a12aa 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -3101,6 +3101,29 @@ in the distribution currently installed. To update your distribution, you should regularly run @command{guix pull} (@pxref{Invoking guix pull}). +@cindex package transformations, upgrades +When upgrading, package transformations that were originally applied +when creating the profile are automatically re-applied (@pxref{Package +Transformation Options}). For example, assume you first installed Emacs +from the tip of its development branch with: + +@example +guix install emacs-next --with-branch=emacs-next=master +@end example + +Next time you run @command{guix upgrade}, Guix will again pull the tip +of the Emacs development branch and build @code{emacs-next} from that +checkout. + +Note that transformation options such as @option{--with-branch} and +@option{--with-source} depend on external state; it is up to you to +ensure that they work as expected. You can also discard a +transformations that apply to a package by running: + +@example +guix install @var{package} +@end example + @item --do-not-upgrade[=@var{regexp} @dots{}] When used together with the @option{--upgrade} option, do @emph{not} upgrade any packages whose name matches a @var{regexp}. For example, to @@ -9193,6 +9216,10 @@ This is a convenient way to create customized packages on the fly without having to type in the definitions of package variants (@pxref{Defining Packages}). +Package transformation options are preserved across upgrades: +@command{guix upgrade} attempts to apply transformation options +initially used when creating the profile to the upgraded packages. + @table @code @item --with-source=@var{source} diff --git a/guix/scripts/package.scm b/guix/scripts/package.scm index 83f8c123d9..2f04652634 100644 --- a/guix/scripts/package.scm +++ b/guix/scripts/package.scm @@ -218,12 +218,13 @@ non-zero relevance score." (output (manifest-entry-output old))) transaction))) - (define (upgrade entry) + (define (upgrade entry transform) (match entry (($ name version output (? string? path)) (match (find-best-packages-by-name name #f) ((pkg . rest) - (let ((candidate-version (package-version pkg))) + (let* ((pkg (transform store pkg)) + (candidate-version (package-version pkg))) (match (package-superseded pkg) ((? package? new) (supersede entry new)) @@ -231,12 +232,14 @@ non-zero relevance score." (case (version-compare candidate-version version) ((>) (manifest-transaction-install-entry - (package->manifest-entry* pkg output) + (manifest-entry-with-transformations + (package->manifest-entry* pkg output)) transaction)) ((<) transaction) ((=) - (let* ((new (package->manifest-entry* pkg output))) + (let* ((new (manifest-entry-with-transformations + (package->manifest-entry* pkg output)))) ;; Here we want to determine whether the NEW actually ;; differs from ENTRY, but we need to intercept ;; 'build-things' calls because they would prevent us from @@ -255,7 +258,14 @@ non-zero relevance score." (if (manifest-transaction-removal-candidate? entry transaction) transaction - (upgrade entry))) + + ;; Upgrade ENTRY, preserving transformation options listed in its + ;; properties. + (let ((transform (options->transformation + (or (assq-ref (manifest-entry-properties entry) + 'transformations) + '())))) + (upgrade entry transform)))) ;;; diff --git a/tests/guix-package.sh b/tests/guix-package.sh index a43496699b..3e5fa71d20 100644 --- a/tests/guix-package.sh +++ b/tests/guix-package.sh @@ -184,6 +184,21 @@ grep -E 'emacs[[:blank:]]+42\.5\.9rc7' "$tmpfile" rm "$emacs_tarball" "$tmpfile" rmdir "$module_dir" +# Install with package transformations. +guix install --bootstrap -p "$profile" sed --with-input=sed=guile-bootstrap +grep "sed=guile-bootstrap" "$profile/manifest" +test "$(readlink -f "$profile/bin/guile")" \ + = "$(guix build guile-bootstrap)/bin/guile" +test ! -f "$profile/bin/sed" + +# Make sure the package transformation is preserved. +guix package --bootstrap -p "$profile" -u +grep "sed=guile-bootstrap" "$profile/manifest" +test "$(readlink -f "$profile/bin/guile")" \ + = "$(guix build guile-bootstrap)/bin/guile" +test ! -f "$profile/bin/sed" +rm "$profile" "$profile"-[0-9]-link + # Profiles with a relative file name. Make sure we don't create dangling # symlinks--see bug report at # . diff --git a/tests/packages.scm b/tests/packages.scm index af8941c2e2..5d5abcbd76 100644 --- a/tests/packages.scm +++ b/tests/packages.scm @@ -187,6 +187,29 @@ (string=? (manifest-pattern-version pattern) "1") (string=? (manifest-pattern-output pattern) "out"))))))) +(test-equal "transaction-upgrade-entry, transformation options preserved" + (derivation-file-name (package-derivation %store grep)) + + (let* ((old (dummy-package "emacs" (version "1"))) + (props '((transformations . ((with-input . "emacs=grep"))))) + (tx (transaction-upgrade-entry + %store + (manifest-entry + (inherit (package->manifest-entry old)) + (properties props) + (item (string-append (%store-prefix) "/" + (make-string 32 #\e) "-foo-1"))) + (manifest-transaction)))) + (match (manifest-transaction-install tx) + (((? manifest-entry? entry)) + (and (string=? (manifest-entry-version entry) + (package-version grep)) + (string=? (manifest-entry-name entry) + (package-name grep)) + (equal? (manifest-entry-properties entry) props) + (derivation-file-name + (package-derivation %store (manifest-entry-item entry)))))))) + (test-assert "transaction-upgrade-entry, grafts" ;; Ensure that, when grafts are enabled, 'transaction-upgrade-entry' doesn't ;; try to build stuff. -- cgit 1.4.1 From 15955e9b54b8d9c1275d68a0f2b6ccbc5f6c3878 Mon Sep 17 00:00:00 2001 From: Christopher Baines Date: Thu, 17 Sep 2020 15:45:04 +0100 Subject: services: guix: Add guix-build-coordinator-service-type. * gnu/services/guix.scm (): New record type. (guix-build-coordinator-configuration, guix-build-coordinator-configuration?, guix-build-coordinator-configuration-package, guix-build-coordinator-configuration-user, guix-build-coordinator-configuration-group, guix-build-coordinator-configuration-datastore-uri-string, guix-build-coordinator-configuration-agent-communication-uri-string, guix-build-coordinator-configuration-client-communication-uri-string, guix-build-coordinator-configuration-allocation-strategy, guix-build-coordinator-configuration-hooks, guix-build-coordinator-configuration-guile, make-guix-build-coordinator-start-script, guix-build-coordinator-shepherd-services, guix-build-coordinator-activation, guix-build-coordinator-account): New procedures. (guix-build-coordinator-service-type): New variable. * gnu/tests/guix.scm (%test-guix-build-coordinator): New variable. * doc/guix.texi (Guix Services): Document it. --- doc/guix.texi | 67 +++++++++++++++++ gnu/services/guix.scm | 200 +++++++++++++++++++++++++++++++++++++++++++++++++- gnu/tests/guix.scm | 75 ++++++++++++++++++- 3 files changed, 338 insertions(+), 4 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index a6260a12aa..3f1de559e6 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -27764,6 +27764,73 @@ The complete list of possible options can be found in the man page for @node Guix Services @subsection Guix Services +@subsubheading Guix Build Coordinator +The @uref{https://git.cbaines.net/guix/build-coordinator/,Guix Build +Coordinator} aids in distributing derivation builds among machines +running an @dfn{agent}. The build daemon is still used to build the +derivations, but the Guix Build Coordinator manages allocating builds +and working with the results. + +@quotation Note +This service is considered experimental. Configuration options may be +changed in a backwards-incompatible manner, and not all features have +been thorougly tested. +@end quotation + +The Guix Build Coordinator consists of one @dfn{coordinator}, and one or +more connected @dfn{agent} processes. The coordinator process handles +clients submitting builds, and allocating builds to agents. The agent +processes talk to a build daemon to actually perform the builds, then +send the results back to the coordinator. + +There is a script to run the coordinator component of the Guix Build +Coordinator, but the Guix service uses a custom Guile script instead, to +provide better integration with G-expressions used in the configuration. + +@defvar {Scheme Variable} guix-build-coordinator-service-type +Service type for the Guix Build Coordinator. Its value must be a +@code{guix-build-coordinator-configuration} object. +@end defvar + +@deftp {Data Type} guix-build-coordinator-configuration +Data type representing the configuration of the Guix Build Coordinator. + +@table @asis +@item @code{package} (default: @code{guix-build-coordinator}) +The Guix Build Coordinator package to use. + +@item @code{user} (default: @code{"guix-build-coordinator"}) +The system user to run the service as. + +@item @code{group} (default: @code{"guix-build-coordinator"}) +The system group to run the service as. + +@item @code{database-uri-string} (default: @code{"sqlite:///var/lib/guix-build-coordinator/guix_build_coordinator.db"}) +The URI to use for the database. + +@item @code{agent-communication-uri} (default: @code{"http://0.0.0.0:8745"}) +The URI describing how to listen to requests from agent processes. + +@item @code{client-communication-uri} (default: @code{"http://127.0.0.1:8746"}) +The URI describing how to listen to requests from clients. The client +API allows submitting builds and currently isn't authenticated, so take +care when configuring this value. + +@item @code{allocation-strategy} (default: @code{#~basic-build-allocation-strategy}) +A G-expression for the allocation strategy to be used. This is a +procedure that takes the datastore as an argument and populates the +allocation plan in the database. + +@item @code{hooks} (default: @var{'()}) +An association list of hooks. These provide a way to execute arbitrary +code upon certian events, like a build result being processed. + +@item @code{guile} (default: @code{guile-3.0-latest}) +The Guile package with which to run the Guix Build Coordinator. + +@end table +@end deftp + @subsubheading Guix Data Service The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores and provides data about GNU Guix. This includes information about diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm index 10a8581a62..1bacd61190 100644 --- a/gnu/services/guix.scm +++ b/gnu/services/guix.scm @@ -17,20 +17,40 @@ ;;; along with GNU Guix. If not, see . (define-module (gnu services guix) + #:use-module (srfi srfi-1) #:use-module (ice-9 match) #:use-module (guix gexp) #:use-module (guix records) + #:use-module (guix packages) #:use-module ((gnu packages base) #:select (glibc-utf8-locales)) #:use-module (gnu packages admin) + #:use-module (gnu packages databases) #:use-module (gnu packages web) + #:use-module (gnu packages guile) + #:use-module (gnu packages guile-xyz) + #:use-module (gnu packages package-management) #:use-module (gnu services) #:use-module (gnu services base) #:use-module (gnu services admin) #:use-module (gnu services shepherd) #:use-module (gnu services getmail) #:use-module (gnu system shadow) - #:export ( + #:export (guix-build-coordinator-configuration + guix-build-coordinator-configuration? + guix-build-coordinator-configuration-package + guix-build-coordinator-configuration-user + guix-build-coordinator-configuration-group + guix-build-coordinator-configuration-datastore-uri-string + guix-build-coordinator-configuration-agent-communication-uri-string + guix-build-coordinator-configuration-client-communication-uri-string + guix-build-coordinator-configuration-allocation-strategy + guix-build-coordinator-configuration-hooks + guix-build-coordinator-configuration-guile + + guix-build-coordinator-service-type + + guix-data-service-configuration guix-data-service-configuration? guix-data-service-package @@ -45,11 +65,185 @@ ;;;; Commentary: ;;; -;;; This module implements a service that to run instances of the Guix Data -;;; Service, which provides data about Guix over time. +;;; Services specifically related to GNU Guix. ;;; ;;;; Code: +(define-record-type* + guix-build-coordinator-configuration make-guix-build-coordinator-configuration + guix-build-coordinator-configuration? + (package guix-build-coordinator-configuration-package + (default guix-build-coordinator)) + (user guix-build-coordinator-configuration-user + (default "guix-build-coordinator")) + (group guix-build-coordinator-configuration-group + (default "guix-build-coordinator")) + (database-uri-string + guix-build-coordinator-configuration-datastore-uri-string + (default "sqlite:///var/lib/guix-build-coordinator/guix_build_coordinator.db")) + (agent-communication-uri-string + guix-build-coordinator-configuration-agent-communication-uri-string + (default "http://0.0.0.0:8745")) + (client-communication-uri-string + guix-build-coordinator-configuration-client-communication-uri-string + (default "http://127.0.0.1:8746")) + (allocation-strategy + guix-build-coordinator-configuration-allocation-strategy + (default #~basic-build-allocation-strategy)) + (hooks guix-build-coordinator-configuration-hooks + (default '())) + (guile guix-build-coordinator-configuration-guile + (default guile-3.0-latest))) + +(define* (make-guix-build-coordinator-start-script database-uri-string + allocation-strategy + pid-file + guix-build-coordinator-package + #:key + agent-communication-uri-string + client-communication-uri-string + (hooks '()) + (guile guile-3.0)) + (program-file + "start-guix-build-coordinator" + (with-extensions (cons guix-build-coordinator-package + ;; This is a poorly constructed Guile load path, + ;; since it contains things that aren't Guile + ;; libraries, but it means that the Guile libraries + ;; needed for the Guix Build Coordinator don't need + ;; to be individually specified here. + (map second (package-inputs + guix-build-coordinator-package))) + #~(begin + (use-modules (srfi srfi-1) + (ice-9 match) + (web uri) + (prometheus) + (guix-build-coordinator hooks) + (guix-build-coordinator datastore) + (guix-build-coordinator build-allocator) + (guix-build-coordinator coordinator)) + + (let* ((metrics-registry (make-metrics-registry + #:namespace + "guixbuildcoordinator_")) + (datastore (database-uri->datastore + #$database-uri-string + #:metrics-registry metrics-registry)) + (hooks + (list #$@(map (match-lambda + ((name . hook-gexp) + #~(cons name #$hook-gexp))) + hooks))) + (hooks-with-defaults + `(,@hooks + ,@(remove (match-lambda + ((name . _) (assq-ref hooks name))) + %default-hooks))) + (build-coordinator (make-build-coordinator + #:datastore datastore + #:hooks hooks-with-defaults + #:metrics-registry metrics-registry + #:allocation-strategy #$allocation-strategy))) + + (run-coordinator-service + build-coordinator + #:update-datastore? #t + #:pid-file #$pid-file + #:agent-communication-uri (string->uri + #$agent-communication-uri-string) + #:client-communication-uri (string->uri + #$client-communication-uri-string))))) + #:guile guile)) + +(define (guix-build-coordinator-shepherd-services config) + (match-record config + (package user group database-uri-string + agent-communication-uri-string + client-communication-uri-string + allocation-strategy + hooks + guile) + (list + (shepherd-service + (documentation "Guix Build Coordinator") + (provision '(guix-build-coordinator)) + (requirement '(networking)) + (start #~(make-forkexec-constructor + (list #$(make-guix-build-coordinator-start-script + database-uri-string + allocation-strategy + "/var/run/guix-build-coordinator/pid" + package + #:agent-communication-uri-string + agent-communication-uri-string + #:client-communication-uri-string + client-communication-uri-string + #:hooks hooks + #:guile guile)) + #:user #$user + #:group #$group + #:pid-file "/var/run/guix-build-coordinator/pid" + ;; Allow time for migrations to run + #:pid-file-timeout 60 + #:environment-variables + `(,(string-append + "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale") + "LC_ALL=en_US.utf8") + #:log-file "/var/log/guix-build-coordinator/coordinator.log")) + (stop #~(make-kill-destructor)))))) + +(define (guix-build-coordinator-activation config) + #~(begin + (use-modules (guix build utils)) + + (define %user (getpw "guix-build-coordinator")) + + (chmod "/var/lib/guix-build-coordinator" #o755) + + (mkdir-p "/var/log/guix-build-coordinator") + + ;; Allow writing the PID file + (mkdir-p "/var/run/guix-build-coordinator") + (chown "/var/run/guix-build-coordinator" + (passwd:uid %user) + (passwd:gid %user)))) + +(define (guix-build-coordinator-account config) + (match-record config + (user group) + (list (user-group + (name group) + (system? #t)) + (user-account + (name user) + (group group) + (system? #t) + (comment "Guix Build Coordinator user") + (home-directory "/var/lib/guix-build-coordinator") + (shell (file-append shadow "/sbin/nologin")))))) + +(define guix-build-coordinator-service-type + (service-type + (name 'guix-build-coordinator) + (extensions + (list + (service-extension shepherd-root-service-type + guix-build-coordinator-shepherd-services) + (service-extension activation-service-type + guix-build-coordinator-activation) + (service-extension account-service-type + guix-build-coordinator-account))) + (default-value + (guix-build-coordinator-configuration)) + (description + "Run an instance of the Guix Build Coordinator."))) + + +;;; +;;; Guix Data Service +;;; + (define-record-type* guix-data-service-configuration make-guix-data-service-configuration guix-data-service-configuration? diff --git a/gnu/tests/guix.scm b/gnu/tests/guix.scm index 6139e31cf0..20b67d55d3 100644 --- a/gnu/tests/guix.scm +++ b/gnu/tests/guix.scm @@ -35,7 +35,80 @@ #:use-module (guix store) #:use-module (guix utils) #:use-module (ice-9 match) - #:export (%test-guix-data-service)) + #:export (%test-guix-build-coordinator + %test-guix-data-service)) + +;;; +;;; Guix Build Coordinator +;;; + +(define %guix-build-coordinator-os + (simple-operating-system + (service dhcp-client-service-type) + (service guix-build-coordinator-service-type))) + +(define (run-guix-build-coordinator-test) + (define os + (marionette-operating-system + %guix-build-coordinator-os + #:imported-modules '((gnu services herd) + (guix combinators)))) + + (define forwarded-port 8745) + + (define vm + (virtual-machine + (operating-system os) + (memory-size 1024) + (port-forwardings `((,forwarded-port . 8745))))) + + (define test + (with-imported-modules '((gnu build marionette)) + #~(begin + (use-modules (srfi srfi-11) (srfi srfi-64) + (gnu build marionette) + (web uri) + (web client) + (web response)) + + (define marionette + (make-marionette (list #$vm))) + + (mkdir #$output) + (chdir #$output) + + (test-begin "guix-build-coordinator") + + (test-assert "service running" + (marionette-eval + '(begin + (use-modules (gnu services herd)) + (match (start-service 'guix-build-coordinator) + (#f #f) + (('service response-parts ...) + (match (assq-ref response-parts 'running) + ((pid) (number? pid)))))) + marionette)) + + (test-equal "http-get" + 200 + (let-values + (((response text) + (http-get #$(simple-format + #f "http://localhost:~A/metrics" forwarded-port) + #:decode-body? #t))) + (response-code response))) + + (test-end) + (exit (= (test-runner-fail-count (test-runner-current)) 0))))) + + (gexp->derivation "guix-build-coordinator-test" test)) + +(define %test-guix-build-coordinator + (system-test + (name "guix-build-coordinator") + (description "Connect to a running Guix Build Coordinator.") + (value (run-guix-build-coordinator-test)))) ;;; -- cgit 1.4.1 From 98ab323a246df47610652eca81494c777b50c251 Mon Sep 17 00:00:00 2001 From: Christopher Baines Date: Thu, 17 Sep 2020 18:49:45 +0100 Subject: services: guix: Add guix-build-coordinator-agent-service-type. * gnu/services/guix.scm (): New record type. (guix-build-coordinator-agent-configuration, guix-build-coordinator-agent-configuration?, guix-build-coordinator-agent-configuration-package, guix-build-coordinator-agent-configuration-user, guix-build-coordinator-agent-configuration-coordinator, guix-build-coordinator-agent-configuration-uuid), guix-build-coordinator-agent-configuration-password, guix-build-coordinator-agent-configuration-password-file, guix-build-coordinator-agent-configuration-systems, guix-build-coordinator-agent-configuration-max-parallel-builds, guix-build-coordinator-agent-configuration-derivation-substitute-urls, guix-build-coordinator-agent-configuration-non-derivation-substitute-urls, guix-build-coordinator-agent-shepherd-services, guix-build-coordinator-agent-activation, guix-build-coordinator-agent-account): New procedures. (guix-build-coordinator-agent-service-type): New variable. * doc/guix.texi (Guix Services): Document it. --- doc/guix.texi | 50 +++++++++++++++++++++ gnu/services/guix.scm | 121 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 171 insertions(+) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 3f1de559e6..a35b718a88 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -27831,6 +27831,56 @@ The Guile package with which to run the Guix Build Coordinator. @end table @end deftp +@defvar {Scheme Variable} guix-build-coordinator-agent-service-type +Service type for a Guix Build Coordinator agent. Its value must be a +@code{guix-build-coordinator-agent-configuration} object. +@end defvar + +@deftp {Data Type} guix-build-coordinator-agent-configuration +Data type representing the configuration a Guix Build Coordinator agent. + +@table @asis +@item @code{package} (default: @code{guix-build-coordinator}) +The Guix Build Coordinator package to use. + +@item @code{user} (default: @code{"guix-build-coordinator-agent"}) +The system user to run the service as. + +@item @code{coordinator} (default: @code{"http://localhost:8745"}) +The URI to use when connecting to the coordinator. + +@item @code{uuid} +The UUID of the agent. This should be generated by the coordinator +process, stored in the coordinator database, and used by the intended +agent. + +@item @code{password} (default: @code{#f}) +The password to use when connecting to the coordinator. A file to read +the password from can also be specified, and this is more secure. + +@item @code{password-file} (default: @code{#f}) +A file containing the password to use when connecting to the +coordinator. + +@item @code{systems} (default: @var{#f}) +The systems for which this agent should fetch builds. The agent process +will use the current system it's running on as the default. + +@item @code{max-parallel-builds} (default: @code{1}) +The number of builds to perform in parallel. + +@item @code{derivation-substitute-urls} (default: @code{1}) +URLs from which to attempt to fetch substitutes for derivations, if the +derivations aren't already available. + +@item @code{non-derivation-substitute-urls} (default: @code{1}) +URLs from which to attempt to fetch substitutes for build inputs, if the +input store items aren't already available. + +@end table +@end deftp + + @subsubheading Guix Data Service The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores and provides data about GNU Guix. This includes information about diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm index 1bacd61190..e4f5cf57b9 100644 --- a/gnu/services/guix.scm +++ b/gnu/services/guix.scm @@ -50,6 +50,21 @@ guix-build-coordinator-service-type + guix-build-coordinator-agent-configuration + guix-build-coordinator-agent-configuration? + guix-build-coordinator-agent-configuration-package + guix-build-coordinator-agent-configuration-user + guix-build-coordinator-agent-configuration-coordinator + guix-build-coordinator-agent-configuration-uuid + guix-build-coordinator-agent-configuration-password + guix-build-coordinator-agent-configuration-password-file + guix-build-coordinator-agent-configuration-systems + guix-build-coordinator-agent-configuration-max-parallel-builds + guix-build-coordinator-agent-configuration-derivation-substitute-urls + guix-build-coordinator-agent-configuration-non-derivation-substitute-urls + + guix-build-coordinator-agent-service-type + guix-data-service-configuration guix-data-service-configuration? @@ -95,6 +110,33 @@ (guile guix-build-coordinator-configuration-guile (default guile-3.0-latest))) +(define-record-type* + guix-build-coordinator-agent-configuration + make-guix-build-coordinator-agent-configuration + guix-build-coordinator-agent-configuration? + (package guix-build-coordinator-agent-configuration-package + (default guix-build-coordinator)) + (user guix-build-coordinator-agent-configuration-user + (default "guix-build-coordinator-agent")) + (coordinator guix-build-coordinator-agent-configuration-coordinator + (default "http://localhost:8745")) + (uuid guix-build-coordinator-agent-configuration-uuid) + (password guix-build-coordinator-agent-configuration-password + (default #f)) + (password-file guix-build-coordinator-agent-configuration-password-file + (default #f)) + (systems guix-build-coordinator-agent-configuration-systems + (default #f)) + (max-parallel-builds + guix-build-coordinator-agent-configuration-max-parallel-builds + (default 1)) + (derivation-substitute-urls + guix-build-coordinator-agent-configuration-derivation-substitute-urls + (default #f)) + (non-derivation-substitute-urls + guix-build-coordinator-agent-configuration-non-derivation-substitute-urls + (default #f))) + (define* (make-guix-build-coordinator-start-script database-uri-string allocation-strategy pid-file @@ -239,6 +281,85 @@ (description "Run an instance of the Guix Build Coordinator."))) +(define (guix-build-coordinator-agent-shepherd-services config) + (match-record config + (package user coordinator uuid password password-file max-parallel-builds + derivation-substitute-urls non-derivation-substitute-urls + systems) + (list + (shepherd-service + (documentation "Guix Build Coordinator Agent") + (provision '(guix-build-coordinator-agent)) + (requirement '(networking)) + (start #~(make-forkexec-constructor + (list #$(file-append package "/bin/guix-build-coordinator-agent") + #$(string-append "--coordinator=" coordinator) + #$(string-append "--uuid=" uuid) + #$@(if password + #~(#$(string-append "--password=" password)) + #~()) + #$@(if password-file + #~(#$(string-append "--password-file=" password-file)) + #~()) + #$(simple-format #f "--max-parallel-builds=~A" + max-parallel-builds) + #$@(if derivation-substitute-urls + #~(#$(string-append + "--derivation-substitute-urls=" + (string-join derivation-substitute-urls " "))) + #~()) + #$@(if non-derivation-substitute-urls + #~(#$(string-append + "--non-derivation-substitute-urls=" + (string-join derivation-substitute-urls " "))) + #~()) + #$@(map (lambda (system) + (string-append "--system=" system)) + (or systems '()))) + #:user #$user + #:pid-file "/var/run/guix-build-coordinator-agent/pid" + #:environment-variables + `(,(string-append + "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale") + "LC_ALL=en_US.utf8") + #:log-file "/var/log/guix-build-coordinator/agent.log")) + (stop #~(make-kill-destructor)))))) + +(define (guix-build-coordinator-agent-activation config) + #~(begin + (use-modules (guix build utils)) + + (mkdir-p "/var/log/guix-build-coordinator") + + ;; Allow writing the PID file + (mkdir-p "/var/run/guix-build-coordinator-agent") + (chown "/var/run/guix-build-coordinator-agent" + (passwd:uid %user) + (passwd:gid %user)))) + +(define (guix-build-coordinator-agent-account config) + (list (user-account + (name (guix-build-coordinator-agent-configuration-user config)) + (group "nogroup") + (system? #t) + (comment "Guix Build Coordinator agent user") + (home-directory "/var/empty") + (shell (file-append shadow "/sbin/nologin"))))) + +(define guix-build-coordinator-agent-service-type + (service-type + (name 'guix-build-coordinator-agent) + (extensions + (list + (service-extension shepherd-root-service-type + guix-build-coordinator-agent-shepherd-services) + (service-extension activation-service-type + guix-build-coordinator-agent-activation) + (service-extension account-service-type + guix-build-coordinator-agent-account))) + (description + "Run a Guix Build Coordinator agent."))) + ;;; ;;; Guix Data Service -- cgit 1.4.1 From c14714cbbfe9239410aec2ed73282192a64fbbdc Mon Sep 17 00:00:00 2001 From: Christopher Baines Date: Fri, 18 Sep 2020 14:21:13 +0100 Subject: services: guix: Add guix-build-coordinator-queue-builds-service-type. * gnu/services/guix.scm (): New record type. (guix-build-coordinator-queue-builds-configuration, guix-build-coordinator-queue-builds-configuration?, guix-build-coordinator-queue-builds-configuration-package, guix-build-coordinator-queue-builds-configuration-user, guix-build-coordinator-queue-builds-coordinator, guix-build-coordinator-queue-builds-configuration-systems, guix-build-coordinator-queue-builds-configuration-system-and-targets, guix-build-coordinator-queue-builds-configuration-guix-data-service, guix-build-coordinator-queue-builds-configuration-processed-commits-file, guix-build-coordinator-queue-builds-shepherd-services, guix-build-coordinator-queue-builds-activation, guix-build-coordinator-queue-builds-account): New procedures. (guix-build-coordinator-queue-builds-service-type): New variable. * doc/guix.texi (Guix Services): Document it. --- doc/guix.texi | 45 ++++++++++++++++++++ gnu/services/guix.scm | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 157 insertions(+) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index a35b718a88..eed7bbec6e 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -27880,6 +27880,51 @@ input store items aren't already available. @end table @end deftp +The Guix Build Coordinator package contains a script to query an +instance of the Guix Data Service for derivations to build, and then +submit builds for those derivations to the coordinator. The service +type below assists in running this script. This is an additional tool +that may be useful when building derivations contained within an +instance of the Guix Data Service. + +@defvar {Scheme Variable} guix-build-coordinator-queue-builds-service-type +Service type for the +guix-build-coordinator-queue-builds-from-guix-data-service script. Its +value must be a @code{guix-build-coordinator-queue-builds-configuration} +object. +@end defvar + +@deftp {Data Type} guix-build-coordinator-queue-builds-configuration +Data type representing the options to the queue builds from guix data +service script. + +@table @asis +@item @code{package} (default: @code{guix-build-coordinator}) +The Guix Build Coordinator package to use. + +@item @code{user} (default: @code{"guix-build-coordinator-queue-builds"}) +The system user to run the service as. + +@item @code{coordinator} (default: @code{"http://localhost:8745"}) +The URI to use when connecting to the coordinator. + +@item @code{systems} (default: @code{#f}) +The systems for which to fetch derivations to build. + +@item @code{systems-and-targets} (default: @code{#f}) +An association list of system and target pairs for which to fetch +derivations to build. + +@item @code{guix-data-service} (default: @code{"https://data.guix.gnu.org"}) +The Guix Data Service instance from which to query to find out about +derivations to build. + +@item @code{processed-commits-file} (default: @code{"/var/cache/guix-build-coordinator-queue-builds/processed-commits"}) +A file to record which commits have been processed, to avoid needlessly +processing them again if the service is restarted. + +@end table +@end deftp @subsubheading Guix Data Service The @uref{http://data.guix.gnu.org,Guix Data Service} processes, stores diff --git a/gnu/services/guix.scm b/gnu/services/guix.scm index e4f5cf57b9..2b1b69589e 100644 --- a/gnu/services/guix.scm +++ b/gnu/services/guix.scm @@ -65,6 +65,18 @@ guix-build-coordinator-agent-service-type + guix-build-coordinator-queue-builds-configuration + guix-build-coordinator-queue-builds-configuration? + guix-build-coordinator-queue-builds-configuration-package + guix-build-coordinator-queue-builds-configuration-user + guix-build-coordinator-queue-builds-coordinator + guix-build-coordinator-queue-builds-configuration-systems + guix-build-coordinator-queue-builds-configuration-system-and-targets + guix-build-coordinator-queue-builds-configuration-guix-data-service + guix-build-coordinator-queue-builds-configuration-processed-commits-file + + guix-build-coordinator-queue-builds-service-type + guix-data-service-configuration guix-data-service-configuration? @@ -137,6 +149,28 @@ guix-build-coordinator-agent-configuration-non-derivation-substitute-urls (default #f))) +(define-record-type* + guix-build-coordinator-queue-builds-configuration + make-guix-build-coordinator-queue-builds-configuration + guix-build-coordinator-queue-builds-configuration? + (package guix-build-coordinator-queue-builds-configuration-package + (default guix-build-coordinator)) + (user guix-build-coordinator-queue-builds-configuration-user + (default "guix-build-coordinator-queue-builds")) + (coordinator guix-build-coordinator-queue-builds-coordinator + (default "http://localhost:8745")) + (systems guix-build-coordinator-queue-builds-configuration-systems + (default #f)) + (systems-and-targets + guix-build-coordinator-queue-builds-configuration-system-and-targets + (default #f)) + (guix-data-service + guix-build-coordinator-queue-builds-configuration-guix-data-service + (default "https://data.guix.gnu.org")) + (processed-commits-file + guix-build-coordinator-queue-builds-configuration-processed-commits-file + (default "/var/cache/guix-build-coordinator-queue-builds/processed-commits"))) + (define* (make-guix-build-coordinator-start-script database-uri-string allocation-strategy pid-file @@ -360,6 +394,84 @@ (description "Run a Guix Build Coordinator agent."))) +(define (guix-build-coordinator-queue-builds-shepherd-services config) + (match-record config + (package user coordinator systems systems-and-targets + guix-data-service processed-commits-file) + (list + (shepherd-service + (documentation "Guix Build Coordinator queue builds from Guix Data Service") + (provision '(guix-build-coordinator-queue-builds)) + (requirement '(networking)) + (start + #~(make-forkexec-constructor + (list + #$(file-append + package + "/bin/guix-build-coordinator-queue-builds-from-guix-data-service") + #$(string-append "--coordinator=" coordinator) + #$@(map (lambda (system) + (string-append "--system=" system)) + (or systems '())) + #$@(map (match-lambda + ((system . target) + (string-append "--system-and-target=" system "=" target))) + (or systems-and-targets '())) + #$@(if guix-data-service + #~(#$(string-append "--guix-data-service=" guix-data-service)) + #~()) + #$@(if processed-commits-file + #~(#$(string-append "--processed-commits-file=" + processed-commits-file)) + #~())) + #:user #$user + #:pid-file "/var/run/guix-build-coordinator-queue-builds/pid" + #:environment-variables + `(,(string-append + "GUIX_LOCPATH=" #$glibc-utf8-locales "/lib/locale") + "LC_ALL=en_US.utf8") + #:log-file "/var/log/guix-build-coordinator/queue-builds.log")) + (stop #~(make-kill-destructor)))))) + +(define (guix-build-coordinator-queue-builds-activation config) + #~(begin + (use-modules (guix build utils)) + + (mkdir-p "/var/log/guix-build-coordinator") + + ;; Allow writing the PID file + (mkdir-p "/var/run/guix-build-coordinator-queue-builds") + (chown "/var/run/guix-build-coordinator-queue-builds" + (passwd:uid %user) + (passwd:gid %user)))) + +(define (guix-build-coordinator-queue-builds-account config) + (list (user-account + (name (guix-build-coordinator-queue-builds-configuration-user config)) + (group "nogroup") + (system? #t) + (comment "Guix Build Coordinator queue-builds user") + (home-directory "/var/empty") + (shell (file-append shadow "/sbin/nologin"))))) + +(define guix-build-coordinator-queue-builds-service-type + (service-type + (name 'guix-build-coordinator-queue-builds) + (extensions + (list + (service-extension shepherd-root-service-type + guix-build-coordinator-queue-builds-shepherd-services) + (service-extension activation-service-type + guix-build-coordinator-queue-builds-activation) + (service-extension account-service-type + guix-build-coordinator-queue-builds-account))) + (description + "Run the guix-build-coordinator-queue-builds-from-guix-data-service +script. + +This is a script to assist in having the Guix Build Coordinator build +derivations stored in an instance of the Guix Data Service."))) + ;;; ;;; Guix Data Service -- cgit 1.4.1 From 34d1c0a03b51fdaef26a3bc630ab096da740e1d6 Mon Sep 17 00:00:00 2001 From: Danny Milosavljevic Date: Sun, 27 Sep 2020 21:06:30 +0200 Subject: services: dnsmasq: Add TFTP configuration fields. * gnu/services/dns.scm (): Add TFTP configuration fields. (dnsmasq-shepherd-service): Use them. * doc/guix.texi (DNS Services): Document them. --- doc/guix.texi | 54 ++++++++++++++++++++++++++++++++++++++++ gnu/services/dns.scm | 70 +++++++++++++++++++++++++++++++++++++++++++++++++--- 2 files changed, 121 insertions(+), 3 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index eed7bbec6e..2b1ffbc55f 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -23363,6 +23363,60 @@ disables caching. @item @code{negative-cache?} (default: @code{#t}) When false, disable negative caching. +@item @code{tftp-enable?} (default: @code{#f}) +Whether to enable the built-in TFTP server. + +@item @code{tftp-no-fail?} (default: @code{#f}) +If true, does not fail dnsmasq if the TFTP server could not start up. + +@item @code{tftp-single-port?} (default: @code{#f}) +Whether to use only one single port for TFTP. + +@item @code{tftp-secure?} (default: @code{#f}) +If true, only files owned by the user running the dnsmasq process are accessible. + +If dnsmasq is being run as root, different rules apply: +@code{tftp-secure?} has no effect, but only files which have the +world-readable bit set are accessible. + +@item @code{tftp-max} (default: @code{#f}) +If set, sets the maximal number of concurrent connections allowed. + +@item @code{tftp-mtu} (default: @code{#f}) +If set, sets the MTU for TFTP packets to that value. + +@item @code{tftp-no-blocksize?} (default: @code{#f}) +If true, stops the TFTP server from negotiating the blocksize with a client. + +@item @code{tftp-lowercase?} (default: @code{#f}) +Whether to convert all filenames in TFTP requests to lowercase. + +@item @code{tftp-port-range} (default: @code{#f}) +If set, fixes the dynamical ports (one per client) to the given range +(@code{","}). + +@item @code{tftp-root} (default: @code{/var/empty,lo}) +Look for files to transfer using TFTP relative to the given directory. +When this is set, TFTP paths which include ".." are rejected, to stop clients +getting outside the specified root. Absolute paths (starting with /) are +allowed, but they must be within the tftp-root. If the optional interface +argument is given, the directory is only used for TFTP requests via that +interface. + +@item @code{tftp-unique-root} (default: @code{#f}) +If set, add the IP or hardware address of the TFTP client as a path component +on the end of the TFTP-root. Only valid if a TFTP root is set and the +directory exists. Defaults to adding IP address (in standard dotted-quad +format). + +For instance, if --tftp-root is "/tftp" and client 1.2.3.4 requests file +"myfile" then the effective path will be "/tftp/1.2.3.4/myfile" if +/tftp/1.2.3.4 exists or /tftp/myfile otherwise. When "=mac" is specified +it will append the MAC address instead, using lowercase zero padded digits +separated by dashes, e.g.: 01-02-03-04-aa-bb Note that resolving MAC +addresses is only possible if the client is in the local network or obtained +a DHCP lease from dnsmasq. + @end table @end deftp diff --git a/gnu/services/dns.scm b/gnu/services/dns.scm index 9caa3611be..572880561c 100644 --- a/gnu/services/dns.scm +++ b/gnu/services/dns.scm @@ -757,7 +757,29 @@ cache.size = 100 * MB (cache-size dnsmasq-configuration-cache-size (default 150)) ;integer (negative-cache? dnsmasq-configuration-negative-cache? - (default #t))) ;boolean + (default #t)) ;boolean + (tftp-enable? dnsmasq-configuration-tftp-enable? + (default #f)) ;boolean + (tftp-no-fail? dnsmasq-configuration-tftp-no-fail? + (default #f)) ;boolean + (tftp-single-port? dnsmasq-configuration-tftp-single-port? + (default #f)) ;boolean + (tftp-secure? dnsmasq-tftp-secure? + (default #f)) ;boolean + (tftp-max dnsmasq-tftp-max + (default #f)) ;integer + (tftp-mtu dnsmasq-tftp-mtu + (default #f)) ;integer + (tftp-no-blocksize? dnsmasq-tftp-no-blocksize? + (default #f)) ;boolean + (tftp-lowercase? dnsmasq-tftp-lowercase? + (default #f)) ;boolean + (tftp-port-range dnsmasq-tftp-port-range + (default #f)) ;string + (tftp-root dnsmasq-tftp-root + (default "/var/empty,lo")) ;string + (tftp-unique-root dnsmasq-tftp-unique-root + (default #f))) ;"" or "ip" or "mac" (define dnsmasq-shepherd-service (match-lambda @@ -765,7 +787,12 @@ cache.size = 100 * MB no-hosts? port local-service? listen-addresses resolv-file no-resolv? servers - addresses cache-size negative-cache?) + addresses cache-size negative-cache? + tftp-enable? tftp-no-fail? + tftp-single-port? tftp-secure? + tftp-max tftp-mtu tftp-no-blocksize? + tftp-lowercase? tftp-port-range + tftp-root tftp-unique-root) (shepherd-service (provision '(dnsmasq)) (requirement '(networking)) @@ -794,7 +821,44 @@ cache.size = 100 * MB #$(format #f "--cache-size=~a" cache-size) #$@(if negative-cache? '() - '("--no-negcache"))) + '("--no-negcache")) + #$@(if tftp-enable? + '("--enable-tftp") + '()) + #$@(if tftp-no-fail? + '("--tftp-no-fail") + '()) + #$@(if tftp-single-port? + '("--tftp-single-port") + '()) + #$@(if tftp-secure? + '("--tftp-secure?") + '()) + #$@(if tftp-max + (list (format #f "--tftp-max=~a" tftp-max)) + '()) + #$@(if tftp-mtu + (list (format #f "--tftp-mtu=~a" tftp-mtu)) + '()) + #$@(if tftp-no-blocksize? + '("--tftp-no-blocksize") + '()) + #$@(if tftp-lowercase? + '("--tftp-lowercase") + '()) + #$@(if tftp-port-range + (list (format #f "--tftp-port-range=~a" + tftp-port-range)) + '()) + #$@(if tftp-root + (list (format #f "--tftp-root=~a" tftp-root)) + '()) + #$@(if tftp-unique-root + (list + (if (> (length tftp-unique-root) 0) + (format #f "--tftp-unique-root=~a" tftp-unique-root) + (format #f "--tftp-unique-root"))) + '())) #:pid-file "/run/dnsmasq.pid")) (stop #~(make-kill-destructor)))))) -- cgit 1.4.1 From e463ee4c7bd44a7fdc372972fbc1d1ae8140f31b Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Thu, 8 Oct 2020 09:32:40 +0200 Subject: doc: Improve choice of options in service examples. * doc/guix.texi (Using the Configuration System): In example, set 'substitute-urls' rather than 'use-substitutes?', and set 'auto-login' for 'mingetty-configuration'. (Service Types and Services): In example, show 'extra-options' rather than 'use-substitutes?'. --- doc/guix.texi | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 2b1ffbc55f..3fc76c8670 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -12062,11 +12062,15 @@ following in your operating system declaration: (guix-service-type config => (guix-configuration (inherit config) - (use-substitutes? #f) - (extra-options '("--gc-keep-derivations")))) + ;; Fetch substitutes from example.org. + (substitute-urls + (list "https://example.org/guix" + "https://ci.guix.gnu.org")))) (mingetty-service-type config => (mingetty-configuration - (inherit config))))) + (inherit config) + ;; Automatially log in as "guest". + (auto-login "guest"))))) (operating-system ;; @dots{} @@ -30431,7 +30435,7 @@ A service of this type is instantiated like this: (service guix-service-type (guix-configuration (build-accounts 5) - (use-substitutes? #f))) + (extra-options '("--gc-keep-derivations")))) @end lisp The second argument to the @code{service} form is a value representing -- cgit 1.4.1 From 6f406e5d15ce477664dbf5c443c0db59ea3a06ec Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Thu, 8 Oct 2020 09:42:54 +0200 Subject: doc: Developers don't need to run "make install" in Guix. Suggested by Emmanuel Agullo. * doc/contributing.texi (Building from Git): Mention that running "make install" is unnecessary. --- doc/contributing.texi | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) (limited to 'doc') diff --git a/doc/contributing.texi b/doc/contributing.texi index 535cfc2a6f..c6b0ca523d 100644 --- a/doc/contributing.texi +++ b/doc/contributing.texi @@ -121,8 +121,9 @@ more information. Then, run @command{./configure} as usual. Make sure to pass @code{--localstatedir=@var{directory}} where @var{directory} is the @code{localstatedir} value used by your current installation (@pxref{The -Store}, for information about this). We recommend to use the value -@code{/var}. +Store}, for information about this), usually @file{/var}. Note that you +will probably not run @command{make install} at the end (you don't have +to) but it's still important to pass the right @code{localstatedir}. Finally, you have to invoke @code{make check} to run tests (@pxref{Running the Test Suite}). If anything -- cgit 1.4.1 From 9022861dc028e99fab930721fe991a682c497bbb Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Thu, 8 Oct 2020 09:57:12 +0200 Subject: doc: Clarify that guix-daemon doesn't have to be launched from the checkout. Suggested by Emmanuel Agullo. * doc/contributing.texi (Running Guix Before It Is Installed): Move the "./pre-inst-env guix-daemon" snippet at the bottom. Make it clear that it's necessary only when hacking on guix-daemon or when guix-daemon is not already running. --- doc/contributing.texi | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) (limited to 'doc') diff --git a/doc/contributing.texi b/doc/contributing.texi index c6b0ca523d..af3601442e 100644 --- a/doc/contributing.texi +++ b/doc/contributing.texi @@ -160,13 +160,11 @@ with all the dependencies available (@pxref{Building from Git}), and then simply prefix each command with @command{./pre-inst-env} (the @file{pre-inst-env} script lives in the top build tree of Guix; it is generated by @command{./configure}). -An example@footnote{The @option{-E} flag to -@command{sudo} guarantees that @code{GUILE_LOAD_PATH} is correctly set -such that @command{guix-daemon} and the tools it uses can find the Guile -modules they need.}: +As an example, here is how you would build the @code{hello} package as +defined in your working tree (this assumes @command{guix-daemon} is +already running on your system; it's OK if it's a different version): @example -$ sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild $ ./pre-inst-env guix build hello @end example @@ -201,6 +199,17 @@ scheme@@(guile-user)> (length snakes) $1 = 361 @end example +If you are hacking on the daemon and its supporting code or if +@command{guix-daemon} is not already running on your system, you can +launch it straight from the build tree@footnote{The @option{-E} flag to +@command{sudo} guarantees that @code{GUILE_LOAD_PATH} is correctly set +such that @command{guix-daemon} and the tools it uses can find the Guile +modules they need.}: + +@example +$ sudo -E ./pre-inst-env guix-daemon --build-users-group=guixbuild +@end example + The @command{pre-inst-env} script sets up all the environment variables necessary to support this, including @env{PATH} and @env{GUILE_LOAD_PATH}. -- cgit 1.4.1 From efbf5fdd01817ea75de369e3dd2761a85f8f7dd5 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Sat, 3 Oct 2020 01:17:54 -0400 Subject: offload: Improve load normalization and configurability. Fixes . The computed normalized load was previously obtained by dividing the load average as found in /proc/loadavg by the number of parallel builds defined for a build machine. This normalized load didn't allow to compare machines with different number of cores, as the load average reported by /proc/loadavg can be as high as the number of cores; thus comparing that value to a fixed threshold of 2.0 would mean machines with multiple cores were more likely to be flagged as overloaded compared to single core machines. This can be fixed by normalizing using the available number of cores instead of the number of parallel jobs. * guix/scripts/offload.scm ()[overload-threshold]: New field. (node-load): Modify to return a normalized load value between 0 and 1, taking into account the number of cores available. (normalized-load): Remove procedure. (report-load): New procedure. (choose-build-machine): Adjust to use the modified 'node-load' and the new 'report-load' and 'build-machine-overload-threshold' procedures. (check-machine-status): Adjust. * doc/guix.texi (Daemon Offload Setup): Document the offload scheduler and the new 'overload-threshold' field. --- doc/guix.texi | 30 ++++++++++++++++++++++++++- guix/scripts/offload.scm | 54 +++++++++++++++++++++++++++++------------------- 2 files changed, 62 insertions(+), 22 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 3fc76c8670..553a3b8ae9 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -1081,7 +1081,28 @@ architecture natively supports it, via emulation (@pxref{Transparent Emulation with QEMU}), or both. Missing prerequisites for the build are copied over SSH to the target machine, which then proceeds with the build; upon success the output(s) of the build are copied back to the -initial machine. +initial machine. The offload facility comes with a basic scheduler that +attempts to select the best machine. The best machine is chosen among +the available machines based on criteria such as: + +@enumerate +@item +The availability of a build slot. A build machine can have as many +build slots (connections) as the value of the @code{parallel-builds} +field of its @code{build-machine} object. + +@item +Its relative speed, as defined via the @code{speed} field of its +@code{build-machine} object. + +@item +Its load. The normalized machine load must be lower than a threshold +value, configurable via the @code{overload-threshold} field of its +@code{build-machine} object. + +@item +Disk space availability. More than a 100 MiB must be available. +@end enumerate The @file{/etc/guix/machines.scm} file typically looks like this: @@ -1185,6 +1206,13 @@ when transferring files to and from build machines. File name of the Unix-domain socket @command{guix-daemon} is listening to on that machine. +@item @code{overload-threshold} (default: @code{0.6}) +The load threshold above which a potential offload machine is +disregarded by the offload scheduler. The value roughly translates to +the total processor usage of the build machine, ranging from 0.0 (0%) to +1.0 (100%). It can also be disabled by setting +@code{overload-threshold} to @code{#f}. + @item @code{parallel-builds} (default: @code{1}) The number of builds that may run in parallel on the machine. diff --git a/guix/scripts/offload.scm b/guix/scripts/offload.scm index 3dc8ccefcb..a5fe98b675 100644 --- a/guix/scripts/offload.scm +++ b/guix/scripts/offload.scm @@ -88,6 +88,10 @@ (default 3)) (daemon-socket build-machine-daemon-socket ; string (default "/var/guix/daemon-socket/socket")) + ;; A #f value tells the offload scheduler to disregard the load of the build + ;; machine when selecting the best offload machine. + (overload-threshold build-machine-overload-threshold ; inexact real between + (default 0.6)) ; 0.0 and 1.0 | #f (parallel-builds build-machine-parallel-builds ; number (default 1)) (speed build-machine-speed ; inexact real @@ -391,30 +395,34 @@ of free disk space on '~a'~%") (* 100 (expt 2 20))) ;100 MiB (define (node-load node) - "Return the load on NODE. Return +∞ if NODE is misbehaving." + "Return the load on NODE, a normalized value between 0.0 and 1.0. The value +is derived from /proc/loadavg and normalized according to the number of +logical cores available, to give a rough estimation of CPU usage. Return +1.0 (fully loaded) if NODE is misbehaving." (let ((line (inferior-eval '(begin (use-modules (ice-9 rdelim)) (call-with-input-file "/proc/loadavg" read-string)) - node))) - (if (eof-object? line) - +inf.0 ;MACHINE does not respond, so assume it is infinitely loaded + node)) + (ncores (inferior-eval '(begin + (use-modules (ice-9 threads)) + (current-processor-count)) + node))) + (if (or (eof-object? line) (eof-object? ncores)) + 1.0 ;MACHINE does not respond, so assume it is fully loaded (match (string-tokenize line) ((one five fifteen . x) - (string->number one)) + (let ((load (/ (string->number one) ncores))) + (if (> load 1.0) + 1.0 + load))) (x - +inf.0))))) - -(define (normalized-load machine load) - "Divide LOAD by the number of parallel builds of MACHINE." - (if (rational? load) - (let* ((jobs (build-machine-parallel-builds machine)) - (normalized (/ load jobs))) - (format (current-error-port) "load on machine '~a' is ~s\ - (normalized: ~s)~%" - (build-machine-name machine) load normalized) - normalized) - load)) + 1.0))))) + +(define (report-load machine load) + (format (current-error-port) + "normalized load on machine '~a' is ~,2f~%" + (build-machine-name machine) load)) (define (random-seed) (logxor (getpid) (car (gettimeofday)))) @@ -472,11 +480,15 @@ slot (which must later be released with 'release-build-slot'), or #f and #f." (let* ((session (false-if-exception (open-ssh-session best %short-timeout))) (node (and session (remote-inferior session))) - (load (and node (normalized-load best (node-load node)))) + (load (and node (node-load node))) + (threshold (build-machine-overload-threshold best)) (space (and node (node-free-disk-space node)))) + (when load (report-load best load)) (when node (close-inferior node)) (when session (disconnect! session)) - (if (and node (< load 2.) (>= space %minimum-disk-space)) + (if (and node + (or (not threshold) (< load threshold)) + (>= space %minimum-disk-space)) (match others (((machines slots) ...) ;; Release slots from the uninteresting machines. @@ -708,13 +720,13 @@ machine." (free (node-free-disk-space inferior))) (close-inferior inferior) (format #t "~a~% kernel: ~a ~a~% architecture: ~a~%\ - host name: ~a~% normalized load: ~a~% free disk space: ~,2f MiB~%\ + host name: ~a~% normalized load: ~,2f~% free disk space: ~,2f MiB~%\ time difference: ~a s~%" (build-machine-name machine) (utsname:sysname uts) (utsname:release uts) (utsname:machine uts) (utsname:nodename uts) - (normalized-load machine load) + load (/ free (expt 2 20) 1.) (- time now)))))))) -- cgit 1.4.1 From 9c1b8390ba36c0b9e3d5580b0f00206255f9b9a8 Mon Sep 17 00:00:00 2001 From: Efraim Flashner Date: Fri, 9 Oct 2020 16:45:12 +0300 Subject: doc: Document missing openssh configure option. * doc/guix.texi (Networking Services): Document configuration option to specify the openssh package for the openssh-service-type. --- doc/guix.texi | 3 +++ 1 file changed, 3 insertions(+) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 553a3b8ae9..94afdffee3 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -15351,6 +15351,9 @@ example: This is the configuration record for OpenSSH's @command{sshd}. @table @asis +@item @code{openssh} (default @var{openssh}) +The Openssh package to use. + @item @code{pid-file} (default: @code{"/var/run/sshd.pid"}) Name of the file where @command{sshd} writes its PID. -- cgit 1.4.1 From 5918cb5341ecd76304eef2cb529e20896b56f4e5 Mon Sep 17 00:00:00 2001 From: Maxim Cournoyer Date: Fri, 9 Oct 2020 16:50:01 -0400 Subject: doc: Sync the client and daemon --rounds options documentation. Fixes . * doc/guix.texi (Common Build Options): Suggest using '--keep-failed' to keep differing build results in the store rather than 'guix archive'. Reported-by: Jonathan Brielmaier --- doc/guix.texi | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 94afdffee3..8514dfe86f 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -9155,11 +9155,9 @@ Non-deterministic build processes are a problem because they make it practically impossible for users to @emph{verify} whether third-party binaries are genuine. @xref{Invoking guix challenge}, for more. -Note that, currently, the differing build results are not kept around, -so you will have to manually investigate in case of an error---e.g., by -stashing one of the build results with @code{guix archive --export} -(@pxref{Invoking guix archive}), then rebuilding, and finally comparing -the two results. +When used in conjunction with @option{--keep-failed}, the differing +output is kept in the store, under @file{/gnu/store/@dots{}-check}. +This makes it easy to look for differences between the two results. @item --no-offload Do not use offload builds to other machines (@pxref{Daemon Offload -- cgit 1.4.1 From 46135ce4cefab9e164d75697d7ea0c8359b842e4 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Mon, 28 Sep 2020 17:36:42 +0200 Subject: packages: Add 'package-with-c-toolchain'. * guix/build-system.scm (build-system-with-c-toolchain): New procedure. * guix/packages.scm (package-with-c-toolchain): New procedure. * tests/packages.scm ("package-with-c-toolchain"): New test. * doc/guix.texi (package Reference): Document 'package-with-c-toolchain'. (Build Systems): Mention it. --- doc/guix.texi | 32 ++++++++++++++++++++++++++++++++ guix/build-system.scm | 35 +++++++++++++++++++++++++++++++++-- guix/packages.scm | 9 +++++++++ tests/packages.scm | 20 ++++++++++++++++++++ 4 files changed, 94 insertions(+), 2 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 8514dfe86f..e084144a82 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -6558,6 +6558,35 @@ cross-compiling: It is an error to refer to @code{this-package} outside a package definition. @end deffn +Because packages are regular Scheme objects that capture a complete +dependency graph and associated build procedures, it is often useful to +write procedures that take a package and return a modified version +thereof according to some parameters. Below are a few examples. + +@cindex tool chain, choosing a package's tool chain +@deffn {Scheme Procedure} package-with-c-toolchain @var{package} @var{toolchain} +Return a variant of @var{package} that uses @var{toolchain} instead of +the default GNU C/C++ toolchain. @var{toolchain} must be a list of +inputs (label/package tuples) providing equivalent functionality, such +as the @code{gcc-toolchain} package. + +The example below returns a variant of the @code{hello} package built +with GCC@tie{}10.x and the rest of the GNU tool chain (Binutils and the +GNU C Library) instead of the default tool chain: + +@lisp +(let ((toolchain (specification->package "gcc-toolchain@@10"))) + (package-with-c-toolchain hello `(("toolchain" ,toolchain)))) +@end lisp + +The build tool chain is part of the @dfn{implicit inputs} of +packages---it's usually not listed as part of the various ``inputs'' +fields and is instead pulled in by the build system. Consequently, this +procedure works by changing the build system of @var{package} so that it +pulls in @var{toolchain} instead of the defaults. @ref{Build Systems}, +for more on build systems. +@end deffn + @node origin Reference @subsection @code{origin} Reference @@ -6694,6 +6723,9 @@ ornamentation---in other words, a bag is a lower-level representation of a package, which includes all the inputs of that package, including some that were implicitly added by the build system. This intermediate representation is then compiled to a derivation (@pxref{Derivations}). +The @code{package-with-c-toolchain} is an example of a way to change the +implicit inputs that a package's build system pulls in (@pxref{package +Reference, @code{package-with-c-toolchain}}). Build systems accept an optional list of @dfn{arguments}. In package definitions, these are passed @i{via} the @code{arguments} field diff --git a/guix/build-system.scm b/guix/build-system.scm index 4174972b98..76d670995c 100644 --- a/guix/build-system.scm +++ b/guix/build-system.scm @@ -1,5 +1,5 @@ ;;; GNU Guix --- Functional package management for GNU -;;; Copyright © 2012, 2013, 2014 Ludovic Courtès +;;; Copyright © 2012, 2013, 2014, 2020 Ludovic Courtès ;;; ;;; This file is part of GNU Guix. ;;; @@ -18,6 +18,7 @@ (define-module (guix build-system) #:use-module (guix records) + #:use-module (srfi srfi-1) #:use-module (ice-9 match) #:export (build-system build-system? @@ -37,7 +38,9 @@ bag-arguments bag-build - make-bag)) + make-bag + + build-system-with-c-toolchain)) (define-record-type* build-system make-build-system build-system? @@ -98,3 +101,31 @@ intermediate representation just above derivations." #:outputs outputs #:target target arguments)))) + +(define (build-system-with-c-toolchain bs toolchain) + "Return a variant of BS, a build system, that uses TOOLCHAIN instead of the +default GNU C/C++ toolchain. TOOLCHAIN must be a list of +inputs (label/package tuples) providing equivalent functionality, such as the +'gcc-toolchain' package." + (define lower + (build-system-lower bs)) + + (define toolchain-packages + ;; These are the GNU toolchain packages pulled in by GNU-BUILD-SYSTEM and + ;; all the build systems that inherit from it. Keep the list in sync with + ;; 'standard-packages' in (guix build-system gnu). + '("gcc" "binutils" "libc" "libc:static" "ld-wrapper")) + + (define (lower* . args) + (let ((lowered (apply lower args))) + (bag + (inherit lowered) + (build-inputs + (append (fold alist-delete + (bag-build-inputs lowered) + toolchain-packages) + toolchain))))) + + (build-system + (inherit bs) + (lower lower*))) diff --git a/guix/packages.scm b/guix/packages.scm index 4f2bb432be..24d6417065 100644 --- a/guix/packages.scm +++ b/guix/packages.scm @@ -124,6 +124,7 @@ package-patched-vulnerabilities package-with-patches package-with-extra-patches + package-with-c-toolchain package/inherit transitive-input-references @@ -790,6 +791,14 @@ specifies modules in scope when evaluating SNIPPET." (append (origin-patches (package-source original)) patches))) +(define (package-with-c-toolchain package toolchain) + "Return a variant of PACKAGE that uses TOOLCHAIN instead of the default GNU +C/C++ toolchain. TOOLCHAIN must be a list of inputs (label/package tuples) +providing equivalent functionality, such as the 'gcc-toolchain' package." + (let ((bs (package-build-system package))) + (package/inherit package + (build-system (build-system-with-c-toolchain bs toolchain))))) + (define (transitive-inputs inputs) "Return the closure of INPUTS when considering the 'propagated-inputs' edges. Omit duplicate inputs, except for those already present in INPUTS diff --git a/tests/packages.scm b/tests/packages.scm index 5d5abcbd76..2d13d91344 100644 --- a/tests/packages.scm +++ b/tests/packages.scm @@ -1430,6 +1430,26 @@ (derivation-file-name (package-derivation %store coreutils)))))))) +(test-assert "package-with-c-toolchain" + (let* ((dep (dummy-package "chbouib" + (build-system gnu-build-system) + (native-inputs `(("x" ,grep))))) + (p0 (dummy-package "thingie" + (build-system gnu-build-system) + (inputs `(("foo" ,grep) + ("bar" ,dep))))) + (tc (dummy-package "my-toolchain")) + (p1 (package-with-c-toolchain p0 `(("toolchain" ,tc))))) + (define toolchain-packages + '("gcc" "binutils" "glibc" "ld-wrapper")) + + (match (bag-build-inputs (package->bag p1)) + ((("foo" foo) ("bar" bar) (_ (= package-name packages) . _) ...) + (and (not (any (cut member <> packages) toolchain-packages)) + (member "my-toolchain" packages) + (eq? foo grep) + (eq? bar dep)))))) + (test-equal "package-patched-vulnerabilities" '(("CVE-2015-1234") ("CVE-2016-1234" "CVE-2018-4567") -- cgit 1.4.1 From abd7a474615353149a44f4504f0b4b248dcc0716 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Mon, 28 Sep 2020 18:56:00 +0200 Subject: guix build: Add '--with-c-toolchain'. * guix/scripts/build.scm (package-dependents/spec) (package-toolchain-rewriting, transform-package-toolchain): New procedures. (%transformations): Add it. (%transformation-options, show-transformation-options-help): Add '--with-c-toolchain'. * tests/scripts-build.scm (depends-on-toolchain?): New procedure. ("options->transformation, with-c-toolchain") ("options->transformation, with-c-toolchain twice") New test. ("options->transformation, with-c-toolchain, no effect"): New tests. * doc/guix.texi (Package Transformation Options): Document it. --- doc/guix.texi | 38 ++++++++++++++++++++++ guix/scripts/build.scm | 84 +++++++++++++++++++++++++++++++++++++++++++++++++ tests/scripts-build.scm | 82 +++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 204 insertions(+) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index e084144a82..7150adeaa8 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -9364,6 +9364,44 @@ must be compatible. If @var{replacement} is somehow incompatible with @var{package}, then the resulting package may be unusable. Use with care! +@cindex tool chain, changing the build tool chain of a package +@item --with-c-toolchain=@var{package}=@var{toolchain} +This option changes the compilation of @var{package} and everything that +depends on it so that they get built with @var{toolchain} instead of the +default GNU tool chain for C/C++. + +Consider this example: + +@example +guix build octave-cli \ + --with-c-toolchain=fftw=gcc-toolchain@@10 \ + --with-c-toolchain=fftwf=gcc-toolchain@@10 +@end example + +The command above builds a variant of the @code{fftw} and @code{fftwf} +packages using version 10 of @code{gcc-toolchain} instead of the default +tool chain, and then builds a variant of the GNU@tie{}Octave +command-line interface using them. GNU@tie{}Octave itself is also built +with @code{gcc-toolchain@@10}. + +This other example builds the Hardware Locality (@code{hwloc}) library +and its dependents up to @code{intel-mpi-benchmarks} with the Clang C +compiler: + +@example +guix build --with-c-toolchain=hwloc=clang-toolchain \ + intel-mpi-benchmarks +@end example + +@quotation Note +There can be application binary interface (ABI) incompatibilities among +tool chains. This is particularly true of the C++ standard library and +run-time support libraries such as that of OpenMP. By rebuilding all +dependents with the same tool chain, @option{--with-c-toolchain} minimizes +the risks of incompatibility but cannot entirely eliminate them. Choose +@var{package} wisely. +@end quotation + @item --with-git-url=@var{package}=@var{url} @cindex Git, using the latest commit @cindex latest commit, building diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm index 72a5d46347..e59e0ee67f 100644 --- a/guix/scripts/build.scm +++ b/guix/scripts/build.scm @@ -26,6 +26,7 @@ #:use-module (guix store) #:use-module (guix derivations) #:use-module (guix packages) + #:use-module (guix memoization) #:use-module (guix grafts) #:use-module (guix utils) @@ -396,6 +397,83 @@ a checkout of the Git repository at the given URL." (rewrite obj) obj))) +(define (package-dependents/spec top bottom) + "Return the list of dependents of BOTTOM, a spec string, that are also +dependencies of TOP, a package." + (define-values (name version) + (package-name->name+version bottom)) + + (define dependent? + (mlambda (p) + (and (package? p) + (or (and (string=? name (package-name p)) + (or (not version) + (version-prefix? version (package-version p)))) + (match (bag-direct-inputs (package->bag p)) + (((labels dependencies . _) ...) + (any dependent? dependencies))))))) + + (filter dependent? (package-closure (list top)))) + +(define (package-toolchain-rewriting p bottom toolchain) + "Return a procedure that, when passed a package that's either BOTTOM or one +of its dependents up to P so, changes it so it is built with TOOLCHAIN. +TOOLCHAIN must be an input list." + (define rewriting-property + (gensym " package-toolchain-rewriting")) + + (match (package-dependents/spec p bottom) + (() ;P does not depend on BOTTOM + identity) + (set + ;; SET is the list of packages "between" P and BOTTOM (included) whose + ;; toolchain needs to be changed. + (package-mapping (lambda (p) + (if (or (assq rewriting-property + (package-properties p)) + (not (memq p set))) + p + (let ((p (package-with-c-toolchain p toolchain))) + (package/inherit p + (properties `((,rewriting-property . #t) + ,@(package-properties p))))))) + (lambda (p) + (or (assq rewriting-property (package-properties p)) + (not (memq p set)))) + #:deep? #t)))) + +(define (transform-package-toolchain replacement-specs) + "Return a procedure that, when passed a package, changes its toolchain or +that of its dependencies according to REPLACEMENT-SPECS. REPLACEMENT-SPECS is +a list of strings like \"fftw=gcc-toolchain@10\" meaning that the package to +the left of the equal sign must be built with the toolchain to the right of +the equal sign." + (define split-on-commas + (cute string-tokenize <> (char-set-complement (char-set #\,)))) + + (define (specification->input spec) + (let ((package (specification->package spec))) + (list (package-name package) package))) + + (define replacements + (map (lambda (spec) + (match (string-tokenize spec %not-equal) + ((spec (= split-on-commas toolchain)) + (cons spec (map specification->input toolchain))) + (_ + (leave (G_ "~a: invalid toolchain replacement specification~%") + spec)))) + replacement-specs)) + + (lambda (store obj) + (if (package? obj) + (or (any (match-lambda + ((bottom . toolchain) + ((package-toolchain-rewriting obj bottom toolchain) obj))) + replacements) + obj) + obj))) + (define (transform-package-tests specs) "Return a procedure that, when passed a package, sets #:tests? #f in its 'arguments' field." @@ -426,6 +504,7 @@ a checkout of the Git repository at the given URL." (with-branch . ,transform-package-source-branch) (with-commit . ,transform-package-source-commit) (with-git-url . ,transform-package-source-git-url) + (with-c-toolchain . ,transform-package-toolchain) (without-tests . ,transform-package-tests))) (define (transformation-procedure key) @@ -455,6 +534,8 @@ a checkout of the Git repository at the given URL." (parser 'with-commit)) (option '("with-git-url") #t #f (parser 'with-git-url)) + (option '("with-c-toolchain") #t #f + (parser 'with-c-toolchain)) (option '("without-tests") #t #f (parser 'without-tests))))) @@ -477,6 +558,9 @@ a checkout of the Git repository at the given URL." (display (G_ " --with-git-url=PACKAGE=URL build PACKAGE from the repository at URL")) + (display (G_ " + --with-c-toolchain=PACKAGE=TOOLCHAIN + build PACKAGE and its dependents with TOOLCHAIN")) (display (G_ " --without-tests=PACKAGE build PACKAGE without running its tests"))) diff --git a/tests/scripts-build.scm b/tests/scripts-build.scm index 5f91360953..6925374baa 100644 --- a/tests/scripts-build.scm +++ b/tests/scripts-build.scm @@ -22,6 +22,8 @@ #:use-module (guix derivations) #:use-module (guix packages) #:use-module (guix git-download) + #:use-module (guix build-system) + #:use-module (guix build-system gnu) #:use-module (guix scripts build) #:use-module (guix ui) #:use-module (guix utils) @@ -30,6 +32,8 @@ #:use-module (gnu packages base) #:use-module (gnu packages busybox) #:use-module (ice-9 match) + #:use-module (srfi srfi-1) + #:use-module (srfi srfi-26) #:use-module (srfi srfi-64)) @@ -270,6 +274,80 @@ ((("x" dep3)) (map package-source (list dep1 dep3)))))))))))) +(define* (depends-on-toolchain? p #:optional (toolchain "gcc-toolchain")) + "Return true if P depends on TOOLCHAIN instead of the default tool chain." + (define toolchain-packages + '("gcc" "binutils" "glibc" "ld-wrapper")) + + (define (package-name* obj) + (and (package? obj) (package-name obj))) + + (match (bag-build-inputs (package->bag p)) + (((_ (= package-name* packages) . _) ...) + (and (not (any (cut member <> packages) toolchain-packages)) + (member toolchain packages))))) + +(test-assert "options->transformation, with-c-toolchain" + (let* ((dep0 (dummy-package "chbouib" + (build-system gnu-build-system) + (native-inputs `(("y" ,grep))))) + (dep1 (dummy-package "stuff" + (native-inputs `(("x" ,dep0))))) + (p (dummy-package "thingie" + (build-system gnu-build-system) + (inputs `(("foo" ,grep) + ("bar" ,dep1))))) + (t (options->transformation + '((with-c-toolchain . "chbouib=gcc-toolchain"))))) + ;; Here we check that the transformation applies to DEP0 and all its + ;; dependents: DEP0 must use GCC-TOOLCHAIN, DEP1 must use GCC-TOOLCHAIN + ;; and the DEP0 that uses GCC-TOOLCHAIN, and so on. + (with-store store + (let ((new (t store p))) + (and (depends-on-toolchain? new "gcc-toolchain") + (match (bag-build-inputs (package->bag new)) + ((("foo" dep0) ("bar" dep1) _ ...) + (and (depends-on-toolchain? dep1 "gcc-toolchain") + (not (depends-on-toolchain? dep0 "gcc-toolchain")) + (string=? (package-full-name dep0) + (package-full-name grep)) + (match (bag-build-inputs (package->bag dep1)) + ((("x" dep) _ ...) + (and (depends-on-toolchain? dep "gcc-toolchain") + (match (bag-build-inputs (package->bag dep)) + ((("y" dep) _ ...) ;this one is unchanged + (eq? dep grep)))))))))))))) + +(test-equal "options->transformation, with-c-toolchain twice" + (package-full-name grep) + (let* ((dep0 (dummy-package "chbouib")) + (dep1 (dummy-package "stuff")) + (p (dummy-package "thingie" + (build-system gnu-build-system) + (inputs `(("foo" ,dep0) + ("bar" ,dep1) + ("baz" ,grep))))) + (t (options->transformation + '((with-c-toolchain . "chbouib=clang-toolchain") + (with-c-toolchain . "stuff=clang-toolchain"))))) + (with-store store + (let ((new (t store p))) + (and (depends-on-toolchain? new "clang-toolchain") + (match (bag-build-inputs (package->bag new)) + ((("foo" dep0) ("bar" dep1) ("baz" dep2) _ ...) + (and (depends-on-toolchain? dep0 "clang-toolchain") + (depends-on-toolchain? dep1 "clang-toolchain") + (not (depends-on-toolchain? dep2 "clang-toolchain")) + (package-full-name dep2))))))))) + +(test-assert "options->transformation, with-c-toolchain, no effect" + (let ((p (dummy-package "thingie")) + (t (options->transformation + '((with-c-toolchain . "does-not-exist=gcc-toolchain"))))) + ;; When it has no effect, '--with-c-toolchain' returns P. + (with-store store + (eq? (t store p) p)))) + (test-assert "options->transformation, without-tests" (let* ((dep (dummy-package "dep")) (p (dummy-package "foo" @@ -286,3 +364,7 @@ '(#:tests? #f)))))))) (test-end) + +;;; Local Variables: +;;; eval: (put 'dummy-package 'scheme-indent-function 1) +;;; End: -- cgit 1.4.1 From 4b0a1543e1f67828c09be4a75d1155227bf526ac Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Wed, 14 Oct 2020 10:53:00 +0200 Subject: doc: Avoid orphan node. Commit 4b5a6fbc9b754c0ca70d033dd99f17c4f028733a turned "Transparent Emulation with QEMU" into a node, but that was an orphan node, with no up/next/previous links. * doc/guix.texi (Virtualization Services): Capitalize subsection title. (Transparent Emulation with QEMU): Remove "@node" and add "@anchor". (Daemon Offload Setup): Adjust cross-reference accordingly. --- doc/guix.texi | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 7150adeaa8..e064411d4e 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -1077,8 +1077,9 @@ is requested, for instance via @code{guix build}, the daemon attempts to offload it to one of the machines that satisfy the constraints of the derivation, in particular its system types---e.g., @code{x86_64-linux}. A single machine can have multiple system types, either because its -architecture natively supports it, via emulation (@pxref{Transparent -Emulation with QEMU}), or both. Missing prerequisites for the build are +architecture natively supports it, via emulation +(@pxref{transparent-emulation-qemu, Transparent Emulation with QEMU}), +or both. Missing prerequisites for the build are copied over SSH to the target machine, which then proceeds with the build; upon success the output(s) of the build are copied back to the initial machine. The offload facility comes with a basic scheduler that @@ -24992,13 +24993,14 @@ an HTTP audio streaming output. @node Virtualization Services -@subsection Virtualization services +@subsection Virtualization Services The @code{(gnu services virtualization)} module provides services for the libvirt and virtlog daemons, as well as other virtualization-related services. @subsubheading Libvirt daemon + @code{libvirtd} is the server side daemon component of the libvirt virtualization management system. This daemon runs on host servers and performs required management tasks for virtualized guests. @@ -25699,7 +25701,8 @@ Maximum number of backup files to keep. Defaults to @samp{3} @end deftypevr -@node Transparent Emulation with QEMU + +@anchor{transparent-emulation-qemu} @subsubheading Transparent Emulation with QEMU @cindex emulation -- cgit 1.4.1 From d96e739f08f13cc1981f9367bbda1eba3c567f5f Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Wed, 14 Oct 2020 10:57:06 +0200 Subject: doc: Mention "i586-gnu" as supported and experimental. * doc/guix.texi (GNU Distribution): Mention "i586-gnu". --- doc/guix.texi | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index e064411d4e..ff33735c3a 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -477,10 +477,10 @@ Packages are currently available on the following platforms: @table @code @item x86_64-linux -Intel/AMD @code{x86_64} architecture, Linux-Libre kernel; +Intel/AMD @code{x86_64} architecture, Linux-Libre kernel. @item i686-linux -Intel 32-bit architecture (IA32), Linux-Libre kernel; +Intel 32-bit architecture (IA32), Linux-Libre kernel. @item armhf-linux ARMv7-A architecture with hard float, Thumb-2 and NEON, @@ -490,6 +490,16 @@ and Linux-Libre kernel. @item aarch64-linux little-endian 64-bit ARMv8-A processors, Linux-Libre kernel. +@item i586-gnu +@uref{https://hurd.gnu.org, GNU/Hurd} on the Intel 32-bit architecture +(IA32). + +This configuration is experimental and under development. The easiest +way for you to give it a try is by setting up an instance of +@code{hurd-vm-service-type} on your GNU/Linux machine +(@pxref{transparent-emulation-qemu, @code{hurd-vm-service-type}}). +@xref{Contributing}, on how to help! + @item mips64el-linux (deprecated) little-endian 64-bit MIPS processors, specifically the Loongson series, n32 ABI, and Linux-Libre kernel. This configuration is no longer fully -- cgit 1.4.1 From cd958802f20e07967d2245039ebadd5911c47333 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Wed, 14 Oct 2020 11:28:30 +0200 Subject: doc: cookbook: Mention "#true" and "#false". * doc/guix-cookbook.texi (A Scheme Crash Course): Mention "#true" and "#false". (Extended example): Likewise. --- doc/guix-cookbook.texi | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) (limited to 'doc') diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi index a783c0ae4c..affb68ca12 100644 --- a/doc/guix-cookbook.texi +++ b/doc/guix-cookbook.texi @@ -128,8 +128,9 @@ REPL. @item Scheme syntax boils down to a tree of expressions (or @emph{s-expression} in Lisp lingo). An expression can be a literal such as numbers and strings, or a -compound which is a parenthesized list of compounds and literals. @code{#t} -and @code{#f} stand for the Booleans ``true'' and ``false'', respectively. +compound which is a parenthesized list of compounds and literals. @code{#true} +and @code{#false} (abbreviated @code{#t} and @code{#f}) stand for the +Booleans ``true'' and ``false'', respectively. Examples of valid expressions: @@ -1090,9 +1091,9 @@ this: #t) @end lisp -The procedure must return @code{#t} on success. It's brittle to rely on the return +The procedure must return @code{#true} on success. It's brittle to rely on the return value of the last expression used to tweak the phase because there is no -guarantee it would be a @code{#t}. Hence the trailing @code{#t} to ensure the right value +guarantee it would be a @code{#true}. Hence the trailing @code{#true} to ensure the right value is returned on success. @subsubsection Code staging -- cgit 1.4.1 From 00014f769233facebd84f13a00b10032a22cb440 Mon Sep 17 00:00:00 2001 From: Oleg Pykhalov Date: Sun, 11 Oct 2020 20:42:48 +0300 Subject: services: nginx: Add lua module. * gnu/services/web.scm () [lua-package-path, lua-package-cpath]: New record types. * gnu/services/web.scm (default-nginx-config): Use them. * doc/guix.texi (Web Services): Document this. * doc/guix-cookbook.texi (System Configuration): Document this. --- doc/guix-cookbook.texi | 58 ++++++++++++++++++++++++++++++++++++++++++++++++++ doc/guix.texi | 24 ++++++++++++++++++++- gnu/services/web.scm | 25 +++++++++++++++++----- 3 files changed, 101 insertions(+), 6 deletions(-) (limited to 'doc') diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi index affb68ca12..db3d7e505b 100644 --- a/doc/guix-cookbook.texi +++ b/doc/guix-cookbook.texi @@ -1353,6 +1353,7 @@ reference. * Running Guix on a Linode Server:: Running Guix on a Linode Server * Setting up a bind mount:: Setting up a bind mount in the file-systems definition. * Getting substitutes from Tor:: Configuring Guix daemon to get substitutes through Tor. +* Setting up NGINX with Lua:: Configuring NGINX web-server to load Lua modules. @end menu @node Customizing the Kernel @@ -2114,6 +2115,63 @@ sudo herd set-http-proxy guix-daemon http://localhost:9250 guix build --substitute-urls=https://bp7o7ckwlewr4slm.onion … @end example +@node Setting up NGINX with Lua +@section Setting up NGINX with Lua +@cindex nginx, lua, openresty, resty + +NGINX could be extended with Lua scripts. + +Guix provides NGINX service with ability to load Lua module and specific +Lua packages, and reply to requests by evaluating Lua scripts. + +The following example demonstrates system definition with configuration +to evaluate @file{index.lua} Lua script on HTTP request to +@uref{http://localhost/hello} endpoint: + +@example +local shell = require "resty.shell" + +local stdin = "" +local timeout = 1000 -- ms +local max_size = 4096 -- byte + +local ok, stdout, stderr, reason, status = + shell.run([[/run/current-system/profile/bin/ls /tmp]], stdin, timeout, max_size) + +ngx.say(stdout) +@end example + +@lisp +(use-modules (gnu)) +(use-service-modules #;… web) +(use-package-modules #;… lua) +(operating-system + ;; … + (services + ;; … + (service nginx-service-type + (nginx-configuration + (modules + (list + (file-append nginx-lua-module "/etc/nginx/modules/ngx_http_lua_module.so"))) + (lua-package-path (list lua-resty-core + lua-resty-lrucache + lua-resty-signal + lua-tablepool + lua-resty-shell)) + (lua-package-cpath (list lua-resty-signal)) + (server-blocks + (list (nginx-server-configuration + (server-name '("localhost")) + (listen '("80")) + (root "/etc") + (locations (list + (nginx-location-configuration + (uri "/hello") + (body (list #~(format #f "content_by_lua_file ~s;" + #$(local-file "index.lua")))))))))))))) +@end lisp + @c ********************************************************************* @node Advanced package management @chapter Advanced package management diff --git a/doc/guix.texi b/doc/guix.texi index ff33735c3a..2374c956e5 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -21984,7 +21984,29 @@ names of loadable modules, as in this example: (modules (list (file-append nginx-accept-language-module "\ -/etc/nginx/modules/ngx_http_accept_language_module.so"))) +/etc/nginx/modules/ngx_http_accept_language_module.so") + (file-append nginx-lua-module "\ +/etc/nginx/modules/ngx_http_lua_module.so"))) +@end lisp + +@item @code{lua-package-path} (default: @code{'()}) +List of nginx lua packages to load. This should be a list of package +names of loadable lua modules, as in this example: + +@lisp +(lua-package-path (list lua-resty-core + lua-resty-lrucache + lua-resty-signal + lua-tablepool + lua-resty-shell)) +@end lisp + +@item @code{lua-package-cpath} (default: @code{'()}) +List of nginx lua C packages to load. This should be a list of package +names of loadable lua C modules, as in this example: + +@lisp +(lua-package-cpath (list lua-resty-signal)) @end lisp @item @code{global-directives} (default: @code{'((events . ()))}) diff --git a/gnu/services/web.scm b/gnu/services/web.scm index b2d1dcf0ee..a74c6c54b4 100644 --- a/gnu/services/web.scm +++ b/gnu/services/web.scm @@ -13,6 +13,7 @@ ;;; Copyright © 2020 Ricardo Wurmus ;;; Copyright © 2020 Tobias Geerinckx-Rice ;;; Copyright © 2020 Arun Isaac +;;; Copyright © 2020 Oleg Pykhalov ;;; ;;; This file is part of GNU Guix. ;;; @@ -525,6 +526,10 @@ (modules nginx-configuration-modules (default '())) (global-directives nginx-configuration-global-directives (default '((events . ())))) + (lua-package-path nginx-lua-package-path ;list of + (default #f)) + (lua-package-cpath nginx-lua-package-cpath ;list of + (default #f)) (extra-content nginx-configuration-extra-content (default "")) (file nginx-configuration-file ;#f | string | file-like @@ -630,6 +635,8 @@ of index files." server-names-hash-bucket-max-size modules global-directives + lua-package-path + lua-package-cpath extra-content) (apply mixed-text-file "nginx.conf" (flatten @@ -646,11 +653,19 @@ of index files." " scgi_temp_path " run-directory "/scgi_temp;\n" " access_log " log-directory "/access.log;\n" " include " nginx "/share/nginx/conf/mime.types;\n" - (if server-names-hash-bucket-size - (string-append - " server_names_hash_bucket_size " - (number->string server-names-hash-bucket-size) - ";\n") + (if lua-package-path + #~(format #f " lua_package_path ~s;~%" + (string-join (map (lambda (path) + (string-append path "/lib/?.lua")) + '#$lua-package-path) + ";")) + "") + (if lua-package-cpath + #~(format #f " lua_package_cpath ~s;~%" + (string-join (map (lambda (cpath) + (string-append cpath "/lib/lua/?.lua")) + '#$lua-package-cpath) + ";")) "") (if server-names-hash-bucket-max-size (string-append -- cgit 1.4.1 From b4369430e3c0d895f7be92e473c1aca261c699dc Mon Sep 17 00:00:00 2001 From: Mathieu Othacehe Date: Thu, 15 Oct 2020 14:15:36 +0200 Subject: doc: Fix typo. * doc/guix.texi (Guix Build Coordinator): Fix typo. --- doc/guix.texi | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 2374c956e5..c9c031ccdb 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -28015,7 +28015,7 @@ allocation plan in the database. @item @code{hooks} (default: @var{'()}) An association list of hooks. These provide a way to execute arbitrary -code upon certian events, like a build result being processed. +code upon certain events, like a build result being processed. @item @code{guile} (default: @code{guile-3.0-latest}) The Guile package with which to run the Guix Build Coordinator. -- cgit 1.4.1 From 7802988201820dd951407521e243557293dceb94 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Thu, 15 Oct 2020 15:16:59 +0200 Subject: doc: cookbook: Fix 'let' form in example. * doc/guix-cookbook.texi (Extended example): Fix 'let' form in example. --- doc/guix-cookbook.texi | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) (limited to 'doc') diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi index db3d7e505b..c4abeda1e1 100644 --- a/doc/guix-cookbook.texi +++ b/doc/guix-cookbook.texi @@ -1084,11 +1084,11 @@ this: @lisp (lambda* (#:key inputs outputs #:allow-other-keys) - (let (((bash-directory (assoc-ref inputs "bash")) - (output-directory (assoc-ref outputs "out")) - (doc-directory (assoc-ref outputs "doc")) - ; ... - #t) + (let ((bash-directory (assoc-ref inputs "bash")) + (output-directory (assoc-ref outputs "out")) + (doc-directory (assoc-ref outputs "doc"))) + ;; ... + #t)) @end lisp The procedure must return @code{#true} on success. It's brittle to rely on the return -- cgit 1.4.1 From 51f34e21c69459d3226bb1bbe0c0aaaf2e00465c Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Thu, 15 Oct 2020 15:24:07 +0200 Subject: doc: cookbook: Use @var for meta-syntactic variables. * doc/guix-cookbook.texi (Extended example): Use @var for meta-syntactic variables in 'modify-phases' description. --- doc/guix-cookbook.texi | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) (limited to 'doc') diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi index c4abeda1e1..a669015269 100644 --- a/doc/guix-cookbook.texi +++ b/doc/guix-cookbook.texi @@ -1067,16 +1067,16 @@ the following forms: @itemize @item -@code{(add-before PHASE NEW-PHASE PROCEDURE)}: Run @code{PROCEDURE} named @code{NEW-PHASE} before @code{PHASE}. +@code{(add-before @var{phase} @var{new-phase} @var{procedure})}: Run @var{procedure} named @var{new-phase} before @var{phase}. @item -@code{(add-after PHASE NEW-PHASE PROCEDURE)}: Same, but afterwards. +@code{(add-after @var{phase} @var{new-phase} @var{procedure})}: Same, but afterwards. @item -@code{(replace PHASE PROCEDURE)}. +@code{(replace @var{phase} @var{procedure})}. @item -@code{(delete PHASE)}. +@code{(delete @var{phase})}. @end itemize -The @code{PROCEDURE} supports the keyword arguments @code{inputs} and @code{outputs}. Each +The @var{procedure} supports the keyword arguments @code{inputs} and @code{outputs}. Each input (whether @emph{native}, @emph{propagated} or not) and output directory is referenced by their name in those variables. Thus @code{(assoc-ref outputs "out")} is the store directory of the main output of the package. A phase procedure may look like -- cgit 1.4.1 From 345136834579940ff85657ccba08cf092e949579 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Thu, 15 Oct 2020 15:35:25 +0200 Subject: doc: cookbook: More "#true" and "#false". * doc/guix-cookbook.texi (Extended example) (Customizing the Kernel, Running Guix on a Linode Server): Use #true and #false. --- doc/guix-cookbook.texi | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) (limited to 'doc') diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi index a669015269..c41daacda9 100644 --- a/doc/guix-cookbook.texi +++ b/doc/guix-cookbook.texi @@ -792,11 +792,11 @@ another, more sophisticated package (slightly modified from the source): (snippet '(begin ;; Remove bundled software. (delete-file-recursively "deps") - #t)))) + #true)))) (build-system cmake-build-system) (outputs '("out" "debug")) (arguments - `(#:tests? #t ; Run the test suite (this is the default) + `(#:tests? #true ; Run the test suite (this is the default) #:configure-flags '("-DUSE_SHA1DC=ON") ; SHA-1 collision detection #:phases (modify-phases %standard-phases @@ -807,12 +807,12 @@ another, more sophisticated package (slightly modified from the source): (substitute* "tests/clar/fs.h" (("/bin/cp") (which "cp")) (("/bin/rm") (which "rm"))) - #t)) + #true)) ;; Run checks more verbosely. (replace 'check (lambda _ (invoke "./libgit2_clar" "-v" "-Q"))) (add-after 'unpack 'make-files-writable-for-tests - (lambda _ (for-each make-file-writable (find-files "." ".*"))))))) + (lambda _ (for-each make-file-writable (find-files "." ".*"))))))) (inputs `(("libssh2" ,libssh2) ("http-parser" ,http-parser) @@ -1030,7 +1030,7 @@ If you want to know more about what happens during those phases, consult the associated procedures. For instance, as of this writing the definition of @code{unpack} for the GNU build -system is +system is: @lisp (define* (unpack #:key source #:allow-other-keys) @@ -1045,13 +1045,13 @@ working directory." ;; Preserve timestamps (set to the Epoch) on the copied tree so that ;; things work deterministically. (copy-recursively source "." - #:keep-mtime? #t)) + #:keep-mtime? #true)) (begin (if (string-suffix? ".zip" source) (invoke "unzip" source) (invoke "tar" "xvf" source)) (chdir (first-subdirectory ".")))) - #t) + #true) @end lisp Note the @code{chdir} call: it changes the working directory to where the source was @@ -1088,7 +1088,7 @@ this: (output-directory (assoc-ref outputs "out")) (doc-directory (assoc-ref outputs "doc"))) ;; ... - #t)) + #true)) @end lisp The procedure must return @code{#true} on success. It's brittle to rely on the return @@ -1386,8 +1386,8 @@ creates a package. #:key ;; A function that takes an arch and a variant. ;; See kernel-config for an example. - (extra-version #f) - (configuration-file #f) + (extra-version #false) + (configuration-file #false) (defconfig "defconfig") (extra-options %default-extra-linux-options) (patches (list %boot-logo-patch))) @@ -1461,7 +1461,7 @@ it: @lisp (define %default-extra-linux-options `(;; https://lists.gnu.org/archive/html/guix-devel/2014-04/msg00039.html - ("CONFIG_DEVPTS_MULTIPLE_INSTANCES" . #t) + ("CONFIG_DEVPTS_MULTIPLE_INSTANCES" . #true) ;; Modules required for initrd: ("CONFIG_NET_9P" . m) ("CONFIG_NET_9P_VIRTIO" . m) @@ -1478,9 +1478,9 @@ it: (string-join (map (match-lambda ((option . 'm) (string-append option "=m")) - ((option . #t) + ((option . #true) (string-append option "=y")) - ((option . #f) + ((option . #false) (string-append option "=n"))) options) "\n")) @@ -1867,7 +1867,7 @@ is below. Save the resulting file as @file{guix-config.scm}. (bootloader (bootloader (inherit grub-bootloader) - (installer #~(const #t)))))) + (installer #~(const #true)))))) (file-systems (cons (file-system (device "/dev/sda") (mount-point "/") @@ -1899,7 +1899,7 @@ is below. Save the resulting file as @file{guix-config.scm}. (service openssh-service-type (openssh-configuration (openssh openssh-sans-x) - (password-authentication? #f) + (password-authentication? #false) (authorized-keys `(("janedoe" ,(local-file "janedoe_rsa.pub")) ("root" ,(local-file "janedoe_rsa.pub")))))) -- cgit 1.4.1 From 48720afb322ab5ad1b6102276f4795a14803fa61 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Thu, 15 Oct 2020 16:13:00 +0200 Subject: doc: cookbook: Use @code for the identifiers of utility functions. * doc/guix-cookbook.texi (Utility functions): Use @code instead of @asis. --- doc/guix-cookbook.texi | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi index c41daacda9..9dffbca196 100644 --- a/doc/guix-cookbook.texi +++ b/doc/guix-cookbook.texi @@ -1119,7 +1119,7 @@ Some of those functions can be found in @samp{$GUIX_CHECKOUT/guix/guix/build/utils.scm}. Most of them mirror the behaviour of the traditional Unix system commands: -@table @asis +@table @code @item which Like the @samp{which} system command. @item find-files -- cgit 1.4.1 From 61263e1ba3a5d9b31da5876b1c5248cee3859996 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Thu, 15 Oct 2020 18:48:30 +0200 Subject: doc: Add "Build Utilities" section. * doc/guix.texi (Build Utilities): New section. (Defining Packages): Refer to it. (Build Systems): Likewise. * doc/guix-cookbook.texi (Extended example): Likewise. --- doc/guix-cookbook.texi | 3 + doc/guix.texi | 276 ++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 278 insertions(+), 1 deletion(-) (limited to 'doc') diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi index 9dffbca196..2845391db4 100644 --- a/doc/guix-cookbook.texi +++ b/doc/guix-cookbook.texi @@ -1143,6 +1143,9 @@ then restore the previous working directory. A ``@command{sed}-like'' function. @end table +@xref{Build Utilities,,, guix, GNU Guix Reference Manual}, for more +information on these utilities. + @subsubsection Module prefix The license in our last example needs a prefix: this is because of how the diff --git a/doc/guix.texi b/doc/guix.texi index c9c031ccdb..26de6790fe 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -252,6 +252,7 @@ Programming Interface * Package Modules:: Packages from the programmer's viewpoint. * Defining Packages:: Defining new packages. * Build Systems:: Specifying how packages are built. +* Build Utilities:: Helpers for your package definitions and more. * The Store:: Manipulating the package store. * Derivations:: Low-level interface to package derivations. * The Store Monad:: Purely functional interface to the store. @@ -6085,6 +6086,7 @@ package definitions. * Package Modules:: Packages from the programmer's viewpoint. * Defining Packages:: Defining new packages. * Build Systems:: Specifying how packages are built. +* Build Utilities:: Helpers for your package definitions and more. * The Store:: Manipulating the package store. * Derivations:: Low-level interface to package derivations. * The Store Monad:: Purely functional interface to the store. @@ -6241,6 +6243,10 @@ represents the familiar GNU Build System, where packages may be configured, built, and installed with the usual @code{./configure && make && make check && make install} command sequence. +When you start packaging non-trivial software, you may need tools to +manipulate those build phases, manipulate files, and so on. @xref{Build +Utilities}, for more on this. + @item The @code{arguments} field specifies options for the build system (@pxref{Build Systems}). Here it is interpreted by @@ -6816,7 +6822,8 @@ The list of phases used for a particular package can be changed with the @end example means that all the phases described above will be used, except the -@code{configure} phase. +@code{configure} phase. @xref{Build Utilities}, for more info on +@code{modify-phases} and build phases in general. In addition, this build system ensures that the ``standard'' environment for GNU packages is available. This includes tools such as GCC, libc, @@ -7646,6 +7653,273 @@ with @code{build-expression->derivation} (@pxref{Derivations, @code{build-expression->derivation}}). @end defvr +@node Build Utilities +@section Build Utilities + +As soon as you start writing non-trivial package definitions +(@pxref{Defining Packages}) or other build actions +(@pxref{G-Expressions}), you will likely start looking for helpers for +``shell-like'' actions---creating directories, copying and deleting +files recursively, manipulating build phases, and so on. The +@code{(guix build utils)} module provides such utility procedures. + +When writing package definitions, most build systems load @code{(guix +build utils)} (@pxref{Build Systems}). Thus, when writing custom build +phases or similar, you can usually assume those procedures are in scope. + +When writing g-expressions, you can import @code{(guix build utils)} on +the ``build side'' using @code{with-imported-modules} and then put it in +scope with the @code{use-modules} form (@pxref{Using Guile Modules,,, +guile, GNU Guile Reference Manual}): + +@lisp +(with-imported-modules '((guix build utils)) ;import it + (computed-file "empty-tree" + #~(begin + ;; Put it in scope. + (use-modules (guix build utils)) + + ;; Happily use its 'mkdir-p' procedure. + (mkdir-p (string-append #$output "/a/b/c"))))) +@end lisp + +The remainder of this section is the reference for most of the utility +procedures provided by @code{(guix build utils)}. + +@c TODO Document what's missing. + +@subsection Dealing with Store File Names + +This section of procedures deals with store file names. + +@deffn {Scheme Procedure} %store-directory +Return the directory name of the store. +@end deffn + +@deffn {Scheme Procedure} store-file-name? @var{file} +Return true if @var{file} is in the store. +@end deffn + +@deffn {Scheme Procedure} strip-store-file-name @var{file} +Strip the @file{/gnu/store} and hash from @var{file}, a store file name. +The result is typically a @code{"PACKAGE-VERSION"} string. +@end deffn + +@deffn {Scheme Procedure} package-name->name+version @var{name} +Given @var{name}, a package name like @code{"foo-0.9.1b"}, return two +values: @code{"foo"} and @code{"0.9.1b"}. When the version part is +unavailable, @var{name} and @code{#f} are returned. The first hyphen +followed by a digit is considered to introduce the version part. +@end deffn + +@subsection File Types + +The procedures below deal with files and file types. + +@deffn {Scheme Procedure} directory-exists? @var{dir} +Return @code{#t} if @var{dir} exists and is a directory. +@end deffn + +@deffn {Scheme Procedure} executable-file? @var{file} +Return @code{#t} if @var{file} exists and is executable. +@end deffn + +@deffn {Scheme Procedure} symbolic-link? @var{file} +Return @code{#t} if @var{file} is a symbolic link (aka. a ``symlink''). +@end deffn + +@deffn {Scheme Procedure} elf-file? @var{file} +@deffnx {Scheme Procedure} ar-file? @var{file} +@deffnx {Scheme Procedure} gzip-file? @var{file} +Return @code{#t} if @var{file} is, respectively, an ELF file, an +@code{ar} archive (such as a @file{.a} static library), or a gzip file. +@end deffn + +@deffn {Scheme Procedure} reset-gzip-timestamp @var{file} [#:keep-mtime? #t] +If @var{file} is a gzip file, reset its embedded timestamp (as with +@command{gzip --no-name}) and return true. Otherwise return @code{#f}. +When @var{keep-mtime?} is true, preserve @var{file}'s modification time. +@end deffn + +@subsection File Manipulation + +The following procedures and macros help create, modify, and delete +files. They provide functionality comparable to common shell utilities +such as @command{mkdir -p}, @command{cp -r}, @command{rm -r}, and +@command{sed}. They complement Guile's extensive, but low-level, file +system interface (@pxref{POSIX,,, guile, GNU Guile Reference Manual}). + +@deffn {Scheme Syntax} with-directory-excursion @var{directory} @var{body}@dots{} +Run @var{body} with @var{directory} as the process's current directory. + +Essentially, this macro changes the current directory to @var{directory} +before evaluating @var{body}, using @code{chdir} (@pxref{Processes,,, +guile, GNU Guile Reference Manual}). It changes back to the initial +directory when the dynamic extent of @var{body} is left, be it @i{via} +normal procedure return or @i{via} a non-local exit such as an +exception. +@end deffn + +@deffn {Scheme Procedure} mkdir-p @var{dir} +Create directory @var{dir} and all its ancestors. +@end deffn + +@deffn {Scheme Procedure} install-file @var{file} @var{directory} +Create @var{directory} if it does not exist and copy @var{file} in there +under the same name. +@end deffn + +@deffn {Scheme Procedure} make-file-writable @var{file} +Make @var{file} writable for its owner. +@end deffn + +@deffn {Scheme Procedure} copy-recursively @var{source} @var{destination} @ + [#:log (current-output-port)] [#:follow-symlinks? #f] [#:keep-mtime? #f] +Copy @var{source} directory to @var{destination}. Follow symlinks if +@var{follow-symlinks?} is true; otherwise, just preserve them. When +@var{keep-mtime?} is true, keep the modification time of the files in +@var{source} on those of @var{destination}. Write verbose output to the +@var{log} port. +@end deffn + +@deffn {Scheme Procedure} delete-file-recursively @var{dir} @ + [#:follow-mounts? #f] +Delete @var{dir} recursively, like @command{rm -rf}, without following +symlinks. Don't follow mount points either, unless @var{follow-mounts?} +is true. Report but ignore errors. +@end deffn + +@deffn {Scheme Syntax} substitute* @var{clause}@dots{} +Substitute @var{regexp} in @var{file} by the string returned by +@var{body}. @var{body} is evaluated with each @var{match-var} bound to +the corresponding positional regexp sub-expression. For example: + +@lisp +(substitute* file + (("hello") + "good morning\n") + (("foo([a-z]+)bar(.*)$" all letters end) + (string-append "baz" letter end))) +@end lisp + +Here, anytime a line of @var{file} contains @code{hello}, it is replaced +by @code{good morning}. Anytime a line of @var{file} matches the second +regexp, @var{all} is bound to the complete match, @var{letters} is bound +to the first sub-expression, and @var{end} is bound to the last one. + +When one of the @var{match-var} is @code{_}, no variable is bound to the +corresponding match substring. + +Alternatively, @var{file} may be a list of file names, in which case +they are all subject to the substitutions. + +Be careful about using @code{$} to match the end of a line; by itself it +won't match the terminating newline of a line. +@end deffn + +@subsection File Search + +@cindex file, searching +This section documents procedures to search and filter files. + +@deffn {Scheme Procedure} file-name-predicate @var{regexp} +Return a predicate that returns true when passed a file name whose base +name matches @var{regexp}. +@end deffn + +@deffn {Scheme Procedure} find-files @var{dir} [@var{pred}] @ + [#:stat lstat] [#:directories? #f] [#:fail-on-error? #f] +Return the lexicographically sorted list of files under @var{dir} for +which @var{pred} returns true. @var{pred} is passed two arguments: the +absolute file name, and its stat buffer; the default predicate always +returns true. @var{pred} can also be a regular expression, in which +case it is equivalent to @code{(file-name-predicate @var{pred})}. +@var{stat} is used to obtain file information; using @code{lstat} means +that symlinks are not followed. If @var{directories?} is true, then +directories will also be included. If @var{fail-on-error?} is true, +raise an exception upon error. +@end deffn + +Here are a few examples where we assume that the current directory is +the root of the Guix source tree: + +@lisp +;; List all the regular files in the current directory. +(find-files ".") +@result{} ("./.dir-locals.el" "./.gitignore" @dots{}) + +;; List all the .scm files under gnu/services. +(find-files "gnu/services" "\\.scm$") +@result{} ("gnu/services/admin.scm" "gnu/services/audio.scm" @dots{}) + +;; List ar files in the current directory. +(find-files "." (lambda (file stat) (ar-file? file))) +@result{} ("./libformat.a" "./libstore.a" @dots{}) +@end lisp + +@deffn {Scheme Procedure} which @var{program} +Return the complete file name for @var{program} as found in +@code{$PATH}, or @code{#f} if @var{program} could not be found. +@end deffn + +@subsection Build Phases + +@cindex build phases +The @code{(guix build utils)} also contains tools to manipulate +@dfn{build phases} as found in @code{gnu-build-system} and in fact most +build systems (@pxref{Build Systems}). Build phases are represented as +association lists or ``alists'' (@pxref{Association Lists,,, guile, GNU +Guile Reference Manual}) where each key is a symbol for the name of the +phase, and the associated value is a procedure that accepts an arbitrary +number of arguments. + +Guile core and the @code{(srfi srfi-1)} module both provide tools to +manipulate alists. The @code{(guix build utils)} module complements +those with tools written with build phases in mind. + +@cindex build phases, modifying +@deffn {Scheme Syntax} modify-phases @var{phases} @var{clause}@dots{} +Modify @var{phases} sequentially as per each @var{clause}, which may +have one of the following forms: + +@lisp +(delete @var{old-phase-name}) +(replace @var{old-phase-name} @var{new-phase}) +(add-before @var{old-phase-name} @var{new-phase-name} @var{new-phase}) +(add-after @var{old-phase-name} @var{new-phase-name} @var{new-phase}) +@end lisp + +Where every @var{phase-name} above is an expression evaluating to a +symbol, and @var{new-phase} an expression evaluating to a procedure. +@end deffn + +The example below is taken from the definition of the @code{grep} +package. It adds a phase to run after the @code{install} phase, called +@code{fix-egrep-and-fgrep}. That phase is a procedure (@code{lambda*} +is for anonymous procedures) that takes a @code{#:outputs} keyword +argument and ignores extra keyword arguments (@pxref{Optional +Arguments,,, guile, GNU Guile Reference Manual}, for more on +@code{lambda*} and optional and keyword arguments.) The phase uses +@code{substitute*} to modify the installed @file{egrep} and @file{fgrep} +scripts so that they refer to @code{grep} by its absolute file name: + +@lisp +(modify-phases %standard-phases + (add-after 'install 'fix-egrep-and-fgrep + ;; Patch 'egrep' and 'fgrep' to execute 'grep' via its + ;; absolute file name instead of searching for it in $PATH. + (lambda* (#:key outputs #:allow-other-keys) + (let* ((out (assoc-ref outputs "out")) + (bin (string-append out "/bin"))) + (substitute* (list (string-append bin "/egrep") + (string-append bin "/fgrep")) + (("^exec grep") + (string-append "exec " bin "/grep"))) + #t)))) +@end lisp + +@c TODO: Add more examples. + @node The Store @section The Store -- cgit 1.4.1 From 699f1de7218421b9ab6752fef7a7109dca6ff7fc Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Thu, 15 Oct 2020 23:32:47 +0200 Subject: doc: Improve "Build Utilities". * doc/guix.texi (Build Utilities): Fix typos. Provide the correct syntax for 'substitute*'. Add a 'modify-phases' example. --- doc/guix.texi | 39 ++++++++++++++++++++++++++++++--------- 1 file changed, 30 insertions(+), 9 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 26de6790fe..51dc42e5a2 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -7663,11 +7663,11 @@ As soon as you start writing non-trivial package definitions files recursively, manipulating build phases, and so on. The @code{(guix build utils)} module provides such utility procedures. -When writing package definitions, most build systems load @code{(guix -build utils)} (@pxref{Build Systems}). Thus, when writing custom build -phases or similar, you can usually assume those procedures are in scope. +Most build systems load @code{(guix build utils)} (@pxref{Build +Systems}). Thus, when writing custom build phases for your package +definitions, you can usually assume those procedures are in scope. -When writing g-expressions, you can import @code{(guix build utils)} on +When writing G-expressions, you can import @code{(guix build utils)} on the ``build side'' using @code{with-imported-modules} and then put it in scope with the @code{use-modules} form (@pxref{Using Guile Modules,,, guile, GNU Guile Reference Manual}): @@ -7690,7 +7690,7 @@ procedures provided by @code{(guix build utils)}. @subsection Dealing with Store File Names -This section of procedures deals with store file names. +This section documents procedures that deal with store file names. @deffn {Scheme Procedure} %store-directory Return the directory name of the store. @@ -7702,7 +7702,7 @@ Return true if @var{file} is in the store. @deffn {Scheme Procedure} strip-store-file-name @var{file} Strip the @file{/gnu/store} and hash from @var{file}, a store file name. -The result is typically a @code{"PACKAGE-VERSION"} string. +The result is typically a @code{"@var{package}-@var{version}"} string. @end deffn @deffn {Scheme Procedure} package-name->name+version @var{name} @@ -7789,7 +7789,8 @@ symlinks. Don't follow mount points either, unless @var{follow-mounts?} is true. Report but ignore errors. @end deffn -@deffn {Scheme Syntax} substitute* @var{clause}@dots{} +@deffn {Scheme Syntax} substitute* @var{file} @ + ((@var{regexp} @var{match-var}@dots{}) @var{body}@dots{}) @dots{} Substitute @var{regexp} in @var{file} by the string returned by @var{body}. @var{body} is evaluated with each @var{match-var} bound to the corresponding positional regexp sub-expression. For example: @@ -7804,8 +7805,8 @@ the corresponding positional regexp sub-expression. For example: Here, anytime a line of @var{file} contains @code{hello}, it is replaced by @code{good morning}. Anytime a line of @var{file} matches the second -regexp, @var{all} is bound to the complete match, @var{letters} is bound -to the first sub-expression, and @var{end} is bound to the last one. +regexp, @code{all} is bound to the complete match, @code{letters} is bound +to the first sub-expression, and @code{end} is bound to the last one. When one of the @var{match-var} is @code{_}, no variable is bound to the corresponding match substring. @@ -7918,6 +7919,26 @@ scripts so that they refer to @code{grep} by its absolute file name: #t)))) @end lisp +In the example below, phases are modified in two ways: the standard +@code{configure} phase is deleted, presumably because the package does +not have a @file{configure} script or anything similar, and the default +@code{install} phase is replaced by one that manually copies the +executable files to be installed: + +@lisp +(modify-phases %standard-phases + (delete 'configure) ;no 'configure' script + (replace 'install + (lambda* (#:key outputs #:allow-other-keys) + ;; The package's Makefile doesn't provide an "install" + ;; rule so do it by ourselves. + (let ((bin (string-append (assoc-ref outputs "out") + "/bin"))) + (install-file "footswitch" bin) + (install-file "scythe" bin) + #t)))) +@end lisp + @c TODO: Add more examples. @node The Store -- cgit 1.4.1 From c0001b8bfca3409e25bd0e7d8925df2132e91d06 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Thu, 15 Oct 2020 23:50:11 +0200 Subject: doc: cookbook: Clarify what keywords are. * doc/guix-cookbook.texi (A Scheme Crash Course): Clarify what keywords are. --- doc/guix-cookbook.texi | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) (limited to 'doc') diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi index 2845391db4..eb73bd33d6 100644 --- a/doc/guix-cookbook.texi +++ b/doc/guix-cookbook.texi @@ -250,8 +250,10 @@ definitions. @end lisp @item -The keyword syntax is @code{#:}; it is used to create unique identifiers. -@pxref{Keywords,,, guile, GNU Guile Reference Manual}. +@dfn{Keywords} are typically used to identify the named parameters of a +procedure. They are prefixed by @code{#:} (hash, colon) followed by +alphanumeric characters: @code{#:like-this}. +@xref{Keywords,,, guile, GNU Guile Reference Manual}. @item The percentage @code{%} is typically used for read-only global variables in -- cgit 1.4.1 From 3483e6444d541c8c157cb5af754d6af7faa4f845 Mon Sep 17 00:00:00 2001 From: Miguel Ángel Arruga Vivas Date: Sun, 11 Oct 2020 19:46:16 +0200 Subject: doc: Fix minor typos. * doc/guix.texi (cargo-build-system): Fix install phase description. (julia-build-system): Fix typo and add information from check. (pulseaudio-configuration): Remove duplicated words. (knot-acl-configuration)[id]: Fix typo. (mpd-output)[extra-options]: Likewise. (libvirt-configuration) [listen-tls?]: Add missing subject. [listen-tcp?]: Add missing subject and period. [tls-port, tcp-port, log-filters]: Add missing period. (hurd-vm-configuration)[netoptions]: Fix typo in example. (ganeti-watcher-service-type): Fix typo. (grub-theme)[gfxmode]: Remove see before @pxref. --- doc/guix.texi | 36 ++++++++++++++++++------------------ 1 file changed, 18 insertions(+), 18 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 51dc42e5a2..935fc98845 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -6961,8 +6961,8 @@ In its @code{configure} phase, this build system will make any source inputs specified in the @code{#:cargo-inputs} and @code{#:cargo-development-inputs} parameters available to cargo. It will also remove an included @code{Cargo.lock} file to be recreated by @code{cargo} during the -@code{build} phase. The @code{install} phase installs any crate the binaries -if they are defined by the crate. +@code{build} phase. The @code{install} phase installs the binaries +defined by the crate. @end defvr @@ -7208,7 +7208,7 @@ implements the build procedure used by @uref{https://julialang.org/, julia} packages, which essentially is similar to running @samp{julia -e 'using Pkg; Pkg.add(package)'} in an environment where @env{JULIA_LOAD_PATH} contains the paths to all Julia package inputs. -Tests are run not run. +Tests are run with @code{Pkg.test}. Julia packages require the source @code{file-name} to be the real name of the package, correctly capitalized. @@ -18147,10 +18147,10 @@ List of settings to set in @file{daemon.conf}, formatted just like @var{client-conf}. @item @var{script-file} (default: @code{(file-append pulseaudio "/etc/pulse/default.pa")}) -Script file to use as as @file{default.pa}. +Script file to use as @file{default.pa}. @item @var{system-script-file} (default: @code{(file-append pulseaudio "/etc/pulse/system.pa")}) -Script file to use as as @file{system.pa}. +Script file to use as @file{system.pa}. @end table @end deftp @@ -23304,7 +23304,7 @@ This type has the following parameters: @table @asis @item @code{id} (default: @code{""}) -An identifier for ether configuration fields to refer to this key. IDs must be +An identifier for other configuration fields to refer to this key. IDs must be unique and must not be empty. @item @code{address} (default: @code{'()}) @@ -25295,7 +25295,7 @@ mixer, the @code{null} mixer (allows setting the volume, but with no effect; this can be used as a trick to implement an external mixer External Mixer) or no mixer (@code{none}). -@item @code{extra-options} (default: @code{'()"}) +@item @code{extra-options} (default: @code{'()}) An association list of option symbols to string values to be appended to the audio output configuration. @@ -25354,7 +25354,7 @@ Libvirt package. @deftypevr {@code{libvirt-configuration} parameter} boolean listen-tls? Flag listening for secure TLS connections on the public TCP/IP port. -must set @code{listen} for this to have any effect. +You must set @code{listen} for this to have any effect. It is necessary to setup a CA and issue server certificates before using this capability. @@ -25364,28 +25364,28 @@ Defaults to @samp{#t}. @end deftypevr @deftypevr {@code{libvirt-configuration} parameter} boolean listen-tcp? -Listen for unencrypted TCP connections on the public TCP/IP port. must +Listen for unencrypted TCP connections on the public TCP/IP port. You must set @code{listen} for this to have any effect. Using the TCP socket requires SASL authentication by default. Only SASL mechanisms which support data encryption are allowed. This is -DIGEST_MD5 and GSSAPI (Kerberos5) +DIGEST_MD5 and GSSAPI (Kerberos5). Defaults to @samp{#f}. @end deftypevr @deftypevr {@code{libvirt-configuration} parameter} string tls-port -Port for accepting secure TLS connections This can be a port number, or -service name +Port for accepting secure TLS connections. This can be a port number, +or service name. Defaults to @samp{"16514"}. @end deftypevr @deftypevr {@code{libvirt-configuration} parameter} string tcp-port -Port for accepting insecure TCP connections This can be a port number, -or service name +Port for accepting insecure TCP connections. This can be a port number, +or service name. Defaults to @samp{"16509"}. @@ -25697,7 +25697,7 @@ Defaults to @samp{3}. Logging filters. A filter allows to select a different logging level for a given category -of logs The format for a filter is one of: +of logs. The format for a filter is one of: @itemize @bullet @item @@ -26228,7 +26228,7 @@ By default, it produces with forwarded ports: @example -@var{ssh-port}: @code{(+ 11004 (* 1000 @var{ID}))} +@var{secrets-port}: @code{(+ 11004 (* 1000 @var{ID}))} @var{ssh-port}: @code{(+ 10022 (* 1000 @var{ID}))} @var{vnc-port}: @code{(+ 15900 (* 1000 @var{ID}))} @end example @@ -26873,7 +26873,7 @@ When true, the daemon performs additional logging for debugging purposes. @defvr {Scheme Variable} ganeti-watcher-service-type @command{ganeti-watcher} is a script designed to run periodically and ensure the health of a cluster. It will automatically restart instances that have -stopped without Ganetis consent, and repairs DRBD links in case a node has +stopped without Ganeti's consent, and repairs DRBD links in case a node has rebooted. It also archives old cluster jobs and restarts Ganeti daemons that are not running. If the cluster parameter @code{ensure_node_health} is set, the watcher will also shutdown instances and DRBD devices if the @@ -29810,7 +29810,7 @@ Data type representing the configuration of the GRUB theme. @table @asis @item @code{gfxmode} (default: @code{'("auto")}) -The GRUB @code{gfxmode} to set (a list of screen resolution strings, see +The GRUB @code{gfxmode} to set (a list of screen resolution strings, @pxref{gfxmode,,, grub, GNU GRUB manual}). @end table @end deftp -- cgit 1.4.1 From c2a382e5108744ef8b0c8a87d383e84f24c7925a Mon Sep 17 00:00:00 2001 From: Miguel Ángel Arruga Vivas Date: Sun, 11 Oct 2020 19:58:22 +0200 Subject: doc: Fixes for Texinfo metainfo. * doc/guix.texi (guix-pack)[GUIX_EXECUTION_ENGINE]: Use @env instead of @code. (zram-device-service-type): Use @code instead of @var for constant or symbolic values. (rshiny-service-type): Use @env instead of @code for R_LIBS_USER. --- doc/guix.texi | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) (limited to 'doc') diff --git a/doc/guix.texi b/doc/guix.texi index 935fc98845..d22bd41bbb 100644 --- a/doc/guix.texi +++ b/doc/guix.texi @@ -5836,7 +5836,7 @@ direct syscalls are not intercepted either, leading to erratic behavior. @vindex GUIX_EXECUTION_ENGINE When running a wrapped program, you can explicitly request one of the execution engines listed above by setting the -@code{GUIX_EXECUTION_ENGINE} environment variable accordingly. +@env{GUIX_EXECUTION_ENGINE} environment variable accordingly. @end quotation @cindex entry point, for Docker images @@ -28587,22 +28587,22 @@ This is the data type representing the configuration for the zram-device service. @table @asis -@item @code{size} (default @var{"1G"}) +@item @code{size} (default @code{"1G"}) This is the amount of space you wish to provide for the zram device. It accepts a string and can be a number of bytes or use a suffix, eg.: -@var{"512M"} or @var{1024000}. -@item @code{compression-algorithm} (default @var{'lzo}) +@code{"512M"} or @code{1024000}. +@item @code{compression-algorithm} (default @code{'lzo}) This is the compression algorithm you wish to use. It is difficult to list all the possible compression options, but common ones supported by -Guix's Linux Libre Kernel include @var{'lzo}, @var{'lz4} and @var{'zstd}. -@item @code{memory-limit} (default @var{0}) +Guix's Linux Libre Kernel include @code{'lzo}, @code{'lz4} and @code{'zstd}. +@item @code{memory-limit} (default @code{0}) This is the maximum amount of memory which the zram device can use. Setting it to '0' disables the limit. While it is generally expected that compression will be 2:1, it is possible that uncompressable data can be written to swap and this is a method to limit how much memory can be used. It accepts a string and can be a number of bytes or use a -suffix, eg.: @var{"2G"}. -@item @code{priority} (default @var{-1}) +suffix, eg.: @code{"2G"}. +@item @code{priority} (default @code{-1}) This is the priority of the swap device created from the zram device. @code{swapon} accepts values between -1 and 32767, with higher values indicating higher priority. Higher priority swap will generally be used @@ -29015,7 +29015,7 @@ The @code{(gnu services science)} module provides the following service. @defvr {Scheme Variable} rshiny-service-type This is a type of service which is used to run a webapp created with -@code{r-shiny}. This service sets the @code{R_LIBS_USER} environment +@code{r-shiny}. This service sets the @env{R_LIBS_USER} environment variable and runs the provided script to call @code{runApp}. @deftp {Data Type} rshiny-configuration -- cgit 1.4.1 From 3cd1a7ac51c73ce636c3c36b3f790829c8374e04 Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Fri, 16 Oct 2020 22:04:33 +0200 Subject: doc: Remove 'build.scm' from the source of the manual. That way we no longer have to rebuild the whole manual when fiddling with 'build.scm'. * doc/build.scm : Define 'select?' and pass it to 'pdf+html-manual'. --- doc/build.scm | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) (limited to 'doc') diff --git a/doc/build.scm b/doc/build.scm index 97f4ab6b83..26ff577d5d 100644 --- a/doc/build.scm +++ b/doc/build.scm @@ -946,7 +946,12 @@ commit date (an integer)." (let* ((root (canonicalize-path (string-append (current-source-directory) "/.."))) - (commit date (latest-commit+date root))) + (commit date (latest-commit+date root)) + (select? (let ((vcs? (git-predicate root))) + (lambda (file stat) + (and (vcs? file stat) + ;; Filter out this file. + (not (string=? (basename file) "build.scm"))))))) (format (current-error-port) "building manual from work tree around commit ~a, ~a~%" commit @@ -954,7 +959,7 @@ commit date (an integer)." (date (time-utc->date time))) (date->string date "~e ~B ~Y"))) (pdf+html-manual (local-file root "guix" #:recursive? #t - #:select? (git-predicate root)) + #:select? select?) #:version (or (getenv "GUIX_MANUAL_VERSION") (string-take commit 7)) #:date date)) -- cgit 1.4.1 From 0f7d0743edb0922988531ac5bbb04026c57492eb Mon Sep 17 00:00:00 2001 From: Ludovic Courtès Date: Fri, 16 Oct 2020 22:51:30 +0200 Subject: doc: Move manual index creation to a separate derivation. * doc/build.scm (normalize-language-code, html-manual-identifier-index): New procedures. (syntax-highlighted-html): Add #:mono-node-indexes and #:split-node-indexes. [build](underscore-decode, anchor-id->key, collect-anchors): Remove. (language+node-anchors, mono-node-anchors, multi-node-anchors): New variables. Use them. --- doc/build.scm | 305 +++++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 207 insertions(+), 98 deletions(-) (limited to 'doc') diff --git a/doc/build.scm b/doc/build.scm index 26ff577d5d..980d11ccf1 100644 --- a/doc/build.scm +++ b/doc/build.scm @@ -40,6 +40,7 @@ (gnu packages iso-codes) (gnu packages texinfo) (gnu packages tex) + (ice-9 match) (srfi srfi-19) (srfi srfi-71)) @@ -204,9 +205,168 @@ content=\"width=device-width, initial-scale=1\" />")) (setenv "XFAIL_TESTS" "htmlprag.scm") #t)))))))) +(define (normalize-language-code language) ;XXX: deduplicate + ;; Normalize LANGUAGE. For instance, "zh_CN" becomes "zh-cn". + (string-map (match-lambda + (#\_ #\-) + (chr chr)) + (string-downcase language))) + +(define* (html-manual-identifier-index manual base-url + #:key + (name "html-manual-identifier-index")) + "Return an index of all the identifiers that appear in MANUAL, a +makeinfo-generated manual. The index is a file that contains an alist; each +key is an identifier and the associated value is the URL reference pointing to +that identifier. The URL is constructed by concatenating BASE-URL to the +actual file name." + (define build + (with-extensions (list guile-lib/htmlprag-fixed) + (with-imported-modules '((guix build utils)) + #~(begin + (use-modules (guix build utils) + (htmlprag) + (srfi srfi-1) + (srfi srfi-26) + (ice-9 ftw) + (ice-9 match) + (ice-9 threads) + (ice-9 pretty-print)) + + (define file-url + (let ((prefix (string-append #$manual "/"))) + (lambda (file) + ;; Return the URL for FILE. + (let ((file (string-drop file (string-length prefix))) + (base #$base-url)) + (if (string-null? base) + file + (string-append base "/" file)))))) + + (define (underscore-decode str) + ;; Decode STR, an "underscore-encoded" string as produced by + ;; makeinfo for indexes, such as "_0025base_002dservices" for + ;; "%base-services". + (let loop ((str str) + (result '())) + (match (string-index str #\_) + (#f + (string-concatenate-reverse (cons str result))) + (index + (let ((char (string->number + (substring str (+ index 1) (+ index 5)) + 16))) + (loop (string-drop str (+ index 5)) + (append (list (string (integer->char char)) + (string-take str index)) + result))))))) + + (define (anchor-id->key id) + ;; Convert ID, an anchor ID such as + ;; "index-pam_002dlimits_002dservice" to the corresponding key, + ;; "pam-limits-service" in this example. Drop the suffix of + ;; duplicate anchor IDs like "operating_002dsystem-1". + (let ((id (if (any (cut string-suffix? <> id) + '("-1" "-2" "-3" "-4" "-5")) + (string-drop-right id 2) + id))) + (underscore-decode + (string-drop id (string-length "index-"))))) + + (define* (collect-anchors file #:optional (anchors '())) + ;; Collect the anchors that appear in FILE, a makeinfo-generated + ;; file. Grab those from
tags, which corresponds to + ;; Texinfo @deftp, @defvr, etc. Return ANCHORS augmented with + ;; more name/reference pairs. + (define string-or-entity? + (match-lambda + ((? string?) #t) + (('*ENTITY* _ ...) #t) + (_ #f))) + + (define (worthy-entry? lst) + ;; Attempt to match: + ;; Scheme Variable: x + ;; but not: + ;; cups-configuration parameter: … + (let loop ((lst lst)) + (match lst + (((? string-or-entity?) rest ...) + (loop rest)) + ((('strong _ ...) _ ...) + #t) + (_ #f)))) + + (let ((shtml (call-with-input-file file html->shtml))) + (let loop ((shtml shtml) + (anchors anchors)) + (match shtml + (('dt ('@ ('id id)) rest ...) + (if (and (string-prefix? "index-" id) + (worthy-entry? rest)) + (alist-cons (anchor-id->key id) + (string-append (file-url file) + "#" id) + anchors) + anchors)) + ((tag ('@ _ ...) body ...) + (fold loop anchors body)) + ((tag body ...) + (fold loop anchors body)) + (_ anchors))))) + + (define (html-files directory) + ;; Return the list of HTML files under DIRECTORY. + (map (cut string-append directory "/" <>) + (scandir #$manual (lambda (file) + (string-suffix? ".html" file))))) + + (define anchors + (sort (concatenate + (n-par-map (parallel-job-count) + (cut collect-anchors <>) + (html-files #$manual))) + (match-lambda* + (((key1 . url1) (key2 . url2)) + (if (string=? key1 key2) + (string blocks (as produced by 'makeinfo --html')." ((? string? str) str)))) - (define (underscore-decode str) - ;; Decode STR, an "underscore-encoded" string as produced by - ;; makeinfo for indexes, such as "_0025base_002dservices" for - ;; "%base-services". - (let loop ((str str) - (result '())) - (match (string-index str #\_) - (#f - (string-concatenate-reverse (cons str result))) - (index - (let ((char (string->number - (substring str (+ index 1) (+ index 5)) - 16))) - (loop (string-drop str (+ index 5)) - (append (list (string (integer->char char)) - (string-take str index)) - result))))))) - - (define (anchor-id->key id) - ;; Convert ID, an anchor ID such as - ;; "index-pam_002dlimits_002dservice" to the corresponding key, - ;; "pam-limits-service" in this example. Drop the suffix of - ;; duplicate anchor IDs like "operating_002dsystem-1". - (let ((id (if (any (cut string-suffix? <> id) - '("-1" "-2" "-3" "-4" "-5")) - (string-drop-right id 2) - id))) - (underscore-decode - (string-drop id (string-length "index-"))))) - - (define* (collect-anchors file #:optional (vhash vlist-null)) - ;; Collect the anchors that appear in FILE, a makeinfo-generated - ;; file. Grab those from
tags, which corresponds to - ;; Texinfo @deftp, @defvr, etc. Return VHASH augmented with - ;; more name/reference pairs. - (define string-or-entity? - (match-lambda - ((? string?) #t) - (('*ENTITY* _ ...) #t) - (_ #f))) - - (define (worthy-entry? lst) - ;; Attempt to match: - ;; Scheme Variable: x - ;; but not: - ;; cups-configuration parameter: … - (let loop ((lst lst)) - (match lst - (((? string-or-entity?) rest ...) - (loop rest)) - ((('strong _ ...) _ ...) - #t) - (_ #f)))) - - (let ((shtml (call-with-input-file file html->shtml))) - (let loop ((shtml shtml) - (vhash vhash)) - (match shtml - (('dt ('@ ('id id)) rest ...) - (if (and (string-prefix? "index-" id) - (worthy-entry? rest)) - (vhash-cons (anchor-id->key id) - (string-append (basename file) - "#" id) - vhash) - vhash)) - ((tag ('@ _ ...) body ...) - (fold loop vhash body)) - ((tag body ...) - (fold loop vhash body)) - (_ vhash))))) - (define (process-html file anchors) ;; Parse FILE and perform syntax highlighting for its Scheme ;; snippets. Install the result to #$output. @@ -444,38 +532,59 @@ its 
 blocks (as produced by 'makeinfo --html')."
             (define (html? file stat)
               (string-suffix? ".html" file))
 
+            (define language+node-anchors
+              (match-lambda
+                ((language files ...)
+                 (cons language
+                       (fold (lambda (file vhash)
+                               (let ((alist (call-with-input-file file read)))
+                                 ;; Use 'fold-right' so that the first entry
+                                 ;; wins (e.g., "car" from "Pairs" rather than
+                                 ;; from "rnrs base" in the Guile manual).
+                                 (fold-right (match-lambda*
+                                               (((key . value) vhash)
+                                                (vhash-cons key value vhash)))
+                                             vhash
+                                             alist)))
+                             vlist-null
+                             files)))))
+
+            (define mono-node-anchors
+              ;; List of language/vhash pairs, where each vhash maps an
+              ;; identifier to the corresponding URL in a single-page manual.
+              (map language+node-anchors '#$mono-node-indexes))
+
+            (define multi-node-anchors
+              ;; Likewise for split-node manuals.
+              (map language+node-anchors '#$split-node-indexes))
+
             ;; Install a UTF-8 locale so we can process UTF-8 files.
             (setenv "GUIX_LOCPATH"
                     #+(file-append glibc-utf8-locales "/lib/locale"))
             (setlocale LC_ALL "en_US.utf8")
 
             ;; First process the mono-node 'guix.html' files.
-            (n-par-for-each (parallel-job-count)
-                            (lambda (mono)
-                              (let ((anchors (collect-anchors mono)))
-                                (process-html mono anchors)))
-                            (find-files
-                             #$input
-                             "^guix(-cookbook|)(\\.[a-zA-Z_-]+)?\\.html$"))
-
-            ;; Next process the multi-node HTML files in two phases: (1)
-            ;; collect the list of anchors, and (2) perform
-            ;; syntax-highlighting.
-            (let* ((multi   (find-files #$input "^html_node$"
-                                        #:directories? #t))
-                   (anchors (n-par-map (parallel-job-count)
-                                       (lambda (multi)
-                                         (cons multi
-                                               (fold collect-anchors vlist-null
-                                                     (find-files multi html?))))
-                                       multi)))
-              (n-par-for-each (parallel-job-count)
-                              (lambda (file)
-                                (let ((anchors (assoc-ref anchors (dirname file))))
-                                  (process-html file anchors)))
-                              (append-map (lambda (multi)
-                                            (find-files multi html?))
-                                          multi)))
+            (for-each (match-lambda
+                        ((language . anchors)
+                         (let ((files (find-files
+                                       (string-append #$input "/" language)
+                                       "^guix(-cookbook|)(\\.[a-zA-Z_-]+)?\\.html$")))
+                           (n-par-for-each (parallel-job-count)
+                                           (cut process-html <> anchors)
+                                           files))))
+                      mono-node-anchors)
+
+            ;; Process the multi-node HTML files.
+            (for-each (match-lambda
+                        ((language . anchors)
+                         (let ((files (find-files
+                                       (string-append #$input "/" language
+                                                      "/html_node")
+                                       "\\.html$")))
+                           (n-par-for-each (parallel-job-count)
+                                           (cut process-html <> anchors)
+                                           files))))
+                      multi-node-anchors)
 
             ;; Last, copy non-HTML files as is.
             (for-each copy-as-is
-- 
cgit 1.4.1


From db1d445357ad7a8221683f44ff7c66e8e408206c Mon Sep 17 00:00:00 2001
From: Ludovic Courtès 
Date: Sat, 17 Oct 2020 14:32:53 +0200
Subject: doc: Allow code snippets in the cookbook to link to the manual.

Until now, only code snippets in the manual itself would contain links
to identifier definitions.  Now snippets in the cookbook also link to
definitions in the manual.

* doc/build.scm (html-manual): Add #:mono-node-indexes and #:multi-node-indexes
and pass it to 'syntax-highlighted-html'.
(pdf+html-manual): Likewise, and pass it to 'html-manual'.
: Factorize 'version' and 'source'.  Define 'guix-manual',
'mono-node-indexes', and 'split-node-indexes'.  Pass #:mono-node-indexes
and #:split-node-indexes to 'pdf+html-manual'.
---
 doc/build.scm | 57 +++++++++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 51 insertions(+), 6 deletions(-)

(limited to 'doc')

diff --git a/doc/build.scm b/doc/build.scm
index 980d11ccf1..cc107ece59 100644
--- a/doc/build.scm
+++ b/doc/build.scm
@@ -42,6 +42,7 @@
              (gnu packages tex)
              (ice-9 match)
              (srfi srfi-19)
+             (srfi srfi-26)
              (srfi srfi-71))
 
 (define file-append*
@@ -595,6 +596,8 @@ its 
 blocks (as produced by 'makeinfo --html')."
 (define* (html-manual source #:key (languages %languages)
                       (version "0.0")
                       (manual %manual)
+                      (mono-node-indexes (map list languages))
+                      (split-node-indexes (map list languages))
                       (date 1)
                       (options %makeinfo-html-options))
   "Return the HTML manuals built from SOURCE for all LANGUAGES, with the given
@@ -683,6 +686,8 @@ makeinfo OPTIONS."
   (let* ((name   (string-append manual "-html-manual"))
          (manual (computed-file name build)))
     (syntax-highlighted-html manual
+                             #:mono-node-indexes mono-node-indexes
+                             #:split-node-indexes split-node-indexes
                              #:name (string-append name "-highlighted"))))
 
 (define* (pdf-manual source #:key (languages %languages)
@@ -1029,6 +1034,8 @@ languages:\n"
                           #:key (languages %languages)
                           (version "0.0")
                           (date (time-second (current-time time-utc)))
+                          (mono-node-indexes (map list %languages))
+                          (split-node-indexes (map list %languages))
                           (manual %manual))
   "Return the union of the HTML and PDF manuals, as well as the indexes."
   (directory-union (string-append manual "-manual")
@@ -1039,7 +1046,12 @@ languages:\n"
                                 #:version version
                                 #:manual manual))
                         (list html-manual-indexes
-                              html-manual pdf-manual))
+                              (lambda (source . args)
+                                (apply html-manual source
+                                       #:mono-node-indexes mono-node-indexes
+                                       #:split-node-indexes split-node-indexes
+                                       args))
+                              pdf-manual))
                    #:copy? #t))
 
 (define (latest-commit+date directory)
@@ -1056,19 +1068,52 @@ commit date (an integer)."
 (let* ((root (canonicalize-path
               (string-append (current-source-directory) "/..")))
        (commit date (latest-commit+date root))
+       (version (or (getenv "GUIX_MANUAL_VERSION")
+                    (string-take commit 7)))
        (select? (let ((vcs? (git-predicate root)))
                   (lambda (file stat)
                     (and (vcs? file stat)
                          ;; Filter out this file.
-                         (not (string=? (basename file) "build.scm")))))))
+                         (not (string=? (basename file) "build.scm"))))))
+       (source (local-file root "guix" #:recursive? #t
+                           #:select? select?)))
+
+  (define guix-manual
+    (html-manual source
+                 #:manual "guix"
+                 #:version version
+                 #:date date))
+
+  (define mono-node-indexes
+    ;; Alist of indexes for GUIX-MANUAL, where each key is a language code and
+    ;; each value is a file-like object containing the identifier index.
+    (html-identifier-indexes guix-manual ""
+                             #:base-url (if (string=? %manual "guix")
+                                            (const "")
+                                            (cut string-append "/manual/" <>))
+                             #:languages %languages))
+
+  (define split-node-indexes
+    ;; Likewise for the split-node variant of GUIX-MANUAL.
+    (html-identifier-indexes guix-manual "/html_node"
+                             #:base-url (if (string=? %manual "guix")
+                                            (const "")
+                                            (cut string-append "/manual/" <>
+                                                 "/html_node"))
+                             #:languages %languages))
+
   (format (current-error-port)
           "building manual from work tree around commit ~a, ~a~%"
           commit
           (let* ((time (make-time time-utc 0 date))
                  (date (time-utc->date time)))
             (date->string date "~e ~B ~Y")))
-  (pdf+html-manual (local-file root "guix" #:recursive? #t
-                               #:select? select?)
-                   #:version (or (getenv "GUIX_MANUAL_VERSION")
-                                 (string-take commit 7))
+
+  (pdf+html-manual source
+                   ;; Always use the identifier index of GUIX-MANUAL.  That
+                   ;; way, "guix-cookbook" can contain link to definitions
+                   ;; that appear in GUIX-MANUAL.
+                   #:mono-node-indexes mono-node-indexes
+                   #:split-node-indexes split-node-indexes
+                   #:version version
                    #:date date))
-- 
cgit 1.4.1


From 97ce30cc1beffdcfb26d54f6db5647993682cbad Mon Sep 17 00:00:00 2001
From: Ludovic Courtès 
Date: Sat, 17 Oct 2020 21:53:26 +0200
Subject: doc: Generate cross-references in code snippets to the Guile manual.

Occurrences of 'list', 'cons', etc. in code snippets now link to the
relevant section of the Guile manual.

* doc/build.scm (guile-manual, %guile-manual-base-url)
(guile-mono-node-indexes, guile-split-node-indexes): New variables.
(for-all-languages, merge-index-alists): New procedures.
: Rename 'mono-node-indexes' and 'split-node-indexes' to
'guix-mono-node-indexes' and 'guix-split-node-indexes' respectively.
Add 'mono-node-indexes' and 'split-node-indexes' defined as calls to
'merge-index-alists'.
---
 doc/build.scm | 100 +++++++++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 95 insertions(+), 5 deletions(-)

(limited to 'doc')

diff --git a/doc/build.scm b/doc/build.scm
index cc107ece59..dac62493f4 100644
--- a/doc/build.scm
+++ b/doc/build.scm
@@ -33,6 +33,7 @@
              (guix utils)
              (git)
              (gnu packages base)
+             (gnu packages compression)
              (gnu packages gawk)
              (gnu packages gettext)
              (gnu packages guile)
@@ -41,6 +42,7 @@
              (gnu packages texinfo)
              (gnu packages tex)
              (ice-9 match)
+             (srfi srfi-1)
              (srfi srfi-19)
              (srfi srfi-26)
              (srfi srfi-71))
@@ -1064,6 +1066,85 @@ commit date (an integer)."
     ;; TODO: Use (git describe) when it's widely available.
     (values (oid->string oid) (commit-time commit))))
 
+
+;;;
+;;; Guile manual.
+;;;
+
+(define guile-manual
+  ;; The Guile manual as HTML, including both the mono-node "guile.html" and
+  ;; the split-node "html_node" directory.
+  (let ((guile guile-3.0-latest))
+    (computed-file (string-append "guile-manual-" (package-version guile))
+                   (with-imported-modules '((guix build utils))
+                     #~(begin
+                         (use-modules (guix build utils)
+                                      (ice-9 match))
+
+                         (setenv "PATH"
+                                 (string-append #+tar "/bin:"
+                                                #+xz "/bin:"
+                                                #+texinfo "/bin"))
+                         (invoke "tar" "xf" #$(package-source guile))
+                         (mkdir-p (string-append #$output "/en/html_node"))
+
+                         (let* ((texi (find-files "." "^guile\\.texi$"))
+                                (documentation (match texi
+                                                 ((file) (dirname file)))))
+                           (with-directory-excursion documentation
+                             (invoke "makeinfo" "--html" "--no-split"
+                                     "-o" (string-append #$output
+                                                         "/en/guile.html")
+                                     "guile.texi")
+                             (invoke "makeinfo" "--html" "-o" "split"
+                                     "guile.texi")
+                             (copy-recursively
+                              "split"
+                              (string-append #$output "/en/html_node")))))))))
+
+(define %guile-manual-base-url
+  "https://www.gnu.org/software/guile/manual")
+
+(define (for-all-languages index)
+  (map (lambda (language)
+         (list language index))
+       %languages))
+
+(define guile-mono-node-indexes
+  ;; The Guile manual is only available in English so use the same index in
+  ;; all languages.
+  (for-all-languages
+   (html-manual-identifier-index (file-append guile-manual "/en")
+                                 %guile-manual-base-url
+                                 #:name "guile-html-index-en")))
+
+(define guile-split-node-indexes
+  (for-all-languages
+   (html-manual-identifier-index (file-append guile-manual "/en/html_node")
+                                 (string-append %guile-manual-base-url
+                                                "/html_node")
+                                 #:name "guile-html-index-en")))
+
+(define (merge-index-alists alist1 alist2)
+  "Merge ALIST1 and ALIST2, both of which are list of tuples like:
+
+  (LANGUAGE INDEX1 INDEX2 ...)
+
+where LANGUAGE is a string like \"en\" and INDEX1 etc. are indexes as returned
+by 'html-identifier-indexes'."
+  (let ((languages (delete-duplicates
+                    (append (match alist1
+                              (((languages . _) ...)
+                               languages))
+                            (match alist2
+                              (((languages . _) ...)
+                               languages))))))
+    (map (lambda (language)
+           (cons language
+                 (append (or (assoc-ref alist1 language) '())
+                         (or (assoc-ref alist2 language) '()))))
+         languages)))
+
 
 (let* ((root (canonicalize-path
               (string-append (current-source-directory) "/..")))
@@ -1084,24 +1165,32 @@ commit date (an integer)."
                  #:version version
                  #:date date))
 
-  (define mono-node-indexes
+  (define guix-mono-node-indexes
     ;; Alist of indexes for GUIX-MANUAL, where each key is a language code and
     ;; each value is a file-like object containing the identifier index.
     (html-identifier-indexes guix-manual ""
+                             #:manual-name "guix"
                              #:base-url (if (string=? %manual "guix")
                                             (const "")
                                             (cut string-append "/manual/" <>))
                              #:languages %languages))
 
-  (define split-node-indexes
+  (define guix-split-node-indexes
     ;; Likewise for the split-node variant of GUIX-MANUAL.
     (html-identifier-indexes guix-manual "/html_node"
+                             #:manual-name "guix"
                              #:base-url (if (string=? %manual "guix")
                                             (const "")
                                             (cut string-append "/manual/" <>
                                                  "/html_node"))
                              #:languages %languages))
 
+  (define mono-node-indexes
+    (merge-index-alists guix-mono-node-indexes guile-mono-node-indexes))
+
+  (define split-node-indexes
+    (merge-index-alists guix-split-node-indexes guile-split-node-indexes))
+
   (format (current-error-port)
           "building manual from work tree around commit ~a, ~a~%"
           commit
@@ -1110,9 +1199,10 @@ commit date (an integer)."
             (date->string date "~e ~B ~Y")))
 
   (pdf+html-manual source
-                   ;; Always use the identifier index of GUIX-MANUAL.  That
-                   ;; way, "guix-cookbook" can contain link to definitions
-                   ;; that appear in GUIX-MANUAL.
+                   ;; Always use the identifier indexes of GUIX-MANUAL and
+                   ;; GUILE-MANUAL.  Both "guix" and "guix-cookbook" can
+                   ;; contain links to definitions that appear in either of
+                   ;; these two manuals.
                    #:mono-node-indexes mono-node-indexes
                    #:split-node-indexes split-node-indexes
                    #:version version
-- 
cgit 1.4.1


From f54149062e769a493207033514401d4f3890bded Mon Sep 17 00:00:00 2001
From: Ludovic Courtès 
Date: Sat, 17 Oct 2020 22:15:07 +0200
Subject: doc: cookbook: Balance parens in custom kernel examples.

* doc/guix-cookbook.texi (Customizing the Kernel): Add missing paren in
example.  Remove extra parens in other example.
---
 doc/guix-cookbook.texi | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

(limited to 'doc')

diff --git a/doc/guix-cookbook.texi b/doc/guix-cookbook.texi
index eb73bd33d6..581b8c3595 100644
--- a/doc/guix-cookbook.texi
+++ b/doc/guix-cookbook.texi
@@ -1435,7 +1435,7 @@ the @code{make-linux-libre} package definition:
       (begin
         (copy-file config ".config")
         (chmod ".config" #o666))
-      (invoke "make" ,defconfig))
+      (invoke "make" ,defconfig)))
 @end lisp
 
 Below is a sample kernel package.  The @code{linux-libre} package is nothing
@@ -1501,7 +1501,7 @@ And in the custom configure script from the `make-linux-libre` package:
   (display extra-configuration port)
   (close-port port))
 
-(invoke "make" "oldconfig"))))
+(invoke "make" "oldconfig")
 @end lisp
 
 So by not providing a configuration-file the @file{.config} starts blank, and
-- 
cgit 1.4.1


From 62fc3e375ce5c8a35c488bc67a4dd38773f5b320 Mon Sep 17 00:00:00 2001
From: Aniket Patil 
Date: Fri, 16 Oct 2020 18:18:22 +0530
Subject: doc: Fix typo.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

* doc/guix.texi (Invoking guix import): Fix typo.

Signed-off-by: Ludovic Courtès 
---
 doc/guix.texi | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

(limited to 'doc')

diff --git a/doc/guix.texi b/doc/guix.texi
index d22bd41bbb..130985d30f 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -10400,7 +10400,7 @@ package expressions for all those packages that are not yet in Guix.
 
 When @option{--archive=bioconductor} is added, metadata is imported from
 @uref{https://www.bioconductor.org/, Bioconductor}, a repository of R
-packages for for the analysis and comprehension of high-throughput
+packages for the analysis and comprehension of high-throughput
 genomic data in bioinformatics.
 
 Information is extracted from the @file{DESCRIPTION} file contained in the
-- 
cgit 1.4.1