* Build hook documentation.

* nix-store options.

5 files changed, 348 insertions, 289 deletions

diff --git a/doc/manual/build-farm.xml b/doc/manual/build-farm.xml
index 4745d205c5..c86e19cdf3 100644
--- a/doc/manual/build-farm.xml
+++ b/doc/manual/build-farm.xml
@@ -65,7 +65,10 @@ will call whenever it wants to build a derivation.  The build hook
 will perform it in the usual way if possible, or it can accept it, in
 which case it is responsible for somehow getting the inputs of the
 build to another machine, doing the build there, and getting the
-results back.</para>
+results back.  The details of the build hook protocol are described in
+the documentation of the <link
+linkend="envar-build-hook"><envar>NIX_BUILD_HOOK</envar>
+variable</link>.</para>
 
 <example id='ex-remote-systems'><title>Remote machine configuration:
 <filename>remote-systems.conf</filename></title>
diff --git a/doc/manual/env-common.xml b/doc/manual/env-common.xml
index cf84dd0549..ec468eba1a 100644
--- a/doc/manual/env-common.xml
+++ b/doc/manual/env-common.xml
@@ -118,6 +118,158 @@ $ mount -o bind /mnt/otherdisk/nix /nix</screen>
 </varlistentry>
 
 
+<varlistentry id="envar-build-hook"><term><envar>NIX_BUILD_HOOK</envar></term>
+
+  <listitem>
+
+  <para>Specifies the location of the <emphasis>build hook</emphasis>,
+  which is a program (typically some script) that Nix will call
+  whenever it wants to build a derivation.  This is used to implement
+  distributed builds (see <xref linkend="sec-distributed-builds"
+  />).  The protocol by which the calling Nix process and the build
+  hook communicate is as follows.</para>
+
+  <para>The build hook is called with the following command-line
+  arguments:
+
+  <orderedlist>
+
+    <listitem><para>A boolean value <literal>0</literal> or
+    <literal>1</literal> specifying whether Nix can locally execute
+    more builds, as per the <link
+    linkend="opt-max-jobs"><option>--max-jobs</option> option</link>.
+    The purpose of this argument is to allow the hook to not have to
+    maintain bookkeeping for the local machine.</para></listitem>
+
+    <listitem><para>The Nix platform identifier for the local machine
+    (e.g., <literal>i686-linux</literal>).</para></listitem>
+
+    <listitem><para>The Nix platform identifier for the derivation,
+    i.e., its <link linkend="attr-system"><varname>system</varname>
+    attribute</link>.</para></listitem>
+
+    <listitem><para>The store path of the derivation.</para></listitem>
+
+  </orderedlist>
+
+  </para>
+
+  <para>On the basis of this information, and whatever persistent
+  state the build hook keeps about other machines and their current
+  load, it has to decide what to do with the build.  It should print
+  out on file descriptor 3 one of the following responses (terminated
+  by a newline, <literal>"\n"</literal>):
+
+  <variablelist>
+
+    <varlistentry><term><literal>decline</literal></term>
+
+      <listitem><para>The build hook is not willing or able to perform
+      the build; the calling Nix process should do the build itself,
+      if possible.</para></listitem>
+
+    </varlistentry>
+
+    <varlistentry><term><literal>postpone</literal></term>
+
+      <listitem><para>The build hook cannot perform the build now, but
+      can do so in the future (e.g., because all available build slots
+      on remote machines are in use).  The calling Nix process should
+      postpone this build until at least one currently running build
+      has terminated.</para></listitem>
+
+    </varlistentry>
+
+    <varlistentry><term><literal>accept</literal></term>
+
+      <listitem><para>The build hook has accepted the
+      build.</para></listitem>
+
+    </varlistentry>
+
+  </variablelist>
+
+  </para>
+
+  <para>If the build hook accepts the build, it is possible that it is
+  no longer necessary to do the build because some other process has
+  performed the build in the meantime.  To prevent races, the hook
+  must read from file descriptor 4 a single line that tells it whether
+  to continue:
+
+  <variablelist>
+
+    <varlistentry><term><literal>cancel</literal></term>
+
+      <listitem><para>The build has already been done, so the hook
+      should exit.</para></listitem>
+
+    </varlistentry>
+  
+    <varlistentry><term><literal>okay</literal></term>
+
+      <listitem><para>The hook should proceed with the build.  At this
+      point, the calling Nix process has acquired locks on the output
+      path, so no other Nix process will perform the
+      build.</para></listitem>
+
+    </varlistentry>
+
+  </variablelist>
+
+  </para>
+
+  <para>If the hook has been told to proceed, Nix will store in the
+  hook’s current directory a number of text files that contain
+  information about the derivation:
+
+  <variablelist>
+
+    <varlistentry><term><filename>inputs</filename></term>
+
+      <listitem><para>The set of store paths that are inputs to the
+      build process (one per line).  These have to be copied
+      <emphasis>to</emphasis> the remote machine (in addition to the
+      store derivation itself).</para></listitem>
+
+    </varlistentry>
+  
+    <varlistentry><term><filename>outputs</filename></term>
+
+      <listitem><para>The set of store paths that are outputs of the
+      derivation (one per line).  These have to be copied
+      <emphasis>from</emphasis> the remote machine if the build
+      succeeds.</para></listitem>
+
+    </varlistentry>
+
+    <varlistentry><term><filename>references</filename></term>
+
+      <listitem><para>The reference graph of the inputs, in the format
+      accepted by the command <link
+      linkend="rsec-nix-store-reg-val"><command>nix-store
+      --register-validity</command></link>.  It is necessary to run
+      this command on the remote machine after copying the inputs to
+      inform Nix on the remote machine that the inputs are valid
+      paths.</para></listitem>
+
+    </varlistentry>
+
+  </variablelist>
+
+  </para>
+
+  <para>The hook should copy the inputs to the remote machine,
+  register the validity of the inputs, perform the remote build, and
+  copy the outputs back to the local machine.  An exit code other than
+  <literal>0</literal> indicates that the hook has failed.</para>
+
+  </listitem>
+
+
+</varlistentry>
+
+
 </variablelist>
 
 </sect1>
diff --git a/doc/manual/nix-store.xml b/doc/manual/nix-store.xml
index ac803ce5ff..d9ef17e26e 100644
--- a/doc/manual/nix-store.xml
+++ b/doc/manual/nix-store.xml
@@ -262,321 +262,225 @@ $ nix-store --gc</screen>
   
 <!--######################################################################-->
 
-  <refsection>
-    <title>Operation <option>--query</option></title>
-
-    <refsection>
-      <title>Synopsis</title>
-      <cmdsynopsis>
-        <command>nix-store</command>
-        <group choice='req'>
-          <arg choice='plain'><option>--query</option></arg>
-          <arg choice='plain'><option>-q</option></arg>
-        </group>
-        <group choice='req'>
-          <arg choice='plain'><option>--list</option></arg>
-          <arg choice='plain'><option>-l</option></arg>
-          <arg choice='plain'><option>--requisites</option></arg>
-          <arg choice='plain'><option>-R</option></arg>
-          <arg choice='plain'><option>--predecessors</option></arg>
-          <arg choice='plain'><option>--graph</option></arg>
-        </group>
-        <arg><option>--normalise</option></arg>
-        <arg><option>-n</option></arg>
-        <arg><option>--force-realise</option></arg>
-        <arg><option>-f</option></arg>
-        <arg choice='plain' rep='repeat'><replaceable>args</replaceable></arg>
-      </cmdsynopsis>
-    </refsection>
-
-    <refsection>
-      <title>Description</title>
+<refsection><title>Operation <option>--query</option></title>
+
+<refsection><title>Synopsis</title>
+
+<cmdsynopsis>
+  <command>nix-store</command>
+  <group choice='req'>
+    <arg choice='plain'><option>--query</option></arg>
+    <arg choice='plain'><option>-q</option></arg>
+  </group>
+  <group choice='req'>
+    <arg choice='plain'><option>--list</option></arg>
+    <arg choice='plain'><option>-l</option></arg>
+    <arg choice='plain'><option>--requisites</option></arg>
+    <arg choice='plain'><option>-R</option></arg>
+    <arg choice='plain'><option>--predecessors</option></arg>
+    <arg choice='plain'><option>--graph</option></arg>
+  </group>
+  <arg><option>--normalise</option></arg>
+  <arg><option>-n</option></arg>
+  <arg><option>--force-realise</option></arg>
+  <arg><option>-f</option></arg>
+  <arg choice='plain' rep='repeat'><replaceable>args</replaceable></arg>
+</cmdsynopsis>
+
+</refsection>
+
+
+<refsection><title>Description</title>
             
-      <para>
-        The operation <option>--query</option> displays various bits
-        of information about store expressions or store paths.  The
-        queries are described below.  At most one query can be
-        specified.  The default query is <option>--list</option>.
-      </para>
-
-    </refsection>
-
-
-    <refsection>
-      <title>Common query options</title>
-
-      <variablelist>
-
-        <varlistentry>
-          <term><option>--normalise</option> / <option>-n</option></term>
-          <listitem>
-            <para>
-              For those queries that take a Nix store expression, this
-              option causes those expressions to be normalised first.
-            </para>
-          </listitem>
-        </varlistentry>
-
-        <varlistentry>
-          <term><option>--force-realise</option> / <option>-f</option></term>
-          <listitem>
-            <para>
-              For those queries that take a Nix store expression, this
-              option causes those expressions to be realised first.
-              This is just a short-cut for the common idiom
-            </para>
-            <screen>
-nix-store --realise /nix/store/bla.store
-x=`nix-store --query --normalise /nix/store/bla.store`
-<emphasis>(do something with the path $x</emphasis></screen>
-            <para>
-              which using this flag can be written as
-            </para>
-            <screen>
-x=`nix-store --query --normalise --force-realise /nix/store/bla.store`
-<emphasis>(do something with the path $x</emphasis></screen>
-          </listitem>
-        </varlistentry>
-
-      </variablelist>
+<para>The operation <option>--query</option> displays various bits of
+information about store paths.  The queries are described below.  At
+most one query can be specified.  The default query is
+<option>--list</option>.</para>
+
+</refsection>
+
+
+<refsection><title>Common query options</title>
+
+<variablelist>
+
+  <varlistentry><term><option>--use-output</option> / <option>-u</option></term>
+  
+    <listitem><para>For each argument to the query that is a store
+    derivation, apply the query to the output path of the derivation
+    instead.</para></listitem>
+    
+  </varlistentry>
+
+  <varlistentry><term><option>--force-realise</option> / <option>-f</option></term>
+  
+    <listitem><para>Realise each argument to the query first (see
+    <link linkend="rsec-nix-store-realise"><command>nix-store
+    --realise</command></link>).</para></listitem>
+
+  </varlistentry>
+
+</variablelist>
         
-    </refsection>
+</refsection>
     
 
-    <refsection id='nixref-queries'>
-      <title>Queries</title>
+<refsection id='nixref-queries'><title>Queries</title>
             
-      <variablelist>
-
-        <varlistentry>
-          <term><option>--list</option> / <option>-l</option></term>
-          <listitem>
-            <para>
-              Prints out the <emphasis>output paths</emphasis> of the
-              store expressions indicated by the identifiers
-              <replaceable>args</replaceable>.  In the case of a
-              derivation expression, these are the paths that will be
-              produced when the derivation is realised.  In the case
-              of a closure expression, these are the paths that were
-              produced the derivation expression of which the closure
-              expression is a successor.
-            </para>
-          </listitem>
-        </varlistentry>
-
-        <varlistentry>
-          <term><option>--requisites</option> / <option>-R</option></term>
-          <listitem>
-            <para>
-              Prints out the requisite paths of the store expressions
-              indicated by the identifiers
-              <replaceable>args</replaceable>.  The requisite paths of
-              a Nix expression are the paths that need to be present
-              in the system to be able to realise the expression.
-              That is, they form the <emphasis>closure</emphasis> of
-              the expression in the file system (i.e., no path in the
-              set of requisite paths points to anything outside the
-              set of requisite paths).
-            </para>
-
-            <para>
-              The notion of requisite paths is very useful when one
-              wants to distribute store expressions.  Since they form a
-              closure, they are the only paths one needs to distribute
-              to another system to be able to realise the expression
-              on the other system.
-            </para>
-
-            <para>
-              This query is generally used to implement various kinds
-              of deployment.  A <emphasis>source deployment</emphasis>
-              is obtained by distributing the requisite paths of a
-              derivation expression.  A <emphasis>binary
-              deployment</emphasis> is obtained by distributing the
-              requisite paths of a closure expression.  A
-              <emphasis>cache deployment</emphasis> is obtained by
-              distributing the requisite paths of a derivation
-              expression and specifying the option
-              <option>--include-successors</option>.  This will
-              include not just the paths of a source and binary
-              deployment, but also all expressions and paths of
-              subterms of the source.  This is useful if one wants to
-              realise on the target system a Nix expression that is
-              similar but not quite the same as the one being
-              distributed, since any common subterms will be reused.
-            </para>
-
-            <para>
-              This query has a number of options:
-            </para> 
-
-            <variablelist>
-
-              <varlistentry>
-                <term><option>--exclude-exprs</option></term>
-                <listitem>
-                  <para>
-                    Excludes the paths of store expressions.  This
-                    causes the closure property to be lost, that is,
-                    the resulting set of paths is not enough to ensure
-                    realisibility.
-                  </para>
-                </listitem>
-              </varlistentry>
-
-              <varlistentry>
-                <term><option>--include-successors</option></term>
-                <listitem>
-                  <para>
-                    Also include the requisites of successors (normal forms).
-                    Only the requisites of <emphasis>known</emphasis>
-                    successors are included, i.e., the normal forms of
-                    derivation expressions that have never been normalised will
-                    not be included.
-                  </para>
-
-                  <para>
-                    Note that not just the successor of a derivation expression
-                    will be included, but also the successors of all input
-                    expressions of that derivation expression.  I.e., all
-                    normal forms of subterms involved in the normalisation of
-                    the top-level term are included.
-                  </para>
-                </listitem>
-              </varlistentry>
-
-            </variablelist>
-
-          </listitem>
-        </varlistentry>
-
-        <varlistentry>
-          <term><option>--predecessors</option></term>
-          <listitem>
-            <para>
-              For each store expression stored at paths
-              <replaceable>args</replaceable>, prints its
-              <emphasis>predecessors</emphasis>.  A derivation
-              expression <varname>p</varname> is a predecessor of a
-              store expression <varname>q</varname> iff
-              <varname>q</varname> is a successor of
-              <varname>p</varname>.
-            </para>
-          </listitem>
-        </varlistentry>
-
-        <varlistentry>
-          <term><option>--graph</option></term>
-          <listitem>
-            <para>
-              Prints a graph of the closure of the store expressions
-              identified by <replaceable>args</replaceable> in the
-              format of the <command>dot</command> tool of AT&amp;T's
-              GraphViz package.
-            </para>
-          </listitem>
-        </varlistentry>
-
-      </variablelist>
-
-    </refsection>
-
-  </refsection>
+<variablelist>
+
+  <varlistentry><term><option>--list</option> / <option>-l</option></term>
+
+    <listitem><para>Prints out the <emphasis>output paths</emphasis>
+    of the store expressions indicated by the identifiers
+    <replaceable>args</replaceable>.  In the case of a derivation
+    expression, these are the paths that will be produced when the
+    derivation is realised.  In the case of a closure expression,
+    these are the paths that were produced the derivation expression
+    of which the closure expression is a successor.</para></listitem>
+    
+  </varlistentry>
+
+  <varlistentry><term><option>--requisites</option> / <option>-R</option></term>
+
+    <listitem><para>Prints out the requisite paths of the store
+    expressions indicated by the identifiers
+    <replaceable>args</replaceable>.  The requisite paths of a Nix
+    expression are the paths that need to be present in the system to
+    be able to realise the expression.  That is, they form the
+    <emphasis>closure</emphasis> of the expression in the file system
+    (i.e., no path in the set of requisite paths points to anything
+    outside the set of requisite paths).</para>
+
+    <para>The notion of requisite paths is very useful when one wants
+    to distribute store expressions.  Since they form a closure, they
+    are the only paths one needs to distribute to another system to be
+    able to realise the expression on the other system.</para>
+
+    <para>This query is generally used to implement various kinds of
+    deployment.  A <emphasis>source deployment</emphasis> is obtained
+    by distributing the requisite paths of a derivation expression.  A
+    <emphasis>binary deployment</emphasis> is obtained by distributing
+    the requisite paths of a closure expression.  A <emphasis>cache
+    deployment</emphasis> is obtained by distributing the requisite
+    paths of a derivation expression and specifying the option
+    <option>--include-successors</option>.  This will include not just
+    the paths of a source and binary deployment, but also all
+    expressions and paths of subterms of the source.  This is useful
+    if one wants to realise on the target system a Nix expression that
+    is similar but not quite the same as the one being distributed,
+    since any common subterms will be reused.</para>
+
+    <para>This query has one option:</para>
+
+    <variablelist>
+
+      <varlistentry><term><option>--include-outputs</option></term>
+      
+        <listitem><para>Also include the output path of store
+        derivations, and their closures.</para></listitem>
+        
+      </varlistentry>
+
+    </variablelist>
+
+    </listitem>
+
+  </varlistentry>
+
+
+  <varlistentry><term><option>--graph</option></term>
+  
+    <listitem><para>Prints a graph of the closure of the store
+    expressions identified by <replaceable>args</replaceable> in the
+    format of the <command>dot</command> tool of AT&amp;T's GraphViz
+    package.</para></listitem>
+
+  </varlistentry>
+
+</variablelist>
+
+</refsection>
+
+
+</refsection>
 
   
 
-  <!--######################################################################-->
+<!--######################################################################-->
 
-  <refsection>
-    <title>Operation <option>--successor</option></title>
+<refsection id="rsec-nix-store-reg-val"><title>Operation <option>--register-validity</option></title>
 
-    <refsection>
-      <title>Synopsis</title>
-      <cmdsynopsis>
-        <command>nix-store</command>
-        <arg choice='req'><option>--successor</option></arg>
-        <arg choice='plain'
-        rep='repeat'><replaceable>srcpath</replaceable> <replaceable>sucpath</replaceable></arg>
-      </cmdsynopsis>
-    </refsection>
+<refsection><title>Synopsis</title>
 
-    <refsection>
-      <title>Description</title>
+<cmdsynopsis>
+  <command>nix-store</command>
+  <arg choice='req'><option>--register-validity</option></arg>
+</cmdsynopsis>
+</refsection>
+
+<refsection><title>Description</title>
             
-      <para>
-        The operation <option>--successor</option> registers that the
-        closure expression in <replaceable>sucpath</replaceable> is a
-        successor of the derivation expression in
-        <replaceable>srcpath</replaceable>.  This is used to implement
-        binary deployment.
-      </para>
-
-    </refsection>
+<para>TODO</para>
+
+</refsection>
             
-  </refsection>
+</refsection>
 
 
 
-  <!--######################################################################-->
+<!--######################################################################-->
 
-  <refsection>
-    <title>Operation <option>--substitute</option></title>
+<refsection><title>Operation <option>--substitute</option></title>
 
-    <refsection>
-      <title>Synopsis</title>
-      <cmdsynopsis>
-        <command>nix-store</command>
-        <arg choice='req'><option>--substitute</option></arg>
-        <arg choice='plain'
-        rep='repeat'><replaceable>srcpath</replaceable> <replaceable>subpath</replaceable></arg>
-      </cmdsynopsis>
-    </refsection>
+<refsection><title>Synopsis</title>
 
-    <refsection>
-      <title>Description</title>
+<cmdsynopsis>
+  <command>nix-store</command>
+  <arg choice='req'><option>--substitute</option></arg>
+  <arg choice='plain'
+       rep='repeat'><replaceable>srcpath</replaceable> <replaceable>subpath</replaceable></arg>
+</cmdsynopsis>
+</refsection>
+
+<refsection><title>Description</title>
             
-      <para>
-        The operation <option>--substitute</option> registers that the
-        store path <replaceable>srcpath</replaceable> can be built by
-        realising the derivation expression in
-        <replaceable>subpath</replaceable>.  This is used to implement
-        binary deployment.
-      </para>
-
-    </refsection>
+<para>The operation <option>--substitute</option> registers that the
+store path <replaceable>srcpath</replaceable> can be built by
+realising the derivation expression in
+<replaceable>subpath</replaceable>.  This is used to implement binary
+deployment.</para>
+
+</refsection>
             
-  </refsection>
+</refsection>
 
 
 
-  <!--######################################################################-->
+<!--######################################################################-->
 
-  <refsection>
-    <title>Operation <option>--verify</option></title>
+<refsection><title>Operation <option>--verify</option></title>
 
-    <refsection>
-      <title>Synopsis</title>
-      <cmdsynopsis>
-        <command>nix-store</command>
-        <arg choice='req'><option>--verify</option></arg>
-      </cmdsynopsis>
-    </refsection>
+<refsection>
+  <title>Synopsis</title>
+  <cmdsynopsis>
+    <command>nix-store</command>
+    <arg choice='req'><option>--verify</option></arg>
+  </cmdsynopsis>
+</refsection>
 
-    <refsection>
-      <title>Description</title>
+<refsection><title>Description</title>
             
-      <para>
-        The operation <option>--verify</option> verifies the internal
-        consistency of the Nix database, and the consistency between
-        the Nix database and the Nix store.  Any inconsistencies
-        encountered are automatically repaired.  Inconsistencies are
-        generally the result of the Nix store or database being
-        modified by non-Nix tools, or of bugs in Nix itself.
-      </para>
-
-    </refsection>
+<para>The operation <option>--verify</option> verifies the internal
+consistency of the Nix database, and the consistency between the Nix
+database and the Nix store.  Any inconsistencies encountered are
+automatically repaired.  Inconsistencies are generally the result of
+the Nix store or database being modified by non-Nix tools, or of bugs
+in Nix itself.</para>
+
+</refsection>
             
-  </refsection>
+</refsection>
 
 
 
diff --git a/doc/manual/opt-common.xml b/doc/manual/opt-common.xml
index a8f2521280..fb35c1a743 100644
--- a/doc/manual/opt-common.xml
+++ b/doc/manual/opt-common.xml
@@ -84,7 +84,7 @@
 </varlistentry>
 
 
-<varlistentry><term><option>--max-jobs</option> / <option>-j</option></term>
+<varlistentry id="opt-max-jobs"><term><option>--max-jobs</option> / <option>-j</option></term>
 
   <listitem><para>Sets the maximum number of build jobs that Nix will
   perform in parallel to the specified number.  The default is 1.  A
diff --git a/doc/manual/writing-nix-expressions.xml b/doc/manual/writing-nix-expressions.xml
index 50cbc4249d..1b177c7dab 100644
--- a/doc/manual/writing-nix-expressions.xml
+++ b/doc/manual/writing-nix-expressions.xml
@@ -1112,7 +1112,7 @@ set, the attributes of which specify the inputs of the build.</para>
 
 <itemizedlist>
 
-  <listitem><para>There must be an attribute named
+  <listitem id="attr-system"><para>There must be an attribute named
   <varname>system</varname> whose value must be a string specifying a
   Nix platform identifier, such as <literal>"i686-linux"</literal> or
   <literal>"powerpc-darwin"</literal><footnote><para>To figure out