Update nix-push manpage and document the binary cache format

2 files changed, 341 insertions, 77 deletions

diff --git a/doc/manual/glossary.xml b/doc/manual/glossary.xml
index efbf96f0af..d74940c90b 100644
--- a/doc/manual/glossary.xml
+++ b/doc/manual/glossary.xml
@@ -160,6 +160,18 @@
 </glossentry>
 
 
+<glossentry xml:id="gloss-nar"><glossterm>NAR</glossterm>
+
+  <glossdef><para>A <emphasis>N</emphasis>ix
+  <emphasis>AR</emphasis>chive.  This is a serialisation of a path in
+  the Nix store.  It can contain regular files, directories and
+  symbolic links.  NARs are generated and unpacked using
+  <command>nix-store --dump</command> and <command>nix-store
+  --restore</command>.</para></glossdef>
+
+</glossentry>
+
+
 
 </glosslist>
 
diff --git a/doc/manual/nix-push.xml b/doc/manual/nix-push.xml
index d2a7e063e5..170b6f55ab 100644
--- a/doc/manual/nix-push.xml
+++ b/doc/manual/nix-push.xml
@@ -12,24 +12,19 @@
 
 <refnamediv>
   <refname>nix-push</refname>
-  <refpurpose>push store paths onto a network cache</refpurpose>
+  <refpurpose>generate a binary cache</refpurpose>
 </refnamediv>
 
 <refsynopsisdiv>
   <cmdsynopsis>
     <command>nix-push</command>
-    <group choice='req'>
-      <arg choice='req'>
-        <arg choice='plain'><replaceable>archivesPutURL</replaceable></arg>
-        <arg choice='plain'><replaceable>archivesGetURL</replaceable></arg>
-        <arg choice='plain'><replaceable>manifestPutURL</replaceable></arg>
-      </arg>
-      <arg choice='req'>
-        <arg choice='plain'><option>--copy</option></arg>
-        <arg choice='plain'><replaceable>archivesDir</replaceable></arg>
-        <arg choice='plain'><replaceable>manifestFile</replaceable></arg>
-      </arg>
-    </group>
+    <arg choice='plain'><option>--dest</option> <replaceable>dest-dir</replaceable></arg>
+    <arg><option>--bzip2</option></arg>
+    <arg><option>--force</option></arg>
+    <arg><option>--link</option></arg>
+    <arg><option>--manifest</option></arg>
+    <arg><option>--manifest-path</option> <replaceable>filename</replaceable></arg>
+    <arg><option>--url-prefix</option> <replaceable>url</replaceable></arg>
     <arg choice='plain' rep='repeat'><replaceable>paths</replaceable></arg>
   </cmdsynopsis>
 </refsynopsisdiv>
@@ -37,93 +32,350 @@
 
 <refsection><title>Description</title>
 
-<para>The command <command>nix-push</command> builds a set of store
-paths (if necessary), and then packages and uploads all store paths in
-the resulting closures to a server.  A network cache thus populated
-can subsequently be used to speed up software deployment on other
-machines using the <command>nix-pull</command> command.</para>
+<para>The command <command>nix-push</command> produces a
+<emphasis>binary cache</emphasis>, a directory containing compressed
+Nix archives (NARs) plus some metadata of the closure of the specified
+store paths.  This directory can then be made available through a web
+server to other Nix installations, allowing them to skip building from
+source and instead download binaries from the cache
+automatically.</para>
 
 <para><command>nix-push</command> performs the following actions.
       
 <orderedlist>
 
   <listitem><para>Each path in <replaceable>paths</replaceable> is
-  realised (using <link
-  linkend='rsec-nix-store-realise'><literal>nix-store
-  --realise</literal></link>).</para></listitem>
+  built (using <link
+  linkend='rsec-nix-store-realise'><command>nix-store
+  --realise</command></link>).</para></listitem>
 
-  <listitem><para>All paths in the closure of the store expressions
-  stored in <replaceable>paths</replaceable> are determined (using
-  <literal>nix-store --query --requisites
-  --include-outputs</literal>).  It should be noted that since the
-  <option>--include-outputs</option> flag is used, you get a combined
-  source/binary distribution.</para></listitem>
+  <listitem><para>All paths in the closure of
+  <replaceable>paths</replaceable> are determined (using
+  <command>nix-store --query --requisites
+  --include-outputs</command>).  Note that since the
+  <option>--include-outputs</option> flag is used, if
+  <replaceable>paths</replaceable> includes a store derivation, you
+  get a combined source/binary distribution (e.g., source tarballs
+  will be included).</para></listitem>
 
   <listitem><para>All store paths determined in the previous step are
-  packaged and compressed into a <command>bzip</command>ped NAR
-  archive (extension <filename>.nar.bz2</filename>).</para></listitem>
-
-  <listitem><para>A <emphasis>manifest</emphasis> is created that
-  contains information on the store paths, their eventual URLs in the
-  cache, and cryptographic hashes of the contents of the NAR
-  archives.</para></listitem>
-
-  <listitem><para>Each store path is uploaded to the remote directory
-  specified by <replaceable>archivesPutURL</replaceable>.  HTTP PUT
-  requests are used to do this.  However, before a file
-  <varname>x</varname> is uploaded to
-  <literal><replaceable>archivesPutURL</replaceable>/</literal><varname>x</varname>,
-  <command>nix-push</command> first determines whether this upload is
-  unnecessary by issuing a HTTP HEAD request on
-  <literal><replaceable>archivesGetURL</replaceable>/</literal><varname>x</varname>.
-  This allows a cache to be shared between many partially overlapping
-  <command>nix-push</command> invocations.  (We use two URLs because
-  the upload URL typically refers to a CGI script, while the download
-  URL just refers to a file system directory on the
-  server.)</para></listitem>
-
-  <listitem><para>The manifest is uploaded using an HTTP PUT request
-  to <replaceable>manifestPutURL</replaceable>.  The corresponding
-  URL to download the manifest can then be used by
-  <command>nix-pull</command>.</para></listitem>
-        
+  packaged into a NAR (using <command>nix-store --dump</command>) and
+  compressed using <command>xz</command> or <command>bzip2</command>.
+  The resulting files have the extension <filename>.nar.xz</filename>
+  or <filename>.nar.bz2</filename>.  Also for each store path, Nix
+  generates a file with extension <filename>.narinfo</filename>
+  containing metadata such as the references, cryptographic hash and
+  size of each path.</para></listitem>
+
+  <listitem><para>Optionally, a single <emphasis>manifest</emphasis>
+  file is created that contains the same metadata as the
+  <filename>.narinfo</filename> files.  This is for compatibility with
+  Nix versions prior to 1.2 (see <command>nix-pull</command> for
+  details).</para></listitem>
+
+  <listitem><para>A file named <option>nix-cache-info</option> is
+  placed in the destination directory.  The existence of this file
+  marks the directory as a binary cache.</para></listitem>
+
 </orderedlist>
 
 </para>
 
-<!--
-<para>TODO:  <option>- -copy</option></para>
--->
-            
+</refsection>
+
+
+<refsection><title>Options</title>
+
+<variablelist>
+
+  <varlistentry><term><option>--dest</option> <replaceable>dest-dir</replaceable></term>
+
+    <listitem><para>Set the destination directory to
+    <replaceable>dir</replaceable>, which is created if it does not
+    exist.  This flag is required.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><option>--bzip2</option></term>
+
+    <listitem><para>Compress NARs using <command>bzip2</command>
+    instead of <command>xz -9</command>.  The latter compresses about
+    30% better on typical archives, decompresses about twice as fast,
+    but compresses a lot slower and is not supported by Nix prior to
+    version 1.2.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><option>--force</option></term>
+
+    <listitem><para>Overwrite <filename>.narinfo</filename> files if
+    they already exist.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><option>--link</option></term>
+
+    <listitem><para>By default, NARs are generated in the Nix store
+    and then copied to <replaceable>dest-dir</replaceable>.  If this
+    option is given, hard links are used instead.  This only works if
+    <replaceable>dest-dir</replaceable> is on the same filesystem as
+    the Nix store.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><option>--manifest</option></term>
+
+    <listitem><para>Force the generation of a manifest suitable for
+    use by <command>nix-pull</command>.  The manifest is stored as
+    <filename><replaceable>dest-dir</replaceable>/MANIFEST</filename>.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><option>--manifest-path</option> <replaceable>filename</replaceable></term>
+
+    <listitem><para>Like <option>--manifest</option>, but store the
+    manifest in <replaceable>filename</replaceable>.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><option>--url-prefix</option> <replaceable>url</replaceable></term>
+
+    <listitem><para>Manifests are expected to contain the absolute
+    URLs of NARs.  For generating these URLs, the prefix
+    <replaceable>url</replaceable> is used.  It defaults to
+    <uri>file://<replaceable>dest-dir</replaceable></uri>.</para></listitem>
+
+  </varlistentry>
+
+</variablelist>
+
 </refsection>
 
 
 <refsection><title>Examples</title>
 
-<para>To upload files there typically is some CGI script on the server
-side.  This script should be be protected with a password.  The
-following example uploads the store paths resulting from building the
-Nix expressions in <filename>foo.nix</filename>, passing appropriate
-authentication information:
-    
+<para>To add the closure of Thunderbird to a binary cache:
+
 <screen>
-$ nix-push \
-    http://foo@bar:server.domain/cgi-bin/upload.pl/cache \
-    http://server.domain/cache \
-    http://foo@bar:server.domain/cgi-bin/upload.pl/MANIFEST \
-    $(nix-instantiate foo.nix)</screen>
+$ nix-push --dest /tmp/cache $(nix-build -A thunderbird)
+</screen>
 
-This will push both sources and binaries (and any build-time
-dependencies used in the build, such as compilers).</para>
+Assuming that <filename>/tmp/cache</filename> is exported by a web
+server as <uri>http://example.org/cache</uri>, you can then use this
+cache on another machine to speed up the installation of Thunderbird:
 
-<para>If we just want to push binaries, not sources and build-time
-dependencies, we can do:
-      
 <screen>
-$ nix-push <replaceable>urls</replaceable> $(nix-store -r $(nix-instantiate foo.nix))</screen>
-    
+$ nix-build -A thunderbird --option binary-caches http://example.org/cache
+</screen>
+
+Alternatively, you could add <literal>binary-caches =
+http://example.org/cache</literal> to
+<filename>nix.conf</filename>.</para>
+
+<para>To also include build-time dependencies (such as source
+tarballs):
+
+<screen>
+$ nix-push --dest /tmp/cache $(nix-instantiate -A thunderbird)
+</screen>
+
 </para>
 
+<para>To generate a manifest suitable for <command>nix-pull</command>:
+
+<screen>
+$ nix-push --dest /tmp/cache $(nix-build -A thunderbird) --manifest
+</screen>
+
+On another machine you can then do:
+
+<screen>
+$ nix-pull http://example.org/cache
+</screen>
+
+to cause the binaries to be used by subsequent Nix operations.</para>
+
 </refsection>
-    
+
+
+<refsection><title>Binary cache format and operation</title>
+
+<para>A binary cache with URL <replaceable>url</replaceable> only
+denotes a valid binary cache if the file
+<uri><replaceable>url</replaceable>/nix-cache-info</uri> exists.  If
+this file does not exist (or cannot be downloaded), the cache is
+ignored.  If it does exist, it must be a text file containing cache
+properties.  Here’s an example:
+
+<screen>
+StoreDir: /nix/store
+WantMassQuery: 1
+</screen>
+
+The properties that are currently supported are:
+
+<variablelist>
+  
+  <varlistentry><term><literal>StoreDir</literal></term>
+
+    <listitem><para>The path of the Nix store to which this binary
+    cache applies.  Binaries are not relocatable — a binary built for
+    <filename>/nix/store</filename> won’t generally work in
+    <filename>/home/alice/store</filename> — so to prevent binaries
+    from being used in a wrong store, a binary cache is only used if
+    its <literal>StoreDir</literal> matches the local Nix
+    configuration.  The default is
+    <filename>/nix/store</filename>.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>WantMassQuery</literal></term>
+
+    <listitem><para>Query operations such as <command>nix-env
+    -qas</command> can cause thousands of cache queries, and thus
+    thousands of HTTP requests, to determine which packages are
+    available in binary form.  While these requests are small, not
+    every server may appreciate a potential onslaught of queries.  If
+    <literal>WantMassQuery</literal> is set to <literal>0</literal>
+    (default), “mass queries” such as <command>nix-env -qas</command>
+    will skip this cache.  Thus a package may appear not to have a
+    binary substitute.  However, the binary will still be used when
+    you actually install the package.  If
+    <literal>WantMassQuery</literal> is set to <literal>1</literal>,
+    mass queries will use this cache.</para></listitem>
+
+  </varlistentry>
+
+</variablelist>
+
+</para>
+
+<para>Every time Nix needs to build some store path
+<replaceable>p</replaceable>, it will check each configured binary
+cache to see if it has a NAR file for <replaceable>p</replaceable>,
+until it finds one.  If no cache has a NAR, Nix will fall back to
+building the path from source (if applicable).  To see if a cache with
+URL <replaceable>url</replaceable> has a binary for
+<replaceable>p</replaceable>, Nix fetches
+<replaceable>url/h</replaceable>, where <replaceable>h</replaceable>
+is the hash part of <replaceable>p</replaceable>.  Thus, if we have a
+cache <uri>http://nixos.org/binary-cache</uri> and we want to obtain
+the store path
+<screen>
+/nix/store/cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13
+</screen>
+then Nix will attempt to fetch
+<screen>
+http://nixos.org/binary-cache/cj7a81wsm1ijwwpkks3725661h3263p5.narinfo
+</screen>
+(Commands such as <command>nix-env -qas</command> will issue an HTTP
+HEAD request, since it only needs to know if the
+<filename>.narinfo</filename> file exists.)  The
+<filename>.narinfo</filename> file is a simple text file that looks
+like this:
+
+<screen>
+StorePath: /nix/store/cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13
+URL: nar/0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21.nar.bz2
+Compression: bzip2
+FileHash: sha256:0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21
+FileSize: 10119014
+NarHash: sha256:120mhshkcgvn48xvxz0fjbd47k615v0viv7jy4gy7dwhmqapgd9d
+NarSize: 33702192
+References: cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13 jfcs9xnfbmiwqs224sb0qqsybbfl3sab-linux-headers-2.6.35.14
+Deriver: xs32zzf2d58f6l5iz5fgwxrds2q5pnvn-glibc-2.13.drv
+System: x86_64-linux
+</screen>
+
+The fields are as follows:
+
+<variablelist>
+  
+  <varlistentry><term><literal>StorePath</literal></term>
+
+    <listitem><para>The full store path, including the name part
+    (e.g., <literal>glibc-2.13</literal>).  It must match the
+    requested store path.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>URL</literal></term>
+
+    <listitem><para>The URL of the NAR, relative to the binary cache
+    URL.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>Compression</literal></term>
+
+    <listitem><para>The compression method; either
+    <literal>xz</literal> or
+    <literal>bzip2</literal>.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>FileHash</literal></term>
+
+    <listitem><para>The SHA-256 hash of the compressed
+    NAR.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>FileSize</literal></term>
+
+    <listitem><para>The size of the compressed NAR.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>NarHash</literal></term>
+
+    <listitem><para>The SHA-256 hash of the uncompressed NAR.  This is
+    equal to the hash of the store path as returned by
+    <command>nix-store -q --hash
+    <replaceable>p</replaceable></command>.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>NarSize</literal></term>
+
+    <listitem><para>The size of the uncompressed NAR.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>References</literal></term>
+
+    <listitem><para>The references of the store path, without the Nix
+    store prefix.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>Deriver</literal></term>
+
+    <listitem><para>The deriver of the store path, without the Nix
+    store prefix.  This field is optional.</para></listitem>
+
+  </varlistentry>
+
+  <varlistentry><term><literal>System</literal></term>
+
+    <listitem><para>The Nix platform type of this binary, if known.
+    This field is optional.</para></listitem>
+
+  </varlistentry>
+
+</variablelist>
+
+</para>
+
+<para>Thus, in our example, after recursively ensuring that the
+references exist (e.g.,
+<filename>/nix/store/jfcs9xnfbmiwqs224sb0qqsybbfl3sab-linux-headers-2.6.35.14</filename>),
+Nix will fetch <screen>
+http://nixos.org/binary-cache/nar/0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21.nar.bz2
+</screen> and decompress and unpack it to
+<filename>/nix/store/cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13</filename>.</para>
+
+</refsection>
+
+
 </refentry>