Sidai team: how it manages packages

ToDo:more

Contents:

• Concepts: installing, deploying, revealing
• Packaging `flow' (and associated methods)
• Un-packaging: going `backwards'
• Packaging `flows' crib sheet
• Directories used in package management
• Your site's package/ALL.xml file
• A package's status field

Concepts: installing, deploying, revealing

• Install: An `install' process produces a single set of `golden' bits for a package (on a particular platform).

  For example, in /sys/.-ark-install-sparc-solaris2.6/emacs--20.7/, you might find the single set of `emacs' bits for your SPARC boxes.

• Deploy: We deploy some set of install bits onto each (revelant) host. One more time: deployment is per-host, not per-site. Different hosts can have different deployments.

  The simplest deployment is:

  ln -s /sys/.-ark-install-sparc-solaris2.6/emacs--20.7 /our/.-ark-deploy/emacs--20.7
  

  (It can get much more fun than that...)

• Reveal: The horrible path names just above suggest (correctly) that users aren't really supposed to see installs or deployments; they are to see what is revealed. We reveal the deployed bits (not the installed bits); again, a simple reveal might be:

  ln -s /our/.-ark-deploy/emacs--20.7/bin/emacs /usr/local/bin/emacs
  

Methods to the madness:

• Why separate install from deployment?

  Basically, it means we can decide host-by-host or package-by-package whether a host gets its own copy of a package (good for resilience and efficiency) or shares a single common copy (the install bits).

• Why separate deployment from revelation?

  Among other things, it means we can have multiple versions of a package live at any one time, and then deal with this in various ways (e.g. the version-wrapper machinery).

Packaging `flow' (and associated methods)

If you are not conversant with this flow, ARK package .xml files will be a mystery. Again, remember that whatever these methods do by default can be overridden, possibly in sneaky ways. What is described here are their conventional actions.

<mkdir-build>:

Create the directory where we will build the package.

<get-source>:

If we need to fetch the source from somewhere (off the 'Net?), this method will do it.

<verify-source>:

This method, if used, verifies the integrity of the source that we have/got; e.g. check the MD5 checksums of the source tarball.

<assemble-source>:

Unpack the source.

<src-linkfarm>:

Create a `link farm' (shadow tree?) called src that points to the original source.

<patch>:

Apply any needed patches to the src link farm; note: we leave the original (pristine) source absolutely untouched.

<mkdir-install>:

Create the directory where we will install the golden set of bits for this package for this platform.

<mkdir-deploy>:

Create the deployment directory(ies) for this package on the host where we will do the build.

<host-linkfarm>:

Create another link farm (that points to the src link farm) for this particular platform. All subsequent compiling, etc., happens in this per-platform directory.

<configure>:

Run ./configure --prefix=<deploy-dir> or some equivalent, if applicable.

<depend>:

Run make depend, if applicable.

<compile>:

Run make all, or some equivalent.

<check>:

Run make check, or some equivalent.

<build>:

Exists solely so we can say ark package build <some-pkg>, and have everything get done up through make check.

<pre-install-bits>:

Do anything that needs doing so that <install-bits> has a chance of succeeding.

<install-bits>:

Run make install, or some equivalent.

<post-install-bits>:

Do any post-make install stuff that needs doing.

<create-golden-copy>:

Copy the just-created bits from the build host's deploy directory into the appropriate install directory.

<create-manifest>:

Take MD5 checksums (or something) of the lovely set of install bits we just created.

<check-install>:

If the package has any bits to check itself after it's been installed, run 'em here.

<install>:

Exists solely so we can say ark package install <some-pkg>, and have everything get done up through <check-install>.

<deploy-bits>:

Do the deployment. (This happens once per [revelant] host.)

<deploy>:

Do any after-the-deployment stuff; one common thing we do is diff the old revealed bits against the just-freshly-deployed bits, so we can see what we are just about to change. (In the case of Quite Delicate Packages -- e.g. the passwd file -- you can't be too careful.)

<reveal-bits>:

Do the revealing. (This happens once per [revelant] host.)

<reveal>:

Do any after-the-bits-have-changed stuff; for example, if you've changed /etc/syslog.conf, you'd probably want to HUP the syslog daemon.

Un-packaging: going `backwards'

Packaging `flows' crib sheet

Directories used in package management

Build directory:

For package `wibble' version 2.3, it would be /sys/ark-builds/wibble--2.3/wibble-2.3/.

Within that build directory, we would expect the pristine source to be in wibble-2.3, the source link farm (shadow tree) [where patches happen] in src, and the host link farm(s) to be in sparc-solaris2.6, ia32-linux-rh7.1, or whatever. (Or ALL if the package is entirely platform-independent.)

Install directory:

For package `wibble' version 2.3 for the sparc-solaris2.6 platform, it would be /sys/.-install-sparc-solaris2.6/wibble--2.3/.

If `wibble' were vendor-supplied package, perhaps just a huge tarball that you simply dump somewhere, we might instead choose an install directory of /sys/.-vendor-install-acme-corp/wibble--2.3/.

Deploy directory:

For package `wibble' version 2.3, it would typically be /our/.-ark-deploy/wibble--2.3, or perhaps /usr/local/.-ark-deploy/wibble--2.3.

Recall that the deploy directory is where we tell the package that it lives (--prefix=... in GNU speak).

Reveal directory:

We use /our as our main reveal directory, with `special' packages being revealed into /usr/local.

How it hangs together: A revealed /our/bin/wibble aims at a deployed /our/.-ark-deploy/wibble--2.3/bin/wibble, which in turn points to the installed /sys/.-ark-install-sparc-solaris2.6/wibble--2.3/bin/wibble.

State directory:

This is where ARK `remembers' that it has done things; for package `wibble' version 2.3, we use /sys/ark-state/package/wibble--2.3/.

Your site's package/ALL.xml file

You should have an `ALL' proto-package for your site, which all of your real packages have as a(n ultimate) prototype.

Now: ARK, in its general sense, does not mandate much about what this proto-package includes. You could do things we haven't even thought of.

That said, most current ARK users probably want to know how a standard-issue Sidai-style package/ALL.xml file works. What follows is an annotation of the `glasli1' team's ALL proto-package.

Remember that packages can-and-do override what's in ALL. What's here are the unless-you-say-otherwise defaults.

First, standard preliminaries:

<package name="ALL" xml-version="1" prototype="yes">
<description>
Say whatever you like...
</description>

<prototypes>
  <prototype team="sidai" name="ALL"/>
</prototypes>

<file-namespaces><list>
  <item>/our</item>
</list></file-namespaces>

A package can be revealed in multiple <file-namespaces>. For example, we put Perl in the usual /our namespace, but we also arrange for the main executable to appear in /usr/local/bin.

The <namespace-deploy-map> is a table that maps file namespaces (reveal areas) to deploy areas:

<namespace-deploy-map><table>
  <!-- the first one (/our) is the primary -->
  <entry name="/our">/our/.-ark-deploy</entry>
  <entry name="/usr/local">/usr/local/.-ark-deploy</entry>
  <entry name="/">/.-ark-deploy</entry>
</table></namespace-deploy-map>

<hosts-supported><list>
  <item>sparc-solaris2.6</item>
  <item>ia32-linux-rh6.1</item>
</list></hosts-supported>

<proxy-hosts><table>
  <entry name="ALL">slicker</entry>
  <entry name="ia32-linux-rh6.1">clarence</entry>
  <entry name="sparc-solaris">slicker</entry>
  <entry name="sparc-solaris-amanda-server">slimy</entry>
  <entry name="sparc-solaris-license-server">slicker</entry>
  <entry name="sparc-solaris-mail-server">slicker</entry>
  <entry name="sparc-solaris-web-server">slicker</entry>
  <entry name="sparc-solaris2.6">slicker</entry>
  <entry name="sparcv9-solaris2.6">slinger</entry>
</table></proxy-hosts>

<packages-install-dir><table>
  <entry name="ALL">/sys/.-ark-install-ALL</entry>
  <entry name="ia32-linux-rh6.1">/sys/.-ark-install-ia32-linux-rh6.1</entry>
  <entry name="sparc-solaris">/sys/.-ark-install-sparc-solaris</entry>
  <entry name="sparc-solaris-amanda-server">
    /sys/.-ark-install-sparc-solaris-amanda-server
  </entry>
  <entry name="sparc-solaris-license-server">
    /sys/.-ark-install-sparc-solaris-license-server
  </entry>
  <entry name="sparc-solaris-mail-server">
    /sys/.-ark-install-sparc-solaris-mail-server
  </entry>
  <entry name="sparc-solaris-web-server">
/sys/.-ark-install-sparc-solaris-web-server</entry>
  <entry name="sparc-solaris2.6">/sys/.-ark-install-sparc-solaris2.6</entry>
  <entry name="sparcv9-solaris2.6">/sys/.-ark-install-sparcv9-solaris2.6</entry>
</table></packages-install-dir>

<ark-dirs><table>
  <entry name="GENERATED_PKGS"> /sys/ark-gen-pkg </entry>
  <entry name="PKGS_BUILD"> /sys/ark-builds </entry>
  <entry name="PKGS_SRC"> /d/open-src </entry>
</table></ark-dirs>

<PKGS-UMASK>002</PKGS-UMASK>

</package>

A package's status field

About a deprecated package: Don't delete anything associated with the package, or do anything to build it, or mess with it. In general, leave everything associated with it alone, in a `frozen' state.