Arusha Project ideas

The casual ARK user can ignore this.

Should 'things' have a magic 'ALL' field?

I'll use a 'package' example but there's nothing package-specific, really. Imagine you have a <package name="xemacs--21.5"> (a 'thing') with

   <prototype team="op" name="xemacs" />

   <compile>
     <param name="CC"> gcc </param>
     <param name="CFLAGS"> -O3 </param>
   </compile>

   <compile>
     <param name="CC"> cc </param>
     <param name="CFLAGS"> -g </param>
   </compile>

   <install>
     <param name="CC"> cc </param>
     <param name="CFLAGS"> -g </param>
   </install>

Had you asked for the <install> field instead, you would get the param values (cc and -g) from the prototype.

This is core ARK behavior.

Notice that all fields (<compile> vs <install>) are independent of each other. If both fields need to be fed the same info (dependencies, for example), you gotta specify it twice or more.

So: would there be any value to a magic <ALL> field? I guess it couldn't have a value (string,table,list,code), but it could set <param>s and <constraint>s, and perhaps other stuff.

In a case like the one above, you would write

    <ALL>
      <param name="CC"> gcc </param>
      <param name="CFLAGS"> -O </param>
    </ALL>

The semantics would be: The <ALL> field would be looked up in the normal way -- following prototype links and all. Whatever you picked up from the <ALL> field are injected into the proceedings immediately before the value (string,table,list,code). In the typical case, think of it as coming after the last <param> before you hit the <code>.

A magic <ALL> field would definitely remove some unsightly duplication. I worry that it is too ugly for words and a complication too far... Anyone got a view?

Toolsets

A user sitting in front of a Unix machine is, knowingly or not, using some toolset. The order of their PATH and what's in the directories along that PATH decide what tools are in the toolset.

At many sites, the exact composition of the toolset(s) used by users is not too important, and it's left to `just happen'.

At other sites, the composition of the users' toolset is critically important and cannot be left to chance. In a software shop, for example, you really don't want to find out that some developer accidentally built a crucial part of a shipped product with an experimental compiler that he forgot he had in his home directory.

One choice you can make is to have a single toolset for your site (which you then maintain carefully). This is essentially what standard Sidai-style package management does.

Another choice is to have one generally-messing-around toolset (e.g. day-to-day reading of mail, news, etc.), and other more-carefully-controlled toolsets used for project work.

Bear in mind that projects often want to `lock in' on a specific toolset for the life of the project, and therefore may be using old tool versions long after newer ones are available. In some environments, toolsets have to be kept functional for legally-mandated periods, e.g. `10 years'.

Another choice is simply to have a whole bunch of (carefully-controlled) toolsets, and users simply bounce between them. Among other things, this can give users a way to revert back to a known-working tool...

Sidai-style package management can do most of the work towards making toolsets happen. However, these matters need attending to:

• The user-interface to the toolsets;
• On the sysadmin side, exactly how the toolsets are specified, built, checked, etc.

Making the 'ark' script self-bootstrapping

The ark script should have a way to make sure it can download, compile, and install Python, PyXML, ersh, and whatever else it needs to ``get going''. This would also solve the ``new host running a new OS'' problem.

Controlling what hosts (can) do things

Thinking about this reminds me of ruminations I had a while ago on the idea of what can proceed automatically, and what should ARK stop and ask about. Whilst I think the --constraint-hosts idea is right (you should always be able to specify on the command line exactly what you want to proceed), perhaps there's an element of policy here.

For instance, at a flexible I-trust-everyone site, I may say that we're happy for everything up to a final deploy can be done automatically anywhere when required by constraints. On a more restricted site where only certain people are supposed to do certain things, I might specify that compiles may proceed automatically on the reference compile boxen, and installs may proceed automatically on the environment test boxen, but everything else must be explicity done.

Would it be possible to specify this information in your team setup file? Perhaps something like:

  <auto-satisfy-map><table>
    <entry name="compile">capybara,cobra</entry>
    <entry name="install">tarantula,toad</entry>
    <entry name="reveal">tarantula,toad</entry>
  </table></auto-satisfy-map>

  <auto-satisfy-map><table>
    <entry name="compile">*</entry>
    <entry name="install">.,!avalon</entry>
    <entry name="reveal">.,!avalon</entry>
  </table></auto-satisfy-map>

  % sudo ark reveal password-stuff

  WARNING: This command will reveal 'password-stuff' on *ALL* hosts. If
  you are sure this is what you wanted to do, then please re-run the
  command as:

    sudo ark reveal --hosts=\* password-stuff
  #

Pre- and post-command hooks

Let's add in hooks to the host (or team?) thing called 'pre-command' and 'post-command' that allow people to specify their own mechanisms for this sort of thing and then pick a reasonable one for sidai. (I know the 'post-command' is not necessary, but I like the symmetry ot it -- perhaps you'd want to use it to send mail to users when a new version of something has been deployed.)

At this point, Will Partain asked:

Does the 'pre-command' go with

    /our/bin/ark package reveal ...

(i.e. assoc w/ the method `reveal'), or

    /our/bin/ark package ...

(i.e. assoc w/ the thing `package') or

    /our/bin/ark ...

(neither of those)? An example, perhaps...

[And Jonathan replies...]

Hmmm... which thing the method should be attached to and when it's called was the hazy bit in my mind. One idea that springs to mind is to have the `pre-command' method in the package thing and have it called before each "step" of the process. It might then look like (insert standard syntax disclaimer):

<!-- in  sidai/package/ALL.xml: -->

  <pre-command><code language="python">
    if constraint and (not constraint_hosts) and (host not in hosts):
      constraint_hosts = team.auto_satisfy_map[command]
  </code></pre-command>

This assumes that the code for the command will fail if it's on the wrong host rather than the constraint collection code failing before even trying to execute a command on the wrong host.

Alternatives include:

• Having the pre-command in the package thing but only executed once before running the "main" package command invoked on the command line. Can still specify package specific policy. Problem: you don't know what the constraints are yet, so you can't do constraint-specific lookups like the one I do in the example above.

• Having the pre-command in the host thing and executed before each invocation of a command on a package on that host. Can do constraint-specific stuff and can specify by-host policy - bypassing the need for my <auto-satisfy-map> since you could just define the pre-command on your compile host prototype(s) to say "go ahead and compile, I don't care".

• Having the pre-command in the team thing and executed once before running any command of any type on any thing. Problem: you can't do anything thing-specific so you're likely to have a bunch of ugly nested if statements.

Perhaps the right location is "E) all of the above", since if it's not there you just carry on regardless. i.e., try calling the team pre-command before doing anything, then try the package pre-command before invoking a method of a package, then try the host pre-command before running any code on a particular host.

Is this constraint functionality cross-over enough to just pick a mechanism and put it straight into ark though? Am I suggesting something too generic and woolly?