Metad
*****

Introduction
************

Metad is a program for running other programs. Unlike the shell (which also satisfies that description), Metad runs background system programs rather than foreground user programs.

Metad serves a purpose similar to cron, inetd, and init, and could replace them all. The significant enhancement metad has over these programs is that it is easily controllable, from anywhere on the network.

Functionality
=============

The basic function of metad is to run programs in response to certain stimuli. These stimuli can be time passing, other programs exiting, I/O activity being possible on some socket, or direct request.

Metad will also, on request, send signals to any of the programs that it has run, report on its status, and disable or re-enable any of the services it offers. The requests are sent to metad over the network by the metac program described below in see metac.

Metad gets a list of services that it should provide from a configuration file (see Config file format). This file describes, for each service, what program to run, how to run it, and when to run it.

The what part is simply the name of the file containing the program. The how part includes command line arguments, working directory and user id. The when part is probably most interesting.

Service Classes
---------------

Meta understands a small number of types or classes or service, each of which respond to different stimuli.

daemon
     The daemon is the simplest form for service (and the only one currently implemented). A daemon service will run its program either repeatedly, periodically, on request, or any combination of these. It may allow multiple instances of the program to be running at once, and can impose an upper and lower limit on this number.

     It may also be able to run its program just once (at startup) or on completion of some other services program (not implemented yet).

listener
     The listener class supports programs which need to listen for input on a network port. Metad will set up the network port and will wait for activity. When activity is noticed, metad will run the program, passing it the network port, so that it can do what ever is needed. When the program exits, metad will resume waiting for more activity.

     The port can be a udp/ip port or a tcp/ip port.

     This is similar to the dgram/wait service of inetd and the stream/wait service of some implementations of inetd.

steam
     The stream class supports programs which need to respond to network stream connections, such as telnetd or fingerd. Metad will listen on a stream socket for a connection request. It will then accept the connection, possibly verify the source, and then run the program passing the the connection.

     As with all metad services, it is possible to limit the number of program instances which are running concurrently.

Usefulness
==========

The main goal that metad was developed to meet, was to assist in the development of various daemon services which needed to be run on various machines around the network. The sort of assistance required involved:

  * Restarting a daemon if it dies, possibly reporting the death.

  * Killing a restarting a daemon when a new version is installed.

  * Taking control of new daemons.

  * Running some program on all machines on the network.

This goal lead to the daemon class and the metac control program being created. The listener and stream classes are later additions brought about due to a dissatisfaction with inetd.

Say something about shortcomings?? hiding failure and not being able to control named.

Configuration
*************

Invocation
==========

Metad takes precisely two command line arguments. The first is used as the name of the configuration file, as described below. The second is used as the name of the file which contains the metad program. This is used when metad is asked to restart itself (for instance, when a new version has been installed).

Metad also inspects its "zeroth" argument. If this is the string metad-restart, then metad assumes that it has just restarted itself and looks on stdin for a description of the previous state (in particular, what processes were running).

Config file format
==================

The configuration file for metad simply lists the different services that metad should monitor, one per line. Any line that is not empty or a comment line (beginning with a hash or number sign #) is a service line. Each line is broken up into words on spaces and tabs. Spaces and tabs can be preserved in words by quoting them with single or double quotes. Quotes and back slashes can be preserved by prefixing them with back slashes.

The words are the services name, its class, some options, the program name, and the programs arguments. The program name is distinguished from options by the fact that it starts with a slash.

Service Name and Class
----------------------

The service name is any unique string of characters and is uninterpreted. In the future, services may be grouped based on a common prefix of the name. This prefix would be up to a period.

The class name is one of a predefined set of classes, currently only daemon is supported.

Options
-------

The options provide details one how and when a service will run a program. Some options are applicable to all classes of service. These are the General options described below (see General options). Others are specific to a class and are detail under the particular class below.

Most options take the form of NAME=VALUE.

Program and arguments
---------------------

As mentioned, the program is the first word on the line which begins with a slash. The full path name of the program must be given. Subsequent words are passed to the program as an argument list. There must always be at least one argument, which is normally the name of the program.

General options
===============

The General options are applicable to all classes of service. The default value can be different with different services. The default listed below is the default used for the daemon service.

Max
---

The max option sets the maximum number of concurrent processes that can be running for a particular service. The default for daemons is one. Format is max=NN when NN is a decimal integer.

Dir
---

The dir option sets the working directory for processes run for this service. The default is the same as metad's working directory. A full path name should be given.

User
----

The user option specifies an alternate userid to for processes run for this service. Metad must be run by root for this to work. The default is to leave the user id unchanged from that of metad.

When setting the userid, the groups ids are also set to those for the named user.

Pidfile
-------

Some daemon programs insist on forking and continuing to perform useful work in the child while the parent exits. This is useful in many circumstances, but makes it difficult for metad to continue to monitor and control the program. To help with this, the pidfile option specifies the name of a file which the program might write the process id of the forked child.

When a process that metad has started exits, metad checks the appropriate PIDFILE to see if it was written since the process started, and if it contains the pid of an active process. If this check succeeds, then metad assumes that the process found is carrying one the work of the original process.

This process will then be the target of any kill commands, and it will be treated just as though it were the original process.

Watch_output
------------

If the watch_output option is listed for a service, then metad will connect the standard output and standard error streams of any processes started for that service to a pipe. Metad will monitor the pipe for two purposes.

 1. Any data received from the pipe will be logged with syslog as an informational message about the processes.

 2. Metad will assume that the program is still active until an end-of-file is read on the pipe --- even if the process that metad first started exits.

Enabled
-------

The enabled option can be used to enabled or disable a service at startup. The value can be one of the words yes and no. The default value is yes.

Crash
-----

The crash option specifies a program to run after a crash (exit due to signal). The program is expected to clean up or report the problem. Currently this is unimplemented so details are scarce.

The Daemon Class
================

The daemon class understands two more options. They are min and period.

Min
---

The min option specifies the minimum number of processes that should be running for the service. If fewer than that minimum are running, new processes will be started until the minimum is reached.

The default minimum is zero.

Period
------

The period option specifies how frequently a daemon service should start a new processes. The value should be a decimal integer with a single character suffix. Suffixes are s for seconds, m for minutes, h for hours, or d for days. If no suffix is given, seconds is assumed.

If this period passes without a processes being start (or attempted to be started), metad will attempt to start a process to execute the services program. If the maximum number of processes are already running, the attempt will fail, but will still be recorded as an attempt and metad will wait for the period to pass again.

The Listener Class
==================

The Listener class is not implemented yet, so details are scarce. It is expected to need a port option (service/protocol, e.g. pop3/tcp).

The Stream Class
================

The Stream class is not yet implemented, so details are not available. It is expected to need a port option (service/protocol e.g. telnet/tcp) and multiple access options (access+.domain access-@netgroup access+nnn.nnn/16)

The identity of the peer will probably be provided in the environment, e.g. METAD_PEER=host.domain.au:port

Control
*******

A particularly useful feature of metad is the fact that it can be controlled from anywhere on the network. To achieve this, metad understands a fairly simple command protocol. Commands can be sent either over a tcp/ip connection, in which case a response is returned, or via a udp/ip datagram. The use of datagrams means that a command can be broadcast to all metads on a given subnet.

As one of the commands the metad will obey is to rebroadcast a command, commands can easily be broadcast around a medium sized internet.

Disabling and Enabling
======================

Individual services can be disabled and re-enabled.

When disabled a service will not start any new processes, though it will not attempt to stop and processes that are already running. A disabled service that would normally listen for some specific network activity will not listen for that activity at all.

Services are disabled be sending the command disable SERVICENAME and are enabled with the command enabled SERVICENAME.

These can be easily sent with metac using option -D SERVICE and -E SERVICE.

Killing
=======

Killing a process involves sending a Unix signal to the process. This does not always cause it to die.

To send a signal to all processes for a particular service, the command is kill SIGNUM SERVICE. To send to a specific process, its process id must be given instead of the servicename.

Sending signal 15 (SIGTERM) can easily be done with metac using -K SERVICE or -K PROCESSID.

Starting
========

Some classes (currently only daemon, definitely not stream, probably listener) will start a new process on request. This request is sent with the run command. Optional extra information can be sent with the run command to modify the action of the started program. This is passed the the program through the environment variable METAD_ARG. The command to run a new process is run SERVICE EXTRA-INFORMATION.

If no extra information is to be sent, this can be done with metac using -r SERVICE. To send extra information simply add -a ARGS.

Listing
=======

Metad can be asked to send a report on the services it is maintaining including the current processes which are running. The information is returned in a binary format which is only understood by metac. Either a single service can be listed (command list SERVICE, option -l SERVICE) or all services can be listed. (command list, option -L).

I should say something about the listing that is returned... later.

Reread and Restart
==================

The reread command causes metad to reread its config file. New services can be added and options, programs and arguments can be changed. If a service has be removed from the config file, then the service is not forgotten by metad (it might have running processes) but it is disabled.

The restart command causes to metad to exec a new instance of itself, replacing the old one. Information about what processes are running for which service is passed through so that there will be no undue interruption of service. Processes for services which no longer exist will be simply forgotten.

This is useful after installing a new version of metad.

Broadcast
=========

The broadcast command will broadcast it arguments as a command to all nearby metads (on the same subnet). This may cause the command to be delivered to some metads multiple times (if they are on machines with multiple interfaces). As most metad commands are fairly idempotent, this is not likely to be a problem, but should be considered.

metac
=====

Metac is the program to use to send commands to metad. It takes are arguments a number of commands and a number of hosts to send the commands to. It may be asked to broadcast the commands, or to send them via datagrams. The default it to send them over a stream connection.

Commands and host names may be intermixed on the command line. All commands will still be sent to all hosts.

Commands can be given explicitly using the -C flag, or more briefly using one of the command flags. The flags understood by metac are:

-u
     Send (udp) datagrams, rather than making stream connection.

-b
     Broadcast datagrams on all network interfaces.

-B
     All commands should be sent with a broad  prefix so that the responding metad will broadcast the command in its subnet.

-v
     Be verbose in reporting responses --- what does this mean???

-C command
     Send the given command as is.

-r service
     Send a run SERVICE command.

-a args
     Append the ARGS to the previous -r command;

-l service
     Ask for a listing of the named service.

-L
     Ask for a listing of all services.

-D service
     Send a disable SERVICE command.

-E service
     Send a enable SERVICE command.

-K pid/service
     Send a kill 15 PID/SERVICE command to kill the process or service.

-R
     Send a reread command.

-X
     Send an exit command. (but thats not documented...);.

-V
     Display version number of metac and send request for version information to metads.

Logs
****