Pipes are a great tool for connecting programs together. It is a real
joy to be able to connected various Unix tools together in a
pipeline. One of my favourite is
... | sort | uniq -c | sort -n
to get a frequency count of some collection of lines (often from logs).
It is worth noting that pipes are only one of the enabling
technologies for this sort of thing. There are two others.
The first is the Bourne shell and derivative. The fact that the shell
makes it easy to create pipelines is fundamental to their success.
The other is the common practice of having program receive and
generate un-decorated line-oriented data. This 'common language' for
passing data around is essential for pipelines to work. It is worth
noting that while line-oriented is very standard, having fields in the
lines is very ad-hoc. Many commands allow you to act of fields, but
they all do so in different ways. If you compare sort, cut,
uniq and others you will see a complete lack of uniformity. This
something that a successor to pipes would need to address.
So a successor to pipes needs an IPC primitive (the pipe), and
platform for building applications (the shell) and a standard data
representation (line == record). It also needs a market - a reason
for existence.
My own feeling of the market is for interactive tools. With pipe
line, there is no opportunity for a conversation between elements of
the pipeline. There is no way to have a "server" which will respond
to requests, or a client that can give more information. So
interactivity would be my choice of market place. Note that this
doesn't mean interactive in a 'human-interacts-with-computer' sense,
but in a 'client interacts with server' sense.
This is by no means a new market - there have been RPC (remote
procedure call) systems for decades. There is even CORBA which is
designed to make parts easily interchangeable. But somehow they don't
seem to have the same degree of use as pipelines.
I also think that the IPC primitive is already available. I see two
primitives as being usable. One is loadable modules, and the other is
Unix domain sockets.
Loadable modules allow the server to run in the same address space as
the client, which should provide high bandwidth and low latency. It
does require the server to be single threaded and to only server one
client at a time. For many applications this would be fine.
Unix domain sockets are an easy way to create a connection between two
processes, much like a pipe. It is better than IP sockets as there is
natural permission checking.
That just leaves a suitable data format, and a usable platform.
I think it would be a mistake to try to define how fields fit into
records, what separator to use, and how to label fields. It might be
worth doing this if we were trying to make pipelines better, but we
are going beyond that.
I see that appropriate abstraction being a namespace line a
filesystem. Each server presents a namespace. Each name can be
"opened" and the resulting channel can be subject to "read" and
"write". Each client sees it's own namespace, so that different
clients don't affect each other unless that is the purpose of the
service.