Beyond Pipes
Pipes are a great tool for connecting programs together. It is a real joy to be able to connected various Unix tools together in a pipeline. One of my favourite is
... | sort | uniq -c | sort -n to get a frequency count of some collection of lines (often from logs).
It is worth noting that pipes are only one of the enabling technologies for this sort of thing. There are two others.
The first is the Bourne shell and derivative. The fact that the shell makes it easy to create pipelines is fundamental to their success.
The other is the common practice of having program receive and generate un-decorated line-oriented data. This 'common language' for passing data around is essential for pipelines to work. It is worth noting that while line-oriented is very standard, having fields in the lines is very ad-hoc. Many commands allow you to act of fields, but they all do so in different ways. If you compare sort, cut, uniq and others you will see a complete lack of uniformity. This something that a successor to pipes would need to address.
So a successor to pipes needs an IPC primitive (the pipe), and platform for building applications (the shell) and a standard data representation (line == record). It also needs a market - a reason for existence.
My own feeling of the market is for interactive tools. With pipe line, there is no opportunity for a conversation between elements of the pipeline. There is no way to have a "server" which will respond to requests, or a client that can give more information. So interactivity would be my choice of market place. Note that this doesn't mean interactive in a 'human-interacts-with-computer' sense, but in a 'client interacts with server' sense.
This is by no means a new market - there have been RPC (remote procedure call) systems for decades. There is even CORBA which is designed to make parts easily interchangeable. But somehow they don't seem to have the same degree of use as pipelines.
I also think that the IPC primitive is already available. I see two primitives as being usable. One is loadable modules, and the other is Unix domain sockets.
Loadable modules allow the server to run in the same address space as the client, which should provide high bandwidth and low latency. It does require the server to be single threaded and to only server one client at a time. For many applications this would be fine.
Unix domain sockets are an easy way to create a connection between two processes, much like a pipe. It is better than IP sockets as there is natural permission checking.
That just leaves a suitable data format, and a usable platform.
Data Format
I think it would be a mistake to try to define how fields fit into records, what separator to use, and how to label fields. It might be worth doing this if we were trying to make pipelines better, but we are going beyond that.
I see that appropriate abstraction being a namespace line a filesystem. Each server presents a namespace. Each name can be "opened" and the resulting channel can be subject to "read" and "write". Each client sees it's own namespace, so that different clients don't affect each other unless that is the purpose of the service.
