Piper Design Proposal

version Thu Feb 15 2001
 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

1. Overview

Piper is a collaborative of separate layers, processes communicating by CORBA. Instances of Piper communicate on P2P basis to each other.

TODO: Write some more..

1.1 UIL

UIL stands for User Interface Layer, and is the only layer that a user will see. There can be more than one UIL for a Piper instance. The UIL doesn't depend on a curtain type of display, it might be a text console, Xwindows display, or anything else.

1.1.1 creation of networks

1.1.2 ...

 

1.2 DL

The Definition Layer (DL) provides the glue for connecting two different parts of Piper:

- The Build-Time System. This part of Piper involves the user acting through the user interface to create an interconnecting set of programs to be executed.
- The Run-Time System. This part of Piper is responsible for execution of programs and piping information between programs.

The DL rests firmly between the User Interface Layer (UIL) and the brokering layer (BL) and mediates communication between these two sections of Piper. In
addition, the DL is responsible for storing XML describing the current status of the userinterface, storing specialized XML descriptions of programs and libraries,
and communicating with remote Piper implementations.
 

1.2.1 UIL-DL communications

Maintain communication with multiple user interfaces through a streaming XML dialog communication protocol. This communication occurs though a local
socket and allows user interfaces to be developed in any programming language that provides support for sockets.

1.2.2 Storage of XML data

Storing an XML representation of the user interface. This XML representation models the connections between different nodes in the user interface and the
interrelationships between them. These XML representation will allow users to save entire user interfaces or individual nodes, and will provide the ability for
recovery after crashes.

1.2.3 Permanent storage

Maintaining a permanent storage location for specially designed nodes that represent speci?c programs and libraries. The information for these nodes will be
stored as XML and be available directly available to the user interface through the streaming XML dialog.

1.2.4 XML transformation

Converting the XML representation generated by the build-time system into an XML format that can be processed by the run-time system.

1.2.5 Controlling the PL

Communication with the processing portion of the program by implementing a CORBA client to a server in the brokering layer. This communication allows
work ?ow diagrams to be passed to the processing layers for implementation, and allows processes to be queried while running.

1.2.6 Remote UIL

Communication with remote Piper programs as both a client and a server using CORBA (through an idl that still has to be de?ned). This allows a speci?c
implementation of Piper to serve out a specialized node or set of nodes to be utilized by other Piper implementations at remote locations. All aspects of this
remote communication will pass through the de?nition layer.

1.2.7 main logging, glib based?

(http://developer.gnome.org/doc/API/glib/glib-message-logging.html)

1.2.5 Communication Protocols

1.2.5.1 User Interface to Definition Layer

Multiple user interfaces can communicate with the de?nition layer through a CORBA interface de?ned in the IDL (interface de?nition language) ?le uil2dl.idl.
Further information on this communication is available in How To Write a User Interface for Piper. Using CORBA for this communication protocol allows user
interfaces to be written in any language supporting CORBA language bindings.

1.2.5.2 Definition Layer to Processing Layer

Communication between the de?nition and processing layers occurs using a CORBA interface described in the interface de?nition language ?le vsh_dl2bl.idl.
This communication occurs in three broad steps:

1. The de?nition layer negotiates a connecting with the processing layer and receives a unique id, the dlid for which to identify itself in future communication.
This connection is established by calling the following idl de?ned function:

interface Connection {
processorS setup(in dlIdT dlid, in passwordT password,
in connectionE connecttype) raises (IdInUse);
};

2. The de?nition layer uploads an XML document representing the programs to be processed and the connections between them. This XML document will then
be parsed by the brokering layer and used to begin running the programs. The brokering layer returns a URI id which the de?nition layer can then use to further
communicate with the brokering layer about the status of the processing. The upload occurs through the following method:

interface Representation {
    uriS upload(in string xmldocument, in dlIdT dlid,
    in passwordT password) raises (UriUndefined);
};

3. The de?nition layer then queries the brokering layer to determine the status of different programs. When programs that return results are ?nished, the de?nition
layer will then retrieve these results and return them to the user interface to be displayed. Query and retrieval of processes are obtained through the following:

interface Obtain {
    statusS uriStatus(in uriS uri, in dlIdT dlid,
        in passwordT password) raises (UriNotFound);
    string uriInfo(in passwordT processid, in uriS uri)
        raises (InfoNotAvailable);
};
 

1.3 BL

BL stands for Brokering Layer. For every instance of Piper there will be only one BL running. The task of the BL are:
 

1.3.1 brokering of DL-PL data flow

Data traffic between the DL and PL will be transported by the BL. Corba is the technology used all over Piper, it's used to do DL-BL and BL-PL transports. All DL-PL communication is XML formatted, and at runtime labeled by BL with a URI identifier :
    /**
     * URI type, a locating id:
     * uriT is in a IP (Internet protocol) style
     *      uriT = 2.0.0 -> All subnets and nodes local in instance 2
     *      uriT = 74.35.0 -> All nodes in subnet 35 of instance 74.
     *      uriT = 1.2.3 -> Node 3 in subnet 2 of instance 1.
     */
    struct uriS {
        dlIdT instanceID;        /* Set by BL to DLid of DL that generated XML stream */
        blIdT subnetID;          /* BL sets this */
        plIdT nodeID;            /* ?PL sets this, ?BL parses XML */
        string description;      /* ?ripped out XML, ?defined by DL */
        statusS status;           /* BL handles this flag */
    };

BL is capable of multiple instance communication, it uses a 'bookmarks' list to locate other Piper instances. Remote Piper systems will be automatically queried for available resources and nodes (?interfaces, ?public resources). The BL will handshake to he BL of remote Piper instances that responded to a query. All available nodes will be send in both ways during a BL handshake. Available nodes include those nodes that are publicly available and those in an area(*) that is active on the receiving BL. An area in a set of nodes that all share the same area id. Those nodes can spread across Piper instances.
 

1.3.2 communication between instances of Piper

BL - BL communications are not defined yet. A good candidate CorbaSec (http://cadse.cs.fiu.edu/corba/corbasec/faq/single-page/CORBASEC-FAQ.html). But maybe it will turn out this implementation is too heavy for the modal internet connection.

TODO: Write idl \ communication API

1.3.3 Security system

The BL should have a system that handles area's. An area is created by the BL when a DL requests one. The area is given a unique number, the areaId. The id and the password supplied by the DL at creation time will be needed for access to that area. On the Piper instance where the area is created (?area creation host), and at remote Piper's that host the area (?area hosts). Closing down the area on a ?area host is not possible. Putting tight quota's on an area on a ?area host is possible. The ?area creation host can shut down the area, but by doing so, all parts of the area located on remote systems will become separate area's, with new unique id's. ?area hosts become ?are creation hosts once this happens.
The implementation of the security system should be based on ?akenti or ?sesame. Internally the BL should user the real node addressing, outside the core, at the plugin- and DL-interfaces level, a different node id system will block the interface to overrule the security.

1.3.4 resource management

The BL policy is to obey resource quota's. Quota's can be set on ?networks executed at once, ?cpu load, ?disk usage, ?ram usage, ? ... The BL does active optimization of network execution. This is done by feeding historic data about network execution times to an GP engine. Once the GP engine starts to return hits with very high fitness, the BL will listen to the GP engine instead of the DL. The GP expectations will overrule one node parameter given by the DL. This is the requested location where the node should be executed. The GP engine will might operate exploration, so produce random results that indicate high finesses.

1.3.5 ...

 

 
 
 

1.4 PL

PL stand for Processing Layer. A PL exists only during the execution of a network. When a new network needs to be executed, a new PL is spawn for that network. After the network is finished that PL quits running to. It should do:

1.4.1 execute networks

1.4.2 handle legacy issues

1.4.3 ...

 

 
 
 

1.5 GPL

GPL stands for Genetic Programming Layer. There is only one GPL for every PIper instance. It's task are to:

1.5.1 support DL, BL and PL with GP facilities

1.5.2 use spare cpu cycles to process historic information for future optimizing

1.5.3 ...

 

 
 
 

1.6 All Layers together

UIL <--> uil2dl <--> DL <--> dl2bl <--> BL <--> bl2pl <--> PL     <==>    Executed Nodes and Applications
                                 |                            |                             |
                              dl2ail                    bl2ail                     pl2ail
                                 |                            |                             |
                                 +------------ AIL -------------+
 
 
 
 
 
 
 
 
 
 
 
 
 

2. Interfaces of Layers

All layers communicate by Corba to each other. These interface have fixed IDL definitions, and some additional properties like 1-1 or 1-N relationships, security issues.
 

2.1 uil2dl

The UIL and DL have a 1-1 relation.

TODO: Add uil2dl API
 

2.2 dl2bl

The DL and BL have a N-1 relation

See ./piper/idl/DL2BL.idl for complete API
 

2.3 dl2ail

The DL and AIL have a N-1 relation.

TODO: Specify dl2ail

2.4 bl2pl

The BL and PL have a 1-N relation.

See ./piper/idl/BL2PL.idl for complete API

2.5 bl2ail

The BL and AIL have a 1-1 relation.

TODO: Specify bl2ail
 

2.6 pl2ail

The PL and AIL have a N-1 relation.

TODO: Specify pl2ail
 
 
 
 
 
 
 
 
 
 
 
 
 

3. Dependencies

These 3th party libraries are needed for Piper:

- glib
- gdk
- gtk+
- omniOrb
- ...
 
 

Jarl van Katwijk
Wed Feb. 14 10:53:42 CET 2001
Den Haag, Netherlands