Note that this document is a "plan for features" and not a "HOWTO".
Please excuse the use of ASCII art in place of screenshots.
First, some definitions...
A "node" with "terminals" (unconnected) or "links" (connected):
+----------+
----->| |<------
<------| |<------
| |
+----------+
|
A network of nodes:
+----------+ +------------+
----->| |<------------------| |
<------| |<--------- | |
| | \ | |
+----------+ \ +------------+
\
+------------+
| |
| |
| |
+------------+
|
A node with its "windowlet" (small window) opened:
+----------+
----->| |<------
<-----| |<------
| |
+-------------+----------+--------------+
| |
| |
| |
| |
| |
+---------------------------------------+
|
An "compiled network", which appears in the windowlet of its parent node:
+----------+
----->| |
<-----| |
| |
+------------------------+----------+------------------------+
| |
| |
| +----------+ +------------+ |
| ----->| |<------------------| | |
| <------| |<--------- | | |
| | | \ | | |
| +----------+ \ +------------+ |
| \ |
| +------------+ |
| | | |
| | | |
| | | |
| +------------+ |
| |
+------------------------------------------------------------+
|
Q: Regarding creating a new GUI from a compiled network, say that you are constructing a command-line with a switch that requires a filename, regex, or some other field. How would you construct the network such that the switch and the text-field become associated during the network compilation (making of a subnet)? Take for example a standard povray command-line:
$ x-povray -I def.ini my_povray_file.pov |
For the unitiated: -I requires the name of the povray config file def.ini. my_povray_file.pov is the input filename.
You are asking, "How would -I be associated with def.ini?"
The command itself (x-povray) and the individual options (-I def.ini my_povray_file.pov) are represented as the nodes of a network, and they are linked by terminals that "add strings".
So, it is the order in which things are linked that determines the order of the command-line fields. It's as simple as that. We're expecting the user/developer would put things in the right order.
So, your povray command-line would be 4 nodes linked like so:
+-----------+ +------+ +----------+ +----------+ | processor | ---> | flag | -----> | filename | -----> | filename | +-----------+ +------+ +----------+ +----+----------+----+ | povray | | -I | | def.ini | | my_povray_file.pov | +-----------+ +------+ +----------+ +--------------------+ |
Notice the node types used here:
1 processor
1 flag
2 filenames
Plus, all of the terminals are "string adders", and all of the nodes "add strings".
But, let's say that you want the filename's to be determined dynamically. Perhaps they are set by nodes upstream in the workflow. We then need to have multiple types of terminals.
| (2)
| ^
v |
(1) |
+-----------+ +------+
| processor | --->(3) | flag | ----->
+-----------+ +------+
| x-povray | | -I |
+-----------+ +------+
|
Here, terminal (1) is a stdin piping type terminal, (2) is a stdout piping type terminal, and (3) is a text string adding (used in all command-line type nodes) type terminal.
Note also that the processor in this example is a special command-line processor type node with a string adding type terminal. So, what makes a node a command-line node? It has a string adding terminal.
+----------+
----->(1)| |(3)<------
(2)<------| x-povray |(4)<------
| |
+--------------------------+----------+-----------------------------+
| | (2) | | |
| | ^ | | |
| v | v v |
| (1) | (3) (4) |
|+-----------+ +------+ +----------+ +----------+|
|| processor | ---> | flag | -----> | filename | -----> | filename ||
|+-----------+ +------+ +----------+ +----------+|
|| x-povray | | -I | |
|+-----------+ +------+ |
+-------------------------------------------------------------------+
|
When the command is complete, terminals (1) through (4) are workflow I/O (pipes) using command-line stdin/stdout.
And each "filename" type node has 2 terminals: One is "string adding", and the other is piping.
Q: Can I put a document in place of the filename nodes?
No, because they have different functions:
(1) Document node: I/O capable node that represents data being held on the filesystem. An important feature of the document node is that it can redirect data flow (pipe).
(2) Command-line filename node: Provides the location of the file to the command-line (prints out a string) and redirects data flow (pipe) to a standard document node.
Document nodes can provide filesystem path information to other linked nodes, but they can't go in-line with other command-line constructing nodes. These command-line constructing nodes (1) spit a string of text out to the command-line and (2) gather data from stdout for piping. They can also (3) pipe in data and use them as stdin. We don't show that in the example, but it would mean 2 more terminals for the command-line processor.
Q: Ordinary nodes can be anywhere on the Internet. Is the same true for these command-line constructing nodes?
It doesn't make sense to treat these command-line nodes like "ordinary" nodes. The problem is, how would you do "string-adding" across the Internet? And, does it even make sense to try it, because these nodes are very simple? They should probably be kept local.
We therefore may want to consider a network of command-line nodes to be a different kind of network, kept in its own workspace.
If (1) through (4) are left unlinked when the compilation (making of a subnet) is done, they'll appear on the compiled command's icon. It so happens that all of the piping terminals are unlinked in our example, so that's where they appear:
+----------+
----->(1)| |(3)<------
(2)<------| x-povray |(4)<------
| |
+----------+
(windowlet closed)
|
Now, let's link the documents to the appropriate terminals:
+----------+ +------------+
----->(1)| |(3)<---------------| document |
(2)<------| x-povray |(4)<------ +------------+
| | \ | def.ini |
+----------+ \ +------------+
\
+------------+
| document |
+---+------------+---+
| my_povray_file.pov |
+--------------------+
|
This would tell Piper to take the contents of the documents, put them into temporary files, then construct the command-line as follows:
$ x-povray -I /path/to/temp.def.ini /path/to/temp.my_povray_file.pov |
Q: Wow! That seems so complicated! You mean that I would have to connect all of those nodes for every command? I think it would be easier to just use the UNIX Command Line Interface (CLI).
If you wanted to create a new network every time you had to run a command, then, yes, it would be simpler to use the CLI. But, you're forgetting one important feature: These compilations can be saved and reused. And, if you leave an option open (unselected), that option remains unselected in the compiled GUI!
Q: What would this look like?
If all of the options were set, you might see something like this, where the listing of options is simply text.
+----------+
----->| |<------
<-----| x-povray |<------
| |
+-------------+----------+--------------+
| x-povray |
+---------------------------------------+
| |
| o Executable x-povray |
| o Flag -I |
| o Filename def.ini |
| o Filename my_povray_file.pov |
| |
+---------------------------------------+
|
However, if some options were left unset, you'd see something like this:
Note that the underscore ____ used to surround option names means that there is an edit box there or a way to make a selection.
+----------+
----->| |<------
<-----| x-povray |<------
| |
+-------------+----------+--------------+
| x-povray |
+---------------------------------------+
| |
| o Executable x-povray |
| o Flag -I |
| o Filename _def.ini_______________ |
| o Filename my_povray_file.pov |
| |
+---------------------------------------+
|
And, for simplicity, the text labels can be hidden:
+----------+
----->| |<------
<-----| x-povray |<------
| |
+-------------+----------+--------------+
| x-povray |
+---------------------------------------+
| |
| o Filename _def.ini_______________ |
| |
+---------------------------------------+
|
Can you imagine the capabilities here?
Q: How do we select which labels (corresponding to nodes) are hidden and which are visible?
Ah, we'll get to that later (in time). It's really neat though!
Q: I don't know about you, but if I look at my /usr/local/bin directory, there are hundreds of programs there, all ordered in a non-useful alphabetical fashion. Jumbled in with the useful programs are a whole ****load of randomprogram-config programs which I never need to run, etc. I personally do not want to look through a list of all these programs just to run a program.
We don't need to have every program in /usr/local/bin be accessible to Piper. The default (root level) directories (which contain these program) are in public and private directories:
$PIPER_DIR/nodes/public/
$PIPER_DIR/nodes/private/
Of course, for security we can't work directly out of /usr/local/bin. Using our restricted sandbox directories is a lot like what some other apps are using. Sendmail, for example, now has its own bin directory and its own shell.
And, if work out of real directories, then the user can sort nodes however they want. They just DnD a node into a different directory, exactly like files and folders in a Mac hard drive.
Q: I agree with this for files, but I don't see how this would work for programs, if we are just finding them all in the /usr/bin directory. I would not expect users to add sub-directories to their /usr/bin, sort the programs by hand, and then adjust their PATH to move into all these directories. Or, would moving a program nodes to a different container only create a link to that program in the filesystem, and it would look to nodes like the program is actually in the new folder?
Again, we're not going to use /usr/bin. But how a user will "import" a program from /usr/bin (or wherever) into our sandbox directories, is a good question, and we'll get to that later. We think it would be best for symlinks to be created at the sysadmin level from /usr/bin/programname to the sandboxes.
Q: Are all nodes "containers"? In other words, is every node a compilation (a subnet) of other nodes?
No, "ls", for example, is a "processor" type node, but it doesn't contain anything. It represents ls, and that's all.
Q: Does dragging a node from one place to another mean that "whatever the node represents" has been physically moved?
In the Piper model, the nodes stay put...in their original "locations". Only the representation (XML!) is dragged to the workspace.
Q: Should the default GUI for a processor have a selection widget to choose the program it represents?
We're talking about a "blank processor" (program name never selected), right? We were thinking about just having an edit box where the user/developer can type in the program name. Remember, for command-line construction, we're just adding strings:
>>> executable_name = 'ls'
>>> flag_name = '-l'
>>> file_name = '*'
>>> command = executable_name + flag_name + file_name
>>> print command
ls -l *
More examples:
$ grep 'hello' myfile.txt |
should be constructed from this network:
+------------+ +-----------+ +-------+ | document | ---> | processor | ---> | regex | +------------+ +-----------+ +-------+ | myfile.txt | | grep | | hello | +------------+ +-----------+ +-------+ |
which should be interpreted by Piper as...
$ cat myfile.txt | grep 'hello' |
They are equivalent.
But let's say grep can't take pipes or work from stdin. Then Piper would have to interpret the same network as...
$ grep 'hello' myfile.txt |
and, if myfile.txt were not in our sandbox directories, Piper would have to get it from its locale, save it in a temporary file and do...
$ grep 'hello' /home/jdoe/nodes/middle/temp/tempfile372873 |
Q: I really think that the chances of being able to develop a "standard" wrapper that will work for many programs is very slim.
Oh no, it's quite simple. All we're thinking about for an "unknown executable" is a processor that spits out the name of the executable. You've got a symlink to povray in your sandbox directory; Piper doesn't know povray from jack; Piper says, "Eh, just spit the name povray out on the command-line". That's all. The user/developer can then work on defining it better later on.
What we are proposing is not to try and have a "catch all" mechanism to try and get every program, but instead to require an XML file for every program you want to run. If we are using Piper as an analogy to a web browser, we think of this XML file as analagous to an HTML page for the info you want to display.
Will a GUI interface to a program work well at all without some information about the program? Otherwise, the GUI interface is just as unintuitive as using the command-line, and then, why use the GUI interface?
Yes, but the experienced user/developer can build network's and compile them, making something simple for the inexperienced user/nondeveloper. We want Piper to be able to accommodate both types of users.
When we started designing Piper, we thought we'd have two main programs: (1) the workspace and (2) a Software Development Kit (SDK). But, then we realized that most SDK's, IDE's, and programming languages organized information into "connectivity trees" (networks), which is what the workspace was supposed to be good at doing. So, SHAZAAM, the workspace took on the responsibility of also being an SDK. And the concept of "compiling" nodes/network's helped make this possible by hiding the complexity of the low-level stuff.
Q: This is the way things work for AppLab with their meta-data files, right?
With XML files, sure! But our XML wrappers start out simple, are then built up via network building and compiling, and then end up large and descriptive.
However, we think a big problem with AppLab is that there is no way to get one of these meta-data files for programs that have already been wrapped (is there wrapper sharing?). To remedy this situtation, I think it would be relatively easy let people share their wrappers (compiled networks).
And one of the best things is that people can be anywhere in the world!
Q: And some script that automatically installs them for the user.
Simply open a node pointing to the remote filesystem, and get what you need.
Q: Then the user can download all of the XML files they want and automatically install them. Am I making any sense?
Yes, automagic retrieval of XML representations is a major, fundamental design feature of Piper.
Q: I really think that when a new node is added (dropped on the workspace) the lines should be automatically linked (the way it works in the current nodes that I'm messing with). Most of the time the user will want them linked anyways, and it is a serious pain to link them by hand because all of the lines are hidden under the GUI. So you have to close the GUI for both nodes, link the lines, and then re-open the GUI. I'm really for automatic linking.
But this would require that every program be well defined about the required options. It's a different plan. We would argue that the effort needed to define a node your way is equivalent to just going ahead and linking all the critters. So, yeah, why bother with user-specified links at all??? Or using more than one node??? But we're saying that the user is/can be the developer who (via XML) defines a node, on the workspace. The workspace is an IDE!
Q: What if you want to uncompile a command to make changes? It seems like this would be very programmatically difficult to do if everything is on the main workspace.
We just need to make a couple routines for adding and deleting icons with their windowlets. Uncompiling a compiled command should just require the network inside the compiled command's windowlet (workspace in a workspace) be copied to the main workspace.
And we think that a compile/decompile option could be put in the node's pop-up menu.
Q: With respect to the text viewer, that should be no big deal to do, but should this be contained in the "document" widget or as a separate viewer widget? It just seems like it might be annoying to have to load the contents of a file every time you want to refrence it, especially if you have huge text files like some FASTA file or something.
You're absolutely right. Documents will not have a viewer in their windowlets. Documents are holding places for the data, which are transient and ephemeral.
To give a document a viewer, the user would have to link an appropriate viewer to the document and make a compilation (a subnet) of the two. But could Piper produce these document-viewer compiled nodes (subnets) automagically, according to mime types??? Interesting.
Documents represent data that are being held in the filesystem somewhere. So, the path is important. But we still maintain that documents can't change their association with a filesystem file. They should be treated as the Mac treats its icons. We should put the path in a label widget rather than an edit box.
Also, we think this would be needed if you wanted to use the document node to supply a file for command-line compilation (making a subnet).