9 May, 2019

## Alternatives In Visual Programming–Updated

### Current Graph Implementations

Visual programming tools have matured over the years to offer key features that users have come to expect:

• Graphs are composed of nodes, that can be connected by data streams
• Nodes are multi-functions, capable of taking multiple inputs and generating multiple outputs.
• Nodes have input and output ports.
• Data is transmitted to ports as streams, represented by wires. A change in the stream (data change) will cause the graph to re-compute
• Output ports can emit multiple data stream to supply data to multiple nodes.
• In most tools, input ports take single data streams.
• Data streams can transmit singletons, lists, or other more complex data structures. Single value inputs yield single value outputs. Complex data structures yield multiple outputs.

• Supplying inputs of different data structures yields multiple results computed by permutation. For example, as a Cartesian product, a mapping of list elements using guides/lacing (shortest list, longest list, etc.), or a sampling, etc.

### Alternatives In Graphs

Design progresses by comparison, which requires the ability to generate alternatives. Exploring alternatives using current visual programming tools is done by:
1. Creating multiple sub-graphs, which allows for viewing results in parallel. That said, the outcomes will remain mostly disconnected, even if they were driven by the same set of controls. Further, the graph can very quickly become densely populated with nodes.
2. Passing multiple data-streams to inputs, which helps avoid the above issue.
3. Passing lists of values along with mechanisms to select items by index, which allows for viewing results serially– one at a time.

### Input Ports: Single Stream vs Multiple Streams

What if ports supported multiple input data-streams? We’ll gain four benefits:
1. Graphs can be greatly compacted.
2. Alternatives are no longer the result of the graph, but rather the possible sequences of computations in the graph–tacit sub graphs.
3. Variations are instances defined by parameter values.
4. Graphs, as forms of expression, become closer to representing solution spaces; making it easier to reason about a problem.

For this to work, one rule should be followed: multiple streams can connect to a single port only if they [the streams] come from nodes that do not share parents. For example, the structure on left is valid (top if viewed on small screen), while the one on the right is not (bottom if viewed on small screen).

### Example

Let’s take this example. The graph below is composed of a mixture of nodes, some receiving single streams while other receiving multiple streams. Note that nodes C and G do not share parents with B and F respectively.

The graph can be exploded (or expanded) to four graphs, the networks of which are marked in red.

### Proposed Methods To Compute And View Results

Computing all graphs raises the question of the computational power and the possibility of generating far too many options. This can be addressed by implementing graph computation modes or settings, for example compute only the first/last connected stream, compute all, do a sampling, etc. Nodes can also have port configurations to favor streams over others using some rule, or to select specific streams (by index) when solving the graph.

Current tools render results to the same view port. It would be helpful if each result was rendered in a dedicated view port, or a layer within a view port. This will permit viewing results side-by-side–each in its own context, or juxtaposing them in place.

### Diff

A big benefit from seperating results is that one can run diff algorithms to identify differences, whether in the representation of the result ot the parameter values that led to them.

### Proposed Graph Operations

I believe graph tools can become easier to use if they supported operations to quickly alter their structures (I am not referring here to node-layout), for example merging and splitting nodes.

When merging multiple nodes, concurrent streams converging on ports [per node] are collected to converge on a single port on the new merged node. This requires that merged nodes have the same signature: same input and output ports (count and type). Splitting does the reverse. A node can be split if it has multiple streams converging on a port. When split, each new node will take one stream.

Here is an example. Merging the (+) below will change the graph from this:

Note that this is different from packaging sub graph into a single node.

Navigating complex graphs can be made easier with a few easy to implement mechanisms. I proposed teleporting and coloring. For example, tabbing between nodes of interest (marked as anchors), and assigning colors to nodes. Here is a mode showing both (this is a tool I am building in my free time)

### Remarks

• Graphs have two connection types: single streams or multiple streams.
• Multiple streams permit building compact graphs.
• I propose that results of multiple streams are rendered each in its own view-port or layer. Users get to decide which streams should be computed using some sort of UI. This is in contrast to computing all results driven by lists in a single view-port. This gives us two computing modes for graphs.
• I propose introducing an external set of controls to modify graph structures and navigate canvases.

### Former Research In This Area

Here is a paper I co-authored with professor Woodbury on a tool I developed to experiment with visual programming.

Hits: 109

Tags: ,