Technical Impediments to Persistent Collaboration Tools

When attempts are made to develop “distributed persistent collaboration” tools, a variety of technical issues stand in the way. This article examines several of these impediments to progress.

Originally published 2003

What “Distributed Collaboration” Means

These days, multiple vendors offer “collaboration tools”. Of course, “distributed collaboration” is implied, because we already have the meeting rooms and whiteboards we need to work with the guy next door. The available tools work with varying degrees of success, but what they have in common is a focus on distributed temporal collaboration. In other words, they attempt to make it possible to “have a meeting” online.

Such tools are valuable, and useful. But I suspect that eventually, cameras and projectors will be sufficient to solve that problem. Today, it’s possible to have a video meeting with someone on a distant coast. It works pretty well, too. The only limitation is the size of the video screen. But with a large enough screen, folks in another location could appear to occupy the other end of the table. LCD screen technology is growing to meet that demand, along with those of the television-viewing audience, so we can expect to see them arrive in the coming decade.

The two-location meeting is therefore very nearly a solved problem. Producing a virtual meeting from multiple locations is harder, but it can be seen in television shows like the MacNeil/Lehr news hour, where each person is on camera, and each has a monitor for all of the others. But, given a large enough LCD screen, all that is required is the software necessary to slice and dice it into multiple virtual monitors, so different portions of the real estate can be allocated to a single individual, to a several small rooms with a few people in each, or to a single large room that contains all of the remote participants.

Of course, the issues of sharing presentation materials and whiteboard space still remain, but these issues, too, are well on their way to being solved:

• Systems exist that recognize what you draw on a whiteboard, saving the image and it’s location in the computer. When the “whiteboard” you draw on is a clear sheet of plexiglass in front of an LCD screen, then the recorded images need only be transmitted to the remote sites and displayed on LCD screens there. A button click can then record a snapshot, after which material can be erased and redrawn.
   
• Most slide-presentations are delivered from a computer, these days. The technology already exists to send slides to remote viewers. The only reason its not in wider use is that the technology for video-based meetings is still expensive, and not really satisfying, given the size of LCD monitors and the problem of sharing a whiteboard. But as those problems are solved, video meetings will come to be greatly desired. Soon after that, the price will begin to drop, hastening the acceptance curve still further.
   
• Even videos are more and more being stored on computers and delivered from there. The same kind of limited broadcast technology will make it possible to share video-based presentations. (Basically, a video is just a stored form of the video stream that records a participant in the meeting. So the same solution that allocates a portion of the LCD monitor applies here.)

The Goal: Distributed Persistent Collaboration

A problem which is much further away from being solved, however, is that having a distributed collaboration that persists — one that continues over time, and which is reflected in a variety of documents and materials that result from the collaboration.

In Requirements for a Collaborative Design/Discussion/Decision System, I described a number of features that are needed to make such a system work. (Eugene Kim produced a similar list of requirements which focused more deeply on technical impediments.)

The goal of this article is to identify the technical problems that stand in the way of building such systems. The motivation for each entry will also be described — that is, the desirable goal or feature that the issue impedes. Along the way, therefore, you should be able to a decent picture of how a persistent collaboration tool would work, even if you haven’t considered them before.

The Issues

Many of the issues discussed here are discussed at greater length in the paper, What’s Wrong with Email?. But here, we’ll try to lift the issues up to center stage and make them the focal point of the conversation. This list follows a general progression from hard, low-level requirements — basic functions which are indispensably necessary — to softer, high-level, wish-list entries — desirable features which would affect the pace of development and adoption, without necessarily preventing it.

Global Identification

The most basic requirement of all is the ability to locate and access information on remote systems. Web hyperlinks give us one way to do that. But they’re long, and they are too intimately tied to the information presentation layer. We’ve been using the wrong metaphor — addressing, which describes a location, rather than identification, which is a unique tag of some kind. (Even so-called Universal Resource Identifiers (URIs) have fallen prey to that problem.)

The plague of broken links that infests the Web points out the inadequacy of links that are tightly tied to information presentation. A link consists of the name of a system, plus a path to a document on it. Now, the name of the system has a desirable layer of indirection. The name is used to look up a system identification number on a DNS host (dynamic name server), and that ID is used to find the system. But paths do not even have that layer of indirection.

The whole system depends on what names my system call things. As a result, any of these operations will break the link:
–change the name of the system
–change the name of a directory in the path
–move a directory to another location
–change the name of the file
–move the file to another location

These are syntactical changes to the syntax — changes to how the information is presented on my system. A deeper level of linking is needed, so that when you link to X, at the very least the link remains valid no matter where X lives on that system. Ideally, the link would remain valid even if you moved it to another system. That is a harder problem to solve, but it is necessary for truly global, truly semantic addressing mechanism.

 

Dynamic Links / Active Links

The display of a Web path provides useful information, however. The names of directories show how material is organized, while the name of a file suggests the content it contains. So, having found a link to some document on the web, it is an easy matter to browse the directories that contain that document.

So the ability to see the path to a document is desirable. But what you see, and what the computer is linking to, obviously need to be decoupled. What needs to happen is this:
1. You make a link, using the path you know about.
2. That path is displayed, but the system looks up the *actual* ID and stores that, as well.
3. The system uses that ID to find the document, rather than the path
4. If the path changes, the path displayed in the document must change, as well.
(Perhaps the old and new values should be shown, but the very least the “current address” must be displayed, rather than the old address.)

 

Localized Paths

A path-address to a location in Japan does me no good whatever. It might as well be a series of numbers. In fact, that would be better — I could read the numbers. But if the semantic ID for the information causes the path address to be displayed, then it would also be possible to view that path through a “localization filter” that shows me path names in a language I recognize.

 

Granular Addressing

As useful as document-level identification is, it is not nearly granular enough. We need to access information down to the level of paragraphs and headings, at least. A paragraph is supposed to be a single unit of thought, so in a sense it is the “atomic” level of document. That makes sentences and words the subatomic level, and letters the quantum level, I suppose. Addressing that deeply might well be useful — or it might introduce inordinate levels of complexity. But at the very least, uniform paragraph-level addressing is necessary — without requiring the author to add a tag for each and every paragraph in the document.

 

Versioning

If a links remain valid when you move or rename a document, that’s a major step forward. But it still leaves the problem of deleting a document, or editing it beyond all recognition. A versioning system is required to solve that problem. Either remote documents must be copied to a local cache, or else the remote site must be trusted to do the versioning. Either way, it is necessary to ensure that information your document depends on is still available.

Again, “dynamic links” are needed, so you are informed when the information has been modified. You then the ability to review the new material, and the ability to decide whether to keep your link to the old material, or upgrade to the new.

 

Typed Links

While manual review of versioned material is desirable, it must also be possible to automate the process. It should be possible to specify “always update”, “never update”, or “manual update” for a any link.

Typed links are also necessary for a global ratings system. Rating a web entity Amazon-style, for example, would mean giving it 1 to 5 stars, and attaching a note to explain my rating. A google-style server could then prioritize search results using global ratings, in addition to the number of links. That process that becomes more difficult in hierarchies, where the sublevel ratings must add to the ratings of parent entities, and sublevel entities must “share” parent ratings. But however the semantics of ratings systems are defined, a “rating link” type is a necessary requirement to make it work.

 

Composite Documents

A composite document is built from multiple segments, using a process that Ted Nelson calls “transclusion”. For example, the shared whiteboard in the temporary collaboration scenario consists of material written in multiple locations. Erasing the board in one location must remove material written in a completely different location.

At first, I thought of document composition as simply requiring a type of link, which was identified as a transcluding link. But that is an implementation detail. The more difficult problem is creating facilities for operating on such documents. It must be possible to edit transcluded material as though it had been originated locally. In other words, the result of the composition is a true “document”. Whether copies are made locally for editing, or versions are maintained on the remote system where the material originated, this problem must be solved.

 

Meta Data

Collaboration is infinitely easier when you can find out what the heck the other guy is talking about. Even before that, it only becomes possible when you can find out that the other guy said something relevant in the first place.

Semantic meta data lets you find relevant information, and it also lets you find related and explanatory material for the results of your search. (Topic maps, in particular, hold out great promise for this.)

But systematic, wide spread meta data requires meta data standards, the ability to rapidly find the appropriate tag and easily apply it to your data, and the ability to searching using meta data.

At its most sophisticated, a search should be capable of finding isopmorphic patterns of information that have one or two entries in common. If the structure of an African bee’s social system is very similar to the organization of your file system, that would be worth knowing, wouldn’t it? Semantic-level meta data is a necessary first step.

 

Tower of Babel

Programs I write in Java can’t talk to programs written in Perl. C hackers live on one island, SmallTalk hackers on another. The world needs transducers so that, no matter what language I coded in, the work products can be used in other languages. (Then, the lifetime I have spent coding would have a lifetime of code to show for it, instead of being limited to the most recent applications on the most recent hardware platform I have been using.)

 

Malleable Archives

In collaboration based on email, an archive of the dialog is maintained. That is a desirable level of persistence. But the conclusions reached during the long, drawn-out dialogs remain buried at the bottom of the heap, where they are inaccessible to all but the most determined archaeologists.

The persistent layer layer — the “conversation log” Douglas Engelbart’s NLS and Augment systems, needs a more dynamic, malleable layer above it. In that layer, it must be possible to take the conclusion, promote it to the highest level of the archive, and subsume under it all of the dialog that led up to that conclusion.

Such an archive will have the current tendency towards expansion and addition balanced by a tendency towards reduction and simplification. It must be possible to introduce new conclusions or new questions at any level of the hierarchy, where new questions provide a way of framing a dialog, helping to resolve disputes by identifying the crucial question(s) that lie at their heart (see The IBIS Manual.)

An interesting aspect of this process is that people both compete and cooperate. They may compete for “the most accurate conclusion” or for the “question at the heart of the issue”. As part of the competition, the subsumed material will be referenced by each candidate. People may also cooperate, using composite documents to refine the presentation of the conclusion or question. Darwinian selection then applies, with the eventual winner being hoisted to a position of prominence.

The nature of the “malleable archive”, coupled with ratings, is in essence the process of science. Theories come and go, rising and falling in popularity. Complexity grows over time, until some simplifying hypothesis resolves differences and causes material to coalesce. The same simplification process occurs in the design and construction of computer systems (and, unfortunately, much less frequently in legal and social systems).

 

Authoring Tools

xHTML has been around for years, now. It’s a tremendous improvement over HTML, in that is well-formed and easy to deal with, with sophisticated capabilities like transcluding links. But XML authoring tools are still either abysmally deficient or overwhelmingly expensive. So few original documents are being produced in xHTML. (The lack of good display components for components to use — in particular, tree-structure viewers that are capable of displaying multi-line entries — is the major limiting factor in creating authoring tools using today’s technologies.

 

Component-level Programming

The concept of component-level programming is still nascent, after all these years. So an editor for an a new paradigm like xHTML is still a complete write-from-scratch project. So it takes many years before new editors come on line. With serious component-level programming, it is (at least conceptually) possible to throw together a bunch of existing components and create a solid program in a fortnight. Sure would be nice.

 

Extensible, Integrated Systems

The world of collaboration systems and knowledge-based systems, like malleable archives, is marked by shifting concepts and changing needs. The best editor I ever used was one in which virtually all of the editing operations were available for programming, so I could create powerful customized routines to get the work done.

In the world of knowledge-based systems and collaboration systems, the tools must first be integrated into a single, coherent system. It should not require one program to view a document, another to author it, and yet another to act as a server. All of these operations need to exist as a single, unified whole, so that the act of authoring leads to the act of publishing, merely by flicking a switch. Similarly, there should be smooth, indistinguishable continuum of behavior that runs from browsing for information, through commenting on information and rating it, up to authoring new documents which incorporate and reference information that has been viewed.

It should be possible to send a feedback message to the author of any page viewed on the web, without having to hunt for a email address that may or may not exist somewhere on the site. (It should be possible to attach a rating to any page or product viewed on the web, and do so in the sure knowledge that unscrupulous vendors won’t shake bad ratings by changing addresses!)

The Minimum Necessary Configuration

A final problem, worthy of a section unto itself, is the fact that the Minimum Necessary Configuration (MNC) consists of several interacting components, each of which must exist for the system to be effective. Ideally, the components should be seamlessly integrated. But whether they are integrated or separated, all of the components must exist to create a usable system.

In this section, we’ll use the internet as our model, examine how it grew, and contrast it with our ability to develop a usable collaboration system.

How the Web Grew

The “Web” as we know it as actually a composite of 5 different software systems. If any of them were missing, the web would not exist:

• HTTP protocol
  A mechanism for transmitting messages between systems, using available hardware.
   
• HTML language
  A standard for encoding pages to make them viewable as documents, rather than as simple text.
   
• Server programs
  To deliver HTML pages to users.
   
• Browser programs
  To decipher the HTML page and render its contents for display.
   
• Production systems
  To create the HTML pages.

The Web grew and evolved because it started small, and grew from there. The first production systems were simple text editors. The original HTML language contained a basic, minimal set of tag. Browsers were equally simple, and server programs delivered only static web pages. Originally, there were none of the animated images or interactive displays that we see today.

But the web only grew to it’s size and scope by virtue of the fact that all of the necessary facets, no matter how minimal, were in place at the outset. The result was a minimal system in which it was possible to create a simple web page, and get it delivered to an end user.

As usage grew, each of the components grew and expanded, acquiring more functionality and getting more complex, until the web as we know it came into existence. But that expansion would never have happened had not the Minimum Necessary Configuration existed at the outset.

Can a Collaboration System Grow the Same Way?

The thrust of this article has been, in the main, to identify problems in the existing web infrastructure that tend to preclude the development of a viable collaboration system. With those defects in the available infrastructure remedied, it should be possible to create a minimal collaboration system, and evolve it over time.

The evidence that irremediably defects exist in the infrastructure is the fact that we still don’t have good engines for remote, persistent collaboration, despite great effort from many brilliant minds.

But if the infrastructure problems were solved, would it then be possible to organically evolve major functionality from simple beginnings? It’s an open question, certainly. And it may be in the nature of the desired system that it is simply not possible.

When we look at the web, we see a nice separation between the authoring process and the delivery process. While the kinds of things we can author have grown in scope and complexity, the authoring process itself is still a single-individual, single-system process. Similarly, the delivery is (in general) to a single-user on a single system. (Various broadcast models have been attempted, but none has so far met with stunning success, to my knowledge, a fact which may also be attributable to infrastructure problems.)

In a collaboration system, however, complexity is immediately and inevitably part of the process of from the start. Global identification and versioning, malleable archives and document composition — all raise the bar of complexity much higher than was necessary to display a simple web page.

On the other hand (and here is the cause for hope), what now looks like a “simple” web system undoubtedly looked like major complexity, at the time. It is entirely possible that even the most complex system we can imagine and build today will still grow and evolve beyond our wildest imagination.

Maybe future generations will look back on the overwhelmingly complexity of the systems we are designing today and say, “that was just a simple collaboration tool”.

References

• https://www.gdss.com/WP/IBIS.htm
  Jeff Conklin: The IBIS Manual: A Short Course in IBIS Methodology
• Video Review: Doug Engelbart’s NLS System
  Review of the 1968 video
• https://sloan.stanford.edu/mousesite/1968Demo.html
  Video segments from the original NLS demonstration
• https://www.bootstrap.org/augment-71279.htm
  Doug Engelbart: Toward Integrated, Evolutionary Office Automation Systems

Copyright © 2003-2017, TreeLight PenWorks

Please share!