Distributed Persistent Collaboration

A Start-Up Scenario and Strategy for Acquiring Resources

Organizations that host standards development efforts need good tools for remote, persistent collaboration. Organizations that focus on software standards, in particular, have the expertise to cooperatively develop such tools. Such an enterprise may well be the ideal starting point for a system that can “bootstrap” itself into one we can use to help solve the major, complex problems that confront mankind.

Originally published 2003


I was privileged to participate in Douglas Engelbart’s Unfinished Revolution colloquium in the Winter of 2000. To say that the colloquium was intellectually stimulating is a gross understatement. The weekly sessions, ten in all, so galvanized my attention that I flooded the mailing list with thoughts, ideas, and reawakened dreams, all precipitated by the colloquium. I was so prolific, in fact, that I was asked to speak at the colloquium, which I did — not just once, but twice.

The basic premise of the colloquium was that we need to harness the power of the computer to solve the enormously complex problems that confront mankind — not by having the computer “solve” the problems, but by enabling the kind of discussions that will allow the most brilliant minds on the planet to do so. In addition one of the major concepts proposed by Doug Engelbart was that we need to “bootstrap” ourselves in that effort, by starting with a small system that evolves over time into one that is capable of handling ever greater problems.

Being part of the colloquium was one of the proudest moments of my life. And the intellectual reverberations from that series of talks continue to echo through my life.

I woke up just the other day with one of Doug Engelbart’s ideas rolling around in my head. It was an idea that I had originally argued against in the strongest possible terms. It made no sense — or so I thought at the time. But on this day, that idea was bouncing around with some recent technology advances, and I saw that there was a way a very real way in which the “ABC” idea made tremendous sense.

Perhaps more importantly, I saw how those ideas might provide the impetus for, and stimulate the commitment of resources to, the development of sophisticated systems for distributed, persistent collaboration.

Distributed, or remote, collaboration, means that the participants are not physically co-located. Persistent collaboration means that the focus is on the final artifacts of the collaboration — the design documents, standards, source code, or what have you, rather than on the temporary whiteboard-interactions that lead to them.

ABC Organizations: For and Against

The idea that struck me as so unrealistic, at the time, was Doug Engelbart’s idea of the “ABC” organization. Here is an example that gives capsule summary of that idea:

  1. The Bigelow Boat company makes boats. Making and selling boats is it’s bread and butter. That is it’s “A” level activity.
  2. To the degree that they can improve their processes, though, they can increase their margins or lower their prices, or both, which allows them to compete more effectively with other boat builders. So instead of ordering a year’s worth of bolts at one time, they may try to order just enough for the next few boats, reducing their investment in inventory. Identifying and implementing process improvements of that kind is a “B” level activity.
  3. Taking things one step further, we have “C” level activities — improving the ability to improve. For example, designing a “just in time” inventory system that automatically replenished the bolt bins when they were getting low would qualify as a C-level activity.

The concept here was that the people involved in C-level activities were an “improvement community” and, as they worked with their counterparts in other (generally competing) organizations, they formed a “networked improvement community”, or NIC.

The culmination of the vision, as best I can recall, was the creation of a meta-NIC — a community formed around the idea of improving the kinds of systems that NICs used to carry out their C-level activities.

Essentially the goal was, and is, to enlist the aid of computers to help people manage the complex problems that confront our society, and to find solutions for them.

At the time, though, I had major problems with the concept of C-level activities. The whole paradigm seemed to hinge on that concept, and it seemed to me that no organization would, or could, engage in such activities.

In fact, I argued, our whole economic system had made it possible for new companies to come into existence and build products that solved C-level problems for other organizations, and that was how C-level activities got carried out.

To take an example like The Bigelow Boat company, it seemed to me that there were a number of reasons that made it impossible for such a company to create a just-in-time inventory system:

  1. Lack of expertise.
    They’re boat builders, rather than software designers.
  2. Lack of a payoff.
    Even if they succeed, the returns that any single company could generate could not possibly repay the huge investment in making such a system work.
  3. Lack of resources.
    Given the size of the payoff, they couldn’t afford to put enough people on it to get it done in reasonable time.
  4. Lack of time.
    Given the lengthy time that it would take, and too few people to do the job, it might never get done at all. Even if it did get done, it would take too long to begin reaping any rewards from the effort.
  5. Lack of recognition.
    Any VP who undertook such a project would be committing to a huge resource sink with tremendous downside potential. If it never worked, the VP could be out of a job. And even if it did work, the rewards would be so long in coming that the VP would never be regarded as a hero. The VP would be lucky, in fact, if anyone connected the dots well enough to identify the software system as a profit-generator. In other words, it was a high-risk, low-reward proposition that no executive in his or her right mind would undertake.

On the other hand, it seemed clear to me that a software company could choose to develop a just-in-time inventory system and sell it to every company in existence. The huge potential return made it possible to devote the necessary resources. And since developing that system was their A-level activity, there was never any question of “siphoning off” resources to do the job, as a boat company would have had to, for example.

For all of these reasons, I felt that the concept of a C-level activities in an organization was a non-starter and that, as a result, any causality chain postulated on that foundation was doomed from the outset.

But today, some five years later, I’m seeing things differently.

A Different View of C-Level Activities

My original analysis basically states that no organization can possibly afford to be solely responsible for a C-level activity. I stand by that analysis, but the key word in that phrase is solely. It occurs to me that organizations can and do compete in C-level activities, as long as they do so cooperatively — essentially joining forces with other organizations.

That strategy lets them siphon off a small percentage of their resources to participate in C-level activities, in a way that provides downstream rewards without impeding the organization’s short-term goals. Seen in this way, in fact, organizations are currently involved in a wide-variety of C-level activities:

  • Attending seminars
    Whether attending as speakers or audience members, organizations are sharing information on best-practices.
  • Standards efforts
    Companies frequently cooperate on standards, in order to compete vigorously in the markets they generate.
  • Open Source Software
    People from different companies operating on a shared base of software is a perfect example of a cooperative C-level activity.

One interesting observation here is that while companies compete head-to-head in their A-level activities, and compete behind the scenes in B-level activities, any ability to engage in a C-level activity depends on cooperation. It only follows from the economics that such activities cannot be undertaken independently, but, at the same time, such activities are necessary to ensure long-term competitiveness.

But what really struck me during this line of thinking was the way that standards efforts reflect C-level activities, the way that standards efforts involve IBIS-style discussions, and the fact those involved in software and XML standards efforts have the expertise to design and develop IBIS-enabling systems.

IBIS and Collaborative Design

An Issue-Based Information System, or IBIS, is one that enables well-structured discussions. Jeff Conklin’s paper, A Short Course in IBIS Methodology, provides a tremendously readable overview of the IBIS concept, so I won’t go into in detail here. I will merely state that the goal of such a system is to identify and record issues that come up during a discussion, to elicit proposals and suggestions, and to record arguments for and against the possible solutions that are proposed during the course of the discussion.

Ultimately, it would be incredibly valuable to build a collaboration tool that would allow open source software developers to collaborate on the design of a software system, much as they can collaborate currently on an implementation. Requirements for a Collaborative Design/Discussion/Decision System was a concept-piece that focused on a preliminary design for such a system.

At the moment, though, there is a dearth of tools to enable remote, persistent collaborations of that kind. There are serious technical problems to be solved, as discussed in Technical Impediments to Persistent Collaboration Tools.

That paper is still in preliminary form. The excellent feedback received from Eugene Kim, Chris Dent, and Albert Selvin and `has yet to be incorporated.

As with any C-level activity, the development of collaborative discussion tool is necessarily a cross-organization, cooperative effort. But the design space is vast. There are a huge number of places to start, possible goals to achieve, and mechanisms to use in the process.

But there is a chicken and egg problem. Good tools are needed to collaborate on a design. But a good design is needed for a collaboration tool. There is also a social problem. No matter what systems are designed, or what standards are defined for such systems, people have to agree to use them. That is the only way that a tool set will become used widely enough to become a collaboration standard.

Given the difficulty of creating an initial design collaboratively, it is clear that some organization must take the lead in defining an “adequate” initial version. Over time, that concept can be extended and expanded, improving the system in the manner that Doug Engelbart’s “bootstrap” vision suggests.

But it is vital to find the right problem to solve. The problem must be one that can be achieved in a reasonable amount of time, but which results in software that can eventually be extended to become a design tool for open source software and, beyond that, a system that allows people around the world to collaborate on the search for solutions to problems as disparate as the energy crisis and the design of social systems that protect the rights of individuals against the advancing encroachment of corporate capitalism, and everything in between.

Software standards efforts look like the right place to start.

A Place to Begin

IBIS and standards efforts are a natural match. A standards effort invariably involves people with a wide range of opinions, each with their own laundry list of goals to achieve in the effort. In such a system, the needs that the participants express can be captured as issues in an IBIS system. And as they argue for and against various proposals, invariably other perceived needs will surface.

For each perceived need, or issue, it is possible to incorporate it into the standard, to make the standard extensible in some way, or make sure that the standard incorporates the components necessary either to create the feature or to create a substitute for it. It may be also be argued that a feature should be deferred until a later released, or that is unnecessary.

The arguments need to be tracked, along with the various proposals they give rise to. Such a system should make it easier to see how different proposals provide for stated needs, and to see which proposals are held in the highest regard.

The natural spearhead for the development of such a system is an organization that develops large numbers of standards — either multi-company organizations like the W3C and Oasis, or a company like Sun Microsystems with the significant level resources it devotes to hosting Java Standards Request (JSR) efforts.

Since all of those efforts are multi-company efforts, it is reasonable to expect that the major organizations involved in the effort could devote a person or two to developing systems that aim to improve the process. For example, organizations like IBM and Apache that have hundreds of people involved in standards efforts would no doubt be willing to devote a developer or two to help make the process more effective.

The requirements would be gathered from participants in existing standards efforts, and one of those efforts would serve as a pilot project, before rolling out the system to other efforts.

Interestingly, there are several interesting starting points for such a system which might be useful individually, or which might form the basis of a technology stack for such a system:

An IBIS-based systems that combines email and shared web pages. Users record their thoughts in IBIS-form on a web page, which sends email messages to notify others that a change has occurred, with a link to the page embedded in the message. (Fundamentally, the right way to combine the archiving of structured information in documents with the interactivity of email.)
Knowledge Management Desktop
A system that turns the web into a huge outline. Potentially useful to view IBIS-based web pages in outline form.
Blue Oxen Associates
In addition to the candidates mentioned above, many others exist. Blue Oxen is an association devoted to analyzing, developing, and promoting collaboration tools. It can claim a highly-technical body of participants with experience in a wide variety of collaboration tools. In addition, its founding members have done a lot of work on purple-izing web documents (adding addressable “purple numbers” to every paragraph in a document). “Fine grained” access of that kind would be invaluable when commenting on and referencing the various proposals that circulate in a standards effort.

It is fairly easy to imagine some minimal requirements for the system:

  • Import existing specifications in HTML or XML
  • Purple-ize the spec automatically, to make it addressable
  • Collect comments on different sections
  • Provide a filtered view that lets you see just the spec, or the spec and commentary
  • Record edits and maintain versions
  • Notify readers at various checkpoints (instead of on every single edit)
  • Provide a filtered view that highlights changes since the last checkpoint
Beyond those humble beginnings, the system could attempt to:
  • Track multiple proposals
  • Record arguments for and against the proposals
  • Maintain a list of the issues (needs) expressed by the participants
  • Keep track of whether a need is satisfied by inclusion, extension, or componentry, or whether it is deferred or rejected.
  • Provide subsystems for rating the proposals and, finally, for voting on them

In other words, a reasonably useful system could be constructed in relatively short order, while almost unbounded opportunities exist to make it even more useful for remote, persistent collaboration. That combination of fast-track beginnings and long-term growth potential makes it an appealing place to start.


Organizations that host standards development efforts need good tools for remote, persistent collaboration. Organizations that focus on software standards, in particular, have the expertise to cooperatively develop such tools. Such an enterprise may well be the ideal starting point for a system that can “bootstrap” itself into one we can use to help solve the major, complex problems that confront mankind.

Open source software and standards efforts show that the cooperation necessary to build such tools is a common thread in today’s “ABC” companies. Given a system that enables collaboration on specification efforts, it should be relatively straight forward to extend it into a collaborative design tool. As the design tool is applied to the task of developing collaborative software, the benefits of “compounded interest” begin to accrue, improving the system at an exponentially increasing rate (up to some bounding asymptote).

The result, at some point, is a system that becomes increasingly useful for solving global and social problems including the use of environmental resources, pollution, energy, overpopulation, and government. And that is the real point of the system. The problem has always been one of finding a small enough place to start, and of getting resources committed to the effort.

Improving the collaboration process for standards efforts seems like the appropriate place to start.

Resources and References

For more information on the Unfinished Revolution colloquium.
More on Douglas Engelbart.
More on the Bootstrap Alliance, founded to achieve Engelbart’s “bootstrap” vision.
The IBIS Manual: A Short Course in IBIS Methodology, by Jeff Conklin.
Requirements for a Collaborative Design/Discussion/Decision System.
Technical Impediments to Persistent Collaboration Tools.
Blue Oxen Associates. A highly-technical group led by Eugene Kim, focused on the analysis and promotion of collaboration systems.
The Elephant system. A combination email/web system that captures IBIS-style discussions.
The Knowledge Management Desktop: an outline-oriented web browser written by Advanced Data Management (ADM).
The “World Wide Outline” concept, ADM’s approach to making the entire Web appear in outline form.

Copyright © 2003-2017, TreeLight PenWorks

Please share!

1 Comment

    Trackbacks & Pingbacks

    1. RuDi – A Ruby-based System for DITA document generation | Treelight.com May 5, 2017 (9:02 pm)

      […] Current technologies make it difficult to carry a profitable design and decision-making discussion online, unless it occurs in real time, as in a face to face meeting. But a good collaboration tool could allow for thoughtful discussion and a careful review of the options, as described in Distributed Persistent Collaboration. […]

    Add your thoughts...

    This site uses Akismet to reduce spam. Learn how your comment data is processed.