 |
The Open Society |
by Mark S. Miller, E. Dean Tribble,
Ravi Pandya, and Marc Stiegler
Xanadu Operating Company
Originally published in Prospects in Nanotechnology,
ed, Markus Krummenacker, James Lewis; Wiley, 1995.
Proceedings of the 1992 First General Conference on Nanotechnology:
Development, Applications, and Opportunities.
Somewhat edited OCR, much work needs to be done before
this is readable.
Figures still missing.
Electronic media present tremendous opportunities for improving the nature of society. I will first talk about how discourse affects society, and how changes in media may improve societal discourse. Then I will describe the Xanadu[l] system, and how it was built to achieve these goals.
16.1 Improving society
Improving society is a difficult task. More generally, improving complex systems is a difficult task. If you cannot figure out which way is up, see if you can figure out which way is down. Doug Engelbart, back in the early 1960s, wanted to explain to people why interactive systems would make a significant difference to their lives, and to their ability to express ideas. In Figure 16.1, the origin on the axis is what people were doing at the time--writing with pencil and paper. When he found himself unable to communicate to people how much better things could be, he contrasted their current experiences with how much worse things could be. He tied a pencil to a brick, handed it to people and said, "Okay, now write." People found it very difficult. The unwieldy nature of the tool interfered with their ability to express ideas. With the pencil and brick for contrast, he effectively asked two questions: "What made the difference?" and, "How can we move further in the other direction?"[2] This experiment showed people how important their tools and their media were to their effectiveness, and helped them start to see the next brick to remove.
Karl Marx performed a similar experiment on society over the course of most of this century. The origin on Figure 16.2 represents where we are now. Karl Marx tied a very large brick to a very large pencil and the last few years have revealed the result to be far worse than the even his harshest critics imagined.[3] What made the difference between the societies? Two important elements were open markets and open media. How can we move farther in the other direction? In this presentation, I will be addressing the nature of open media, how they differ from closed media, and how social hypertext systems can enhance the advantages of those media. Applying information technologies to the further opening of markets is left as a mission for the reader.
16.2 Media matter
Media matter, because it is in media that the knowledge of society evolves. The health of the process by which that knowledge evolves is critical to the way society changes. Karl Popper, the epistemologist, had the insight that knowledge evolves by a process of variation, replication, and selection, much as biology does. "Variation of knowledge" is what we call "conjecture"--hypothesis formation, tossing new ideas out there. "Replication of knowledge" is the spread of ideas through publication and conversation. "Selection of knowledge" is the discrediting of conjectures through the process of criticism.[4] The ability of our knowledge to progress over time depends on an ongoing process of criticism, and criticism of criticism. The ideas that survive the critical process tend, in general, to be better than those that do not.
In closed societies, when arguments cannot be spoken, hard truths cannot be figured out. When people cannot openly criticize, cannot openly defend against criticism, or cannot openly propose ideas that conflict with the official truths, then they are left with mistrust and cynicism as their only defense. This leads to the simple heuristic of assuming the official truth is always wrong. For example, because science was promoted by the Soviet propaganda machine, pseudo-science is on the rise in Russia. Because anti-Nazism was promoted by the East German propaganda machine, Neo-Nazism is on the rise in East Germany. The official truth is neither always right nor always wrong. Society needs a more sophisticated process for judging claims.
Our society does have open media. Are we in the best of all possible worlds? Are our media good enough? Can they be made significantly better? Among our media, TV is so bad that it is a joke. Only slogan-sized ideas can be expressed. We prize the quality of discourse in our books and journals, but critical discussions in them are only loosely connected. Starting from the expression of an idea, it is hard to find articles that criticize that idea. When arguments cannot be found and navigated, the next harder truths still cannot be figured out.
16.3 Xanadu
I rejoined Xanadu in 1988 largely because of fear about the dangers of nanotechnology, coupled with incredible excitement about the promises of nanotechnology. In looking at the dangers, I saw that none of us individually is clever enough to figure out how to solve those problems. The only hope that I saw in 1988--1 no longer believe it is the only hope--is that by creating better media for the process of societal discourse and societal decision-making, we
stand a much better chance of surviving the dangers posed by new technologies, so that we may live to enjoy their benefits.
I am about to go through the elements of the hypertext system we built. Xanadu has frequently been called Golden Vaporware, and many people have wondered whether this is a never-ending project. One of the things I want to emphasize when I go through all of these features is that I am only referring to the features that are now running in the software. We planned on and anticipate other features, some of which will be mentioned in the future plans discussion, but the body of this presentation will only cover what is implemented and running.
First, I will discuss the four fundamental features--links, transclusion, versioning, and detectors. Marc Stiegler will then present an example using them. Then, I will describe the remaining four features--permissions, reputation-based filtering, multimedia, and external transclusion, followed by some concluding remarks.
16.4 Links
Hypertext links are directly inspired by literary practice. Literature has many different kinds of links connecting documents into a vast web. Textual examples of these links include bibliographical references, marginal notes, quotation, footnotes, and Post-it notes.
We propose to build engines of citation, so that people can navigate this vast web of literature at the click of a mouse. Most computer text systems are predicated on a misconception that the meaning of a document is represented purely or primarily by its content. Documents are not islands. Conventional computer text systems put their effort into the appearance of individual documents. My experience in reading documents (especially reading a literature with which I am not familiar) is that it is difficult to understand documents without their context. A conte~t helps answer questions such as, "What were the ongoing controversies that the author had in mind?" "What views was he supporting or attacking?" "What attacks was he guarding against?" We must understand this whole web of connections in order to understand the documents we are reading. The Xanadu system is built to provide as much support for this contextual information as for content.
With the ability to follow the links in this vast web of documents, is it not easy to get lost? How does one stay oriented? One answer to these questions is guides, a new kind of document that provides an orienting view together with links into the existing literature. I expect guides to come largely from people making their own organizing views of a literature and then cleaning them up for publication, so others may benefit from their work.
16.5 Hyperlinks
Because "nanotechnology" is now used by many to mean any technology approaching the nanometer scale, we have been forced to retreat to the term "molecular nanotechnology." Hypertext terminology has gone through a drift similar to nanotech terminology. The Xanadu project is the one that coined the term "hypertext" and originated the notion of the hypertext "link." However, because the term link has come to be viewed as something much less capable than what we meant by it, we are now calling it the hyperlink. The distinction between the link and the hyperlink is crucial for supporting active criticism in open media.
Hyperlinks are fine-grained, bidirectional, and extrinsic. Frequently, an argument is not with a document or chapter as a whole. It is with a particular point that someone made at a particular place in the text. For example, someone refers to the fourth law of thermodynamics, and someone else writes a criticism saying there is no fourth law of thermodynamics, linking it to the original (see Figure 16.3). The fine-grained property allows the link to designate the particular piece of text with which one is taking issue. Bidirectionality enables readers of the original document to find the criticism, enabling them to exercise fine-grained skepticism, and to constantly ask themselves, "What is the best argument against the thing I am reading right now?" and then, "What is the best argument against that, in turn?" Links provided by other hypertext systems generally have been only in the forward direction, enabling a reader to find those documents referenced by a given document. However, to find criticism, the reader must find the documents that refer to the document they are reading.
Extrinsic linking is the ability to link into a document without editing it. Several other systems support the creation of links that are fine-grained at the targeted end, but these others do so only by modifying both source and target documents.[5] Critics normally will not have the ability to modify the documents they are criticizing. They could spin off their own version into which they attach these links, but then other readers still cannot find these criticisms from the original documents.
Part of what we mean by "open media" is that everyone who is connected to the system can read what they are permitted to read, can write new things, and can make them accessible for others to read. This includes making links to anything that they have read, so that anyone else who reads the original can find the material that has been linked to it. All readers of the system are potential authors. We can think of this process as active reading. Frequently, people make marginal notes to themselves. This is a medium in which readers can share such things with each other. When much writing is commentary about other text, the commented-on text is the best rendezvous point for the authors and readers of commentary to find each other.
16.6 Emergent properties
This kind of accessible criticism can provide for decentralized consumer reports. When people post on the system documents that are either products or descriptions of products, customers of those products can post criticisms of them. What did they think of using them? This commentary can guide the purchasing decisions of others.[6]
A particular capability we are used to in conversation (one that is almost impossible to successfully attain using paper-based literature) is hearing the absence of a good response to an argument. A reader not only can see what the most compelling arguments are against some statement, but also see when there are none, or when all the seemingly compelling arguments have been successfully refuted. Such absences are quite obvious in conversation. Electronic media can make these absences obvious as well, but in a context where the absence will be much more telling, because the missing argument could have come from a much larger audience over a more extended period of time.
Other hypertext systems with their unidirectional links reproduce the asymmetry present in our paper-based media--it is much easier to find something that a document cites, than it is to find those documents that cite a given document. One of the effects of this asymmetry in paper media is the pathological division of scholarly fields into disjoint "schools." Instead of healthy intellectual engagement, debate, and cross-fertilization of ideas, we see a process of increasing inability to communicate between schools, and more preaching to the converted within a school. The terrible irony of attempting scholarship with unidirectional links is that the very attempt to engage in healthy debate across schools accelerates the pathological division process. How does this occur?
Let us consider two schools within a discipline. Generally, students within a school see the documents supporting the positions of that school. The students also see criticisms of documents in the other school. Intellectually eager and honest students, seeking to know both sides, occasionally will follow these criticism links forward. The result is that they will see the parts of the other school's literature that is most soundly criticized by their own school, immunizing them more and more against the foreign ideas. With bidirectional links, these students can also find the greatest challenges to their own school. Bidirectional links also enable them to find the most telling criticisms of the ideas they are inclined to accept.
16.7 Transclusion
Before there were modem economies, there were many little villages, each with their own little manufacturers having to go through a large amount of the production process themselves. These economies were, therefore, much less productive. An individual baker or shoemaker, for example, would reproduce the same kind of work that was being reproduced in many other villages, and would have to fashion a shoe, not quite from raw materials, but without intermediate goods. In extended economies, people can build on one another's work, and there can be a finer-grained division of labor and knowledge, with better specialization.
Now, with respect to literature, authors are frequently faced with the task of re-explaining and Iestating background material that has been explained well elsewhere. If you could just borrow that material, those existing good explanations, and incorporate them (with automatic credit where due), your efforts could be spent stating what is new. We introduce the concept of transclusion to separate the arrangement of a document from its content. There is an underlying shared pool of contents, and all documents are just arrangements of pieces from that pool. In Figure 16.4, the three circled appearances of the same text are actually just one piece of text in the underlying shared pool of contents, and it just happens to appear in three different arrangements which constitute three different documents. We refer to the three documents as transcluding that piece of text. The separation of content and arrangement also leads to good support for
incremental editing. Different versions of a document are just different arrangements of mostly shared content.
This is more than just a hack to avoid the storage cost of making separate copies. Hyperlinks are linked to the content, not to a span in an arrangement. Therefore, when someone writes a criticism of content as it appears in one arrangement, that criticism is visible for the same content as it appears in all other arrangements, including arrangements that were made before the criticism was attached. The normal incremental editing process of a single document is analogous to evolution by point mutation. The ability to transclude text from other documents allows the analog of sexual recombination. Were links visible only from the arrangement into which they were made, both variation processes would destroy selection pressures by leaving criticisms behind.
16.8 Remembering the past: historical trails
As you are editing, an historical trail gets left behind--bread crumbs in history space. The historical trail is simply a sequential arrangement of the successive arrangements of contents. This is yet another kind of context important for understanding. "Things are the way they are because they got that way."[7]
Understanding how they got that way often aids our understanding of what theyare.
16.9 Preparing for the future: detectors
In addition to looking into the past, one also reads a literature knowing it will be changing. How can one keep up? To keep track of what is happening, to keep up with changes, we introduce detectors. One can post a revision detector to find out when things are edited, when new versions of something appear, and then one can use version compare to find out how they are different. With version compare, one can engage in differential reading--reading just the differences between the current version and the version most recently read.
Link detectors are a way of finding out when new links are made to existing material. Let us say that you published something, and you want to find out when others post comments on it. You would like to be informed of comments, but you do not want to have to go back and constantly Iecheck all the things that you have written, so you post a link detector on all the things that you have written, as well as on other documents on which you are interested in seeing further comments. You want to see what people will say about them. As new comments are posted on those documents, you are continually informed.
E-mail is just the special case where you establish a canonical point in the literature, for each person--a place others link to in order to send that person a message. That person simply has a link detector there saying, "Show me all new things that are attached to here." This generalizes to treating any shared point of interest in the literature, as in some sense, a "mailbox," or a "meeting room" for further conversation or conferencing about a topic. Canonical documents become meeting places. Should two disjoint discussions about the same topic spontaneously form in two places, anyone who notices can just make a link between them. The link detectors of each community will then inform them of the existence of the other.
At this point, 1 will shift over to Mark Stiegler and Dean Tribble, who will demonstrate, using the Xanadu software, an example involving exactly the elements discussed so far.
16.10 The WidgetPerfect saga
This is a true story about how a hypertext system was able to save several thousand jobs, One special characteristic about this true story is that it is a true story from the year 1997. It is a story about one of the events that took place at the company--most of you have heard of it--called WidgetPerfect. WidgetPerfect is the second largest manufacturer of widgets in the world, second only to their
big competitor, Microwidget. The people at WidgetPerfect in the year 1997 had identified a xeally significant opportunity in the upcoming expanding environment of widget components technology.
They were developing the world's first fully modular widget. They had a team working on it. Dan was in charge of the preparation of the marketing materials for the modular widget. Ruth was in charge of the technical work team, and John was in charge of the budget and finance, as well as all the costing. At this point, the modular widget was in prototype stage when a very unfortunate thing happened. Microwidget, the big competitor, came out with a partially modular widget, hitting the marketplace first with an inferior product. It was technically inferior, but nonetheless it was in the marketplace first.
Dan was examining this Microwidget, partially modular widget, and it was overall inferior. Nonetheless, it had one really striking improved feature. It had a funculator made out of titanalum, whereas the fully modular widget that was being developed by Ruth only had a duralum funculator. This was an important improvement for certain key market sectors. Even though the partially modular widget did not have anything comparable to a thermoplastic coupler or a hypervelocity rotator, they had to make a change.
So, Dan created a new document in the marketing requirements describing this titanalum funculator. He attached a link to the part of the technical plan that specifically referred to the duralum funculator in the current plan. He made that a new requirement (see Figure 16.5).
Now, Dan knew that in order to get anything to happen with improving the widget prototype, he would have to talk to Ruth. He was reaching for the telephone to call Ruth when Boeing, the largest purchaser of widgets in the world, called him about a $15 million widget order. He got distracted with this purchase, and he never quite got around to calling Ruth.
We have good news. Ruth, knowing that the success of her technical design depended on her being able to respond promptly to new requirements, had attached a link detector to her technical plan. This link detector would be constantly watching for new links of the link-type requirement to be attached. When Dan attached the new requirement to the duralum funculator, Ruth's link detector went off. Ruth was alerted. She followed the link detector out to the link, followed the link back to the new requirement, saw what the required change was, and modified the technical plan to reflect the use of a titanalum funculator.
Well, this was all very fine, except for an additional problem. As I think everyone here knows, titanalum is considerably more expensive than duralum, and so this had some significant effect on the manufacturing cost. Ruth knew that this would have an impact on the budget, and she was reaching for the telephone to call John when smoke started billowing from the laboratory where the
prototype of the modular widget was being manufactured. She ran off to deal with the emergency and never quite got around to calling John.
We have good news. John, knowing the success of his budget was completely dependent on his responding to modifications to the technical plan, had attached a revision detector to the technical plan and this detector was constantly watching for updates. So, when the technical plan was indeed updated, John's revision detector went
off. He followed the revision detector up to the technical plan, used the hypertextual version compare capabilities based on the transclusion relations, compared the new version of the plan to the old, and found that the change was deleting duralum and replacing it with titanalum. He then went back into the budget and updated the budget documents to reflect the increased costs caused by the use of titanalum.
As a consequence of this, the modular widget program was completed on time with a fully adequate specification. It was a completely superior product. It blew Microwidget off the face of the Earth. As a consequence, thousands of jobs at WidgetPerfect were saved.
16.11 Permissions
A social system, to a large extent, is a system of rights and responsibilities. Xanadu has an extensive permission system called the club system, intended to deal with some of these issues. Figure 16.6 shows a document that Bob can edit. Bob has sent it as a mail message to various people in a blind carbon copy ("bcc") relationship. Alice and Chuck are both members of the bcc club of people who have permission to read this document. Bob, though, is the only member who can read or edit the bcc club. If this were a cc list, Bob would still be the only person who could edit it, but it would be self-reading. Everybody who was a member of such a cc club could see who else was a member of that same club.
This demonstrates a principled answer to permissions meta-issues--0ne can distinguish between who can read a document, who can read the list of people who can read a document, and who can read that list, out to any desired degree of distinction (and similarly for the editing dimension). However, infinite regress and needless complexity are avoided by using clubs that are self-reading or self-editing (or both) whenever further distinction is currently not necessary. Should such distinction later become necessary, it can always be introduced by someone with appropriate edit permission to the club in question. Users only grow meta-Ievels on an as-needed basis.
Our permission system also supports the notion of accountability. All actions in the system are taken by someone. When you look at information in the system, you see some identity attached to the actions taken on the information. There are no official truths. There is only who said what, and the structure of the system reflects that.
16.12 Reputation-based filtering
One of the potential pitfalls of an open hypertext system is the junk problem. The ability to find good commentary and crit~cism will be especially important
when reading very important documents, but it is precisely on these documents that one expects to be inundated with tons of worthless or irrelevant links. Without a filtering mechanism, it would be on exactly the documents for which one most needs good commentary that the provision of commentary would be most useless. For example, imagine how many links there would be onto the First Amendment to the Constitution.
Links can be endorsed as worth reading by various readers. However, no
one may endorse with the identity of another. Different endorsers will establish varying reputations with different readers, much as with movie reviewers. Readers can filter their views of links into a document both by who endorsed as well as by link-type. When even this mechanism gives too coarse an answer, one can rely on documents such as a hypothetical Guide to the Citations to the Bill of Rights, endorsed by a reputable publishing house. This very same link filtering ability is also what allows one to find such guides in the presence of a swamp of links.
16.13 Hypertext + multimedia = hypermedia
Increasingly, ideas are being expressed in media other than text, and increasingly, computers are used to handle these other media. We usually refer to hypertext because text is the most important case and the clearest example. However, nothing I have presented, none of the things you have seen the system do, is in any way specific to text, or even to media that have linear flow to them. It all applies equally well to a variety of other media (such as sound, engineering drawings, Postscript images, and compressed video). In all cases, one can make fine-grained links, edits, transclusions, and version compares ( even if the data is block-compressed or block-encrypted). Although the implementation has some optimizations targeted at text, in no way does the architecture make any special cases for text. Documents can, of course, be composite arrangements in which several media are mixed together.
16.14 External transclusion
No software system is an island. We do not imagine that once the product is available, everyone will instantly take all information to which they want access and transfer it into Xanadu. We have to coexist with many other systems for many good reasons.
We handle that with external transclusion. Our documents are able to transclude into anangements that are within the system. These, in turn, are able to represent transclusions of materials that are stored elsewhere. By perceiving other systems through the window of Xanadu, you can see those other systems as if all those documents were within the Xanadu system. Through Xanadu, I could follow a link from a WAIS document into a Lexis document, even though neither system has any notion that such a link even exists. It is not just that the Xanadu system is not an island, that we have to coexist with everything else, it is that through Xanadu, those systems are able to coexist with each other in a way they are unable to now, making them into nonislands.
16.15 Conclusions
When we started building the system, we were thinking purely in terms of paper-based literature-of writing. What we have built is something that has many of the best aspects of both writing and conversation (see Table 16.1). Many of the aspects of each are complementary. Many virtues of conversation make up for flaws in writing and vice versa. We found ourselves building a system that supports the dynamic give-and-take of conversation, and the persistence and thoughtfulness of literature.
Our status is that we currently have a working, portable server. It has some bugs in it, including some performance bugs, but we are working on it. However, all the features that I talked about so far, work. We are continuing ahead with the effort on both the server and the front end. The front end is in a preliminary stage. We consider it adequate to show that the server is real, and to exercise its features. We plan to do a much better front end. The protocol between the front end and the server is very stable, and has been stable for a long time now. Our plans are to get investors, and to finish both the front end and the server. The target for our first product is small- to medium-sized workgroups within companies that have a large body of documents they need to be managing and evolving.
Our first product lacks one major feature. We provide hypertext because documents are not islands. We make the system interpersonal because people are not islands. We provide for the transparent windowing into other systems because no product is an island. However, for the moment, each server is still an island with respect to the other servers, and so each workgroup is also an island. We have designed the system so that, soon after first product, we will be able to weave all the servers together into a transparent distributed system. When you follow a link from one document to another, if the other document is not here but in some server in Tokyo, it will be transparently fetched for you, and the only thing you will notice is that following that link took longer.
For any media to radically improve the process of opinion formation in society, we believe it needs features equivalent to fine-grained, bidirectional, extrinsic, filtered links. These links must not get lost when the documents to which they are attached change. Issues of authority, privacy, and responsibility must be handled in a robust and secure fashion. Open entry of readers and editors is crucial for open discussion. Open entry of server providers is less obvious, but equally important, in order to make centralized control impossible. We will be providing support for people who want to do online services based on our software. All of this is necessary to achieve our open electronic publishing dream. In so doing, we hope to improve the quality of public debate, in order to obtain the benefits of the open society yet again.
16.16 Acknowledgments
We thank the whole extended Xanadu team for having struggled together for many years on a project that has been at least as much a cause as a business. We thank Eric Drexler for exploring the relationship of hypertext publishing to evolutionary epistemology.[8] We thank Anita Shreve for extensive help in editing this presentation.
Footnotes
[1] The Xanadu TM trademark has since become the sole property of Ted Nelson.
[2] Engelbart, D. C., "Augmenting Human Intellect: A Conceptual Framework," SRI Project no.3578, October 1962.
[3] Popper, K. R. The Open Society and its Enemies. (Princeton, N.J.: Princeton University Press, 1950)
[4] Karl Popper originally proposed that selection proceeds by a process of refutation. See Sir Karl R. Popper, The Logic of Scientific Discovery (New York: Harper & Row, 1959). His student, William Bartley, generalized this to criticism. See William W. Bartley, III, The Retreat to Commitment (Open Court Publishing, 1962).
[5] Examples include World Wide Web anchors, Microsoft Word bookmarks, Lotus Notes, and Folio Views Popup text.
[6] The use of bidirectional links for decentralized consumer reports is already happening on the American Infonnation Exchange.
[7] Weinberg, G. M., The Secrets of Consulting (Dorset House Publishing, 1985)
[8] Drexler, K. E., "Hypertext Publishing and the Evolution of Knowledge," Social Intelligence. 1 (1991): Number 2.
Home
