Final year project report 2003
by Laurie Voss
Supervisor: Dr. Mike Joy
Abstract
Web2 is a proposal for a unified architecture for the web: a cohesive set of related technologies designed to enable information presentation over the Internet. These technologies touch every aspect of the current set of technologies loosely known as the World Wide Web, on both the client and the server side. Web2 is expected to mesh well with a variety of other emerging technologies, including XHTML, RDF and other technologies for providing semantic data. At the same time, web2 includes updates intended to provide the functionality currently provided by HTML, CSS and HTTP.
Keyword list
WWW, web, language, XML, HTML, semantics, protocol
Table of Contents
Part 1: The web2 Project 6
Part 2: web2 10
Introduction
Terminology and notation
A brief history of WWW development
The need for web2
The web2 architecture
The content level: shape 2 and face2
The network level
Part 3: link2Apache 76
Implementation goals
Choices of hardware and software
Implementation
Program design
Part 4: Project summary and conclusions 83
Acknowledgements
List of Appendices
Full table of contents
Author’s assessment of the project
1. What is the
technical contribution of this project?
2. Why
should this contribution be considered relevant or important to computer
science?
3. How can
others make use of the work in this project?
4. Why
should this project be considered an achievement?
5. What are
the weaknesses of this project?
Shape2 – structure and content
Face2 – appearance and behaviour
Link2 – location and relations
Talk2 – authoring/publishing protocol
WebTorrent – P2P caching and load-balancing
A brief history of WWW development
Evolution of the web: the Browser Wars
Five problems of an ever-growing web
Capabilities and failures of the WWW
What the proposed standards lack
Web2 as a disruptive technology
The content level: shape2 and face2
A high-level language for the web
Libraries: scalability and maintainability
Integrated behaviour and appearance; descriptive behaviour
A better alternative to frames: incremental loading
A note on accessibility and design focus
Semantic vs. presentational markup
Divorcing URIs from physical locations
Distributed caching and load balancing
WebTorrent: the peer to peer web
Extending the concept to the web
Choices of hardware and software
Project summary and conclusions
Difficulties and experience gained
A. Web2
content-level design document
B. Web2
network-level design document
C. Web2
design document: link2 extensions to shape2
Table of figures
Diagram 1: The basic architecture of the web
Diagram 2:
Generalised network-level architecture
Diagram 3: The
network level in WWW
Diagram 4: Levels of
abstraction at the content level
Diagram 5: The
architecture of the WWW with reference to the 5-level model
Diagram 6: The relationship
between XML, XSLT and XSL-FO
Diagram 7: Web2 at
the content level with reference to the 5-level model
Diagram 10: The
WWW network-level architecture.
Diagram 11: The
web2 network-level architecture.
Diagram 12: A
document with a single physical location may have more than one semantic
location
Diagram 13: By
abstracting paths to semantic locations, the dependence on the file system is
removed
Diagram 14: A
portion of a WebTorrent network.
Diagram 16: The
arrangement of classes and methods in link2apache
Diagram 17: The
database structure of link2Apache
Part 1: the web2 project
How to read this report
This report is divided into several parts.
Part 1: Introduction to the project
This is what you are reading now.
Part 2: Web2
This is a self-contained description of web2, an architecture for information presentation. It is a summary of the results of the research and development. The development of web2, being essentially a design project, was not at all linear and hence a conventional “development report” would not have been informative.
The descriptions of the technologies in this section do not go into details of syntax, but focus on the reasoning behind the design decisions in the language. Technical details are contained in the appendices.
Part 3: Link2Apache
This is an account of the development of a proof-of-concept implementation of a portion of web2. It follows a more conventional chronological approach to the development process.
Part 4: Project summary
Author’s assessment of the project
This section follows the suggestion for self-assessment of the project mention in the project handbook.
1. What is the technical contribution of this project?
Web2 is an elegant solution to a number of largely un-addressed problems in the field of web languages. It contains a number of wholly original ideas that have not presently been suggested in any forum on the topic of web standards. These include:
Ø A higher-level language for web design (shape2)
Ø Inheritance and generalisation applied to web structures; including libraries of web interfaces using style sheets as parameters (shape2 and face2)
Ø Semantic locations as separate from physical ones, and the additional functionality this can provide (link2)
Ø P2P applied to the distribution of static web content (WebTorrent)
2. Why should this contribution be considered relevant or important to computer science?
The web is one of the largest and still one of the fast-growing applications of computer science in the world. The web now touches the daily lives of almost everyone living in the Western world on a daily basis, even if they do not access it themselves. The problems facing the expansion of the web are serious and cannot be overlooked.
3. How can others make use of the work in this project?
This project does not completely specify the design of any of these technologies and barely scratches the surface of implementing even one of them. The concepts of web2 must be expanded, refined, discussed, and eventually implemented.
4. Why should this project be considered an achievement?
An enormous amount of research and development went into producing the ideas for web2.
On a personal level, I regard this project as an achievement because I sincerely believe it to be some of the most original and creative work I have ever done, with genuine potential to be useful to all the millions of people who use the web.
5. What are the weaknesses of this project?
The implementation of link2 was weak, mainly a result of being dwarfed by the amount of effort that went into the research and development of web2 as a whole. This is a reflection of the author’s relative motivation to create new ideas rather than implement existing ones.
Final assessment
Web2 is by no means perfect; it is barely even started. But it is a very good start.[1]
Part 2: web2
Introduction
I have been designing and publishing web sites since 1996, when the web was still young, and began with the most basic language tools. I have watched web technology, and with it web standards evolve and change. I have used the web and the technologies that build it constantly, often since their invention. This has given me insights into how I would like the web to work as a user, and how as a programmer I would like to be able to create websites. However, the web doesn't work the way it should. Sometimes it doesn't work at all. Web2 is an attempt to fix that, to "scratch the itch".
What is web2?
Web2 is a better version of the World Wide Web.
More formally, web2 is a proposal for a unified architecture for the web: a cohesive set of related technologies designed to enable information presentation over the Internet. These technologies touch every aspect of the current set of technologies loosely known as the World Wide Web, on both the client and the server side. Web2 is expected to mesh well with a variety of other emerging technologies, including XHTML, RDF and other technologies for providing semantic data. At the same time, web2 includes updates intended to provide the functionality currently provided by HTML, CSS and HTTP.
It is however difficult to refer to web2 in a general sense. The technologies that make up web2 include:
Shape2 – structure and content
An application of XML to describe user interfaces in terms of a collection of related documents also authored in Shape2 or any other compatible language, such as XHTML. Central to Shape2 is the concept of a website as an "information presentation application" or GUI for information. This is distinct from and complementary to XHTML's document-centric approach. The relationship between the two technologies can be thought of as XHTML describing the information in a document, and Shape2 providing the interface to access that document.
Shape2 includes significant enhancements to the extendibility, reusability and maintainability of web code, as well as incidental improvements to bandwidth efficiency that are made use of by other parts of web2, namely WebTorrent.
Face2 – appearance and behaviour
A language to describe the appearance and behaviour of interfaces defined in Shape2. In a break with the other language elements of web2, face2 is not an application of XML. It is also unique in that it replaces, not supplements, the emerging standard for describing presentation, XSL-FO. The reasoning behind these design decisions is described later in this report. The innovation behind face2 is its functional approach to describing interactive behaviour, rather than procedural methods such as ECMAScript.
Link2 – location and relations
The most abstract of the web2 technologies, link2 is a server-side architecture closely tied to the concept of "packages" used in shape2 that increases the robustness of web links by divorcing URLs of web content from the file system. Benefits of using link2 in conjunction with shape2 include many standard CMS features such as automatic queuing, expiry, archiving and versioning. A proof-of-concept implementation of link2, called link2Apache, forms the software part of this project.
Talk2 – authoring/publishing protocol
A protocol for accessing web content on remote servers, this extension to standard HTTP also allows publishing, version control and multi-user authoring, fulfilling one of the original concepts of the web by allowing the viewing application to also be the authoring agent, as well as enhancing the web's utility as a tool for collaboration. talk2 is based on the emerging DeltaV standard, itself based on WebDAV.
WebTorrent – P2P caching and load-balancing
A P2P client network, implemented as a local HTTP or talk2 proxy, that increases the robustness of the web by caching downloaded web content and providing a distributed network to supply it to other clients. WebTorrent is based on the existing BitTorrent concept.
More detailed descriptions of the web2 architecture and the design philosophy behind the various parts of web2 are included later in this report.
Terminology and Notation
So far I have been using various terms loosely. Before going into a more formal description of web2, however, it is worth formally defining the various terms used in this report.
General terms
Diagram 1: The basic architecture of the web
The web is used to refer generally to technologies that provide access to specially formatted content over the Internet. Web content is information that has been formatted specifically for use over the Internet, abbreviated to simply "content". A discrete piece of content is referred to as a document. Documents are written in various web languages, but the web is more than just the languages in which it is written.
WWW will refer collectively to the current set of loosely associated technologies which combine to form the web.
A user is any individual who uses the Internet to access content. Users are collectively known as the audience. As with all emerging web standards, no assumptions about the physical abilities of users are made: users may be visually, aurally, or otherwise impaired.
A user agent is any device used to access web content. User agents can take a variety of form factors and have a wide range of capabilities, including mobile devices, text-to-speech converters and Braille systems.
A user agent is expected to render content, i.e. process it into a form accessible to the user. There are no restrictions on how this may be done.
A browser is a specific type of user agent that is primarily visual, with a relatively large viewing area, in the order of hundreds of thousands of pixels.
Accessibility refers to the ability of non-typical users to access content through devices other than standard browsers. In the context of this report accessibility refers to user-agents with non-typical form factors (e.g. mobile phones, PDAs) as well as non-typical methods of rendering content (e.g. text to speech).
Most non-obvious abbreviations will be expanded on first use.
Use of terminology
For the sake of readability, this report will often use the term “browser” instead of the more correct “user agent”, as well as specifically visual terms such as “display” and “view”. This should not be taken as an indication of neglect of the obligations of portability and accessibility. This is discussed in more depth later in the report.
Some technologies of the WWW
The reader is assumed to be well acquainted with the day-to-day operation of the web and have a basic understanding of its technologies. These summaries are included merely to highlight relevant points referred to later within the report. A more comprehensive list of WWW components with analysis occurs later in the report.
HTTP: HyperText Transfer Protocol: the basic level of communication between clients and web servers. Modern HTTP is version 1.1 of the standard. In addition to basic retrieval of documents from remote servers, HTTP supports authentication, encryption, and upstream data transfer from client to server such as entries in web forms, search queries, and binary files.
HTML: HyperText Markup Language. The lingua franca of web documents, HTML is a subset of SGML, Standard Generalised Markup Language. HTML contains code for controlling appearance, structure and meaning (semantics) of information in documents. There are many versions of HTML, the most recent being HTML 4.0. Technically speaking, HTML has been phased out in favour of XHTML, but HTML of various versions still comprises the vast majority of available documents on the web.
CSS: Cascading Style Sheets. A non-SGML language used for associating HTML with presentational information in a more centralised, maintainable way. The term “cascading” refers to the process by which successive style sheets override the declarations of previous ones for progressively fine levels of presentation control.
JavaScript: a simple scripting language used to provide client-side interactivity to HTML. JavaScript is technically a trademark for an implementation of the standardised form of the language (ECMAScript), but the term is used to refer to all implementations.
DHTML: Dynamic HTML. Loosely used to refer to “HTML and associated languages”, including CSS and JavaScript, when used to produce advanced HTML effects.
XML: eXtensible Markup Language is a new general-purpose subset of SGML. XML is not a markup language per se, but a means of creating many compatible languages called applications of XML that can be processed in a standard way. XML is very HTML-like in structure but has a stricter syntax.
XHTML 1.0 (or just XHTML) is an application of XML that mimics the vocabulary of HTML 4.0 exactly; it is merely an XML-compatible version of HTML 4.0. XHTML has been the official standard language of the web since August 2002 but has not yet been widely adopted.
XHTML 2.0 (or XHTML 2) is an extensive revision to the XHTML 1.0 specification and is the current candidate to replace XHTML. XHTML 2.0 is not backwards compatible with HTML or XHTML 1.0; this is significant as all previous versions of HTML have been at least partially backwards compatible with the immediately preceding version.
Other terms
TBL: Tim Berners-Lee. The creator of the WWW is frequently referenced throughout the text by this abbreviation.
W3C: the World Wide Web Consortium, the standards body for the web, currently chaired by Tim Berners-Lee. The W3C concerns itself with directing the development of the web. It is essentially the only body with any real authority over web standards, and it currently has the support of most of the major browser manufacturers.
Semantic and presentational elements: in a web context, these are opposites. Semantics refers to the meaning or implication of an element to a human viewer; presentation refers to its (usually, but not always) visual appearance. For instance, the presentation of a button widget could be, for instance, a grey rectangle that responds to mouse clicks by appearing to move. Semantically, however, it is a form submission mechanism. Given semantic information, a user agent that was not primarily visual might render a button in a completely different manner on the understanding that it was the method by which to submit a form. For instance, an aural browser might simply have a spoken command used to submit forms, in which case it would not need to render the button separately at all.
Web page refers to a single set of information that is intended to be viewed more or less simultaneously: for instance, a page of HTML and all associated images.
Web site is a collection of web pages related by a theme and usually by location. This is not the same as domain; domains such as Yahoo! Geocities often host thousands of separate sites under a single domain.
CGI is the Common Gateway Interface for HTTP. As its most basic function, CGI allows web servers to interface directly with programs, and hence allows web servers to serve the output of a program rather the contents of a file.
Notation and nomenclature
Ø The name “web2” was a deliberate choice. Web2 is not WWW 2.0, which is owned by the W3C in any case. Web2 is complementary to the existing WWW and can be used wholly or partially alongside it.
Ø Both the content-level and network-level technologies are referred to as simply “web2”.
Ø The names of the web2 technologies were chosen to reflect their purpose and function rather than simply producing another list of meaningless acronyms, and also to suggest their common nature.
Ø “Web2”, “talk2”, “link2”, “shape2” and “face2” should all be spelled entirely in lower case unless at the beginning of sentences.
Ø WebTorrent should be spelled as shown. Its name is a derivative of BitTorrent.
A brief history of WWW development
In order to better understand the motivations for web2 and the current state of web technology, it is worth looking briefly at the history of web technologies.
Where the web came from
A history of the web is relatively easy to piece together, since nearly all the key figures are still very much alive, and indeed most of the relevant documents were digital and are still available online.
In 1989, Tim Berners-Lee (TBL) proposed “a global hypertext project, to be known as the World Wide Web … designed to allow people to work together by combining their knowledge in a web of hypertext documents.”[2] This is as close to a “mission statement” for the web as it is possible to get. The project included the HTTP protocol, the HTML language, the first web server httpd and the first web browser, WorldWideWeb, later renamed Nexus when WWW became a more general term for the technology. Promoted as an internal information-management tool at his then-employer CERN, the name of the technology leaves no doubt as to TBL’s intentions for the technology.
Crucial to the WWW was the concept of hypertext: documents containing “links” which led to other documents. However, hypertext was already an established concept in 1989: the word was coined in 1965, and there had been conferences about it in 1987 and `88. The concept had been developed by the visionary Douglas Engelbart at Stanford Research Institute[3], inspired by the Memex system described in Vannevar Bush’s seminal article, As we may think[4], published in 1945. TBL’s innovation was not the idea of hypertext, but its application to a distributed data network rather than a single coherent system. In addition to documents in the file system, links in HTML could also refer to the full network address of a document on another machine.
The benefits of this system were immediately obvious to users of the system, and use of the web exploded from that point. The popularity of the web is the major cause of the vast increase in Internet use since 1990.
The original WWW
It is helpful to look at the very first version of WWW to discover what its inventor considered the important parts of the system, before later developments and other minds complicated the picture.
- Firstly, the WWW was originally
a coherent system: HTML was designed for HTTP and vice versa.
Subsequent developments have focussed mainly on the client side, improving
browser software.
- Presentation was not a consideration. The original WWW proposal refers to as “bells and whistles” anything more complicated than the display of ASCII text on a screen 24x80 characters in size.
- The client was an authoring
tool. Nexus was both a viewer and a WYSIWYG
editor of HTML. It was not, however, particularly good at this; the
original Nexus browser produced very poor-quality HTML, which led to a
tradition of “hand-coding” in text editors. However, TBL intended that it
be easily possible to annotate web documents within the context of
the browser itself.
- The aim was to produce an
information management system, not just a convenient method of presentation.
TBL envisioned HTML as the medium of storage, not just of
presentation. Individual documents were nodes and links were vertices in
an arbitrary graph of documents that could be used to represent any
information structure.
- The WWW was platform-agnostic. Nexus was written for the NeXTStep OS, and was quickly
followed by browsers for a range of platforms. This ability to share a
common data format across operating systems was a key “selling-point” in
TBL’s initial proposal to CERN to develop the technology.
The birth of the web gave a clear indication of the purposes to which TBL intended to put it. Although the early version of the web was in many ways inferior to its current form, many of these characteristics of the original WWW are no longer present:
- Despite the best efforts of the W3C, the set of technologies which compose the WWW are only loosely connected, frequently overlapping and in some cases conflicting. Development has also been one-sided: HTML has gone through more than five different versions while HTTP has had only one incremental revision.
- Presentational control is a
major part of web technology. Of the four
original characteristics, this is where the current web most obviously
diverges. Current WWW languages focus heavily on precise control of
presentation (XHTML much less so, but it has not been widely adopted).
- Most web browsers cannot author
HTML. Most authoring is done offline or
outside the sphere of influence of the web server software, then
transferred to the web server, using methods other than HTTP.
- The web has become a medium of
publication, not storage. This is largely a
result of the high levels of presentational markup, making HTML an
inefficient and unwieldy data storage format. XHTML attempts to address
these problems.
Evolution of the web: the Browser Wars
To understand why the web currently works the way it does, one needs to look at the evolution of web technology. By far the biggest influence on the current state of web technology was the fierce competition between Netscape Communications and Microsoft known as the “Browser Wars”.
In late 1995, Microsoft released version 1.0 of its browser software, Internet Explorer (MSIE), as part of the Windows 95 operating system. It quickly followed this up with version 2.0 in December of the same year. This was the first real challenge to the dominance of Netscape’s eponymous browser (NS), which at that point had more than 90% share of the browser market. The two companies began to compete fiercely. Netscape had only recently started charging a fee for its browser, and quickly reverted to giving away its browser for free, to match Internet Explorer.
Unable to compete on price, the companies competed by adding features to their browsers. In the case of Netscape, this included adding an integrated e-mail client, newsgroup reader and other programs to its software. But more importantly, both companies made unofficial “extensions” to HTML, to increase the range of features developers could include on pages designed for that browser. These included features in common use today, such as tables, frames, fonts and JavaScript.
Initially Netscape, which had pioneered extensions to HTML by introducing tables in version 1.1, led this development and Microsoft’s efforts consisted mainly of playing “catch-up”[5]. However, Microsoft, perceiving the cross-platform nature of Netscape as a threat to its operating system monopoly, poured vast amounts of resources into its browser project, and in version 3.0 introduced support for style sheets, widely recognized as an important step forward in web design.
The fast-paced and hostile nature of this competition had several important effects:
- Development focussed on the client side. An improvement on the server side would benefit both browsers, and would only improve a single site. An improvement at the client side would improve the user’s experience on every site they visited. More importantly, server administrators are very slow to upgrade their software and prefer to use extensively tested software. This did not suit the rate at which both companies were introducing new features, and neither company was well known for producing reliable software.
- Buggy and incomplete implementations. The pressure to quickly introduce new features overrode the need to extensively test them. Netscape introduced the practice of “beta” releases to counter this problem, a practice quickly adopted by Microsoft. Both companies would make available untested versions of their software and encourage enthusiastic users and developers to field-test the software, essentially out-sourcing the testing process to uncompensated members of the public. This increased the quality of the eventual releases, but both browsers were infamous for their many bugs.
- Competing and incompatible standards. The pace of development in both browsers far outstripped the ability of the W3C to produce formal standards, and the companies were in any case unwilling to cooperate on standards, as common standards would eliminate a “feature” that one company could claim over the other. This manifested in three forms:
- Entirely different syntax, such as Netscape’s LAYER tag in response to Microsoft’s CSS
layers. These completely different methods of achieving the same
functionality often necessitated the production entirely separate pages
for each browser, and persuaded developers to choose a “preferred”
browser.
- Similar but incompatible
syntax, such as Netscape’s JavaScript versus
Microsoft’s Jscript language. To compound this problem, each browser
would often attempt to interpret the syntax of the other, resulting in
errors.
- Different rendering of
identical syntax. The worst of the three,
subtle differences in interpretation of HTML syntax would result in minor
differences in the rendered version of the same page for even quite
simple HTML.
As the competitors leapfrogged each other with successive releases of their browsers, the range of common features available to developers expanded greatly. However, as the features become more complex, the differences between the two browsers meant developers had to develop double versions of their sites, or simply forgo the more advanced features. The usability of HTML was badly damaged by the side effects of the browser wars.
Microsoft eventually won the browser wars, with more than 85% of the browser market[6]. This left the WWW in an undesirable state: dominated by a single product with its own particular syntactical quirks and rendering bugs, with little competition to motivate further innovation. More importantly, it left web languages in disarray: developed mainly in response to user demands without central control or an overall plan of direction, HTML was feature-rich but technically unsound. The lack of coordination between the two main sources of innovation had produced a great deal of overlap in functionality in some areas and left other types of functionality unexplored by incomplete or partial implementations.
Furthermore, the browser market is still sufficiently fragmented to make the remaining incompatibilities a major hindrance to further development. It is theoretically possible to produce quite complex effects using HTML and associated web languages. However, the most complex and useful of these features are still usually browser-specific, either for reasons of differing syntax or quirks in implementation. Even when developing for MSIE specifically, relying on these features results in a page unusable or severely degraded to often more than 10% of a site’s audience; an unreasonable figure. This discourages developers from using these features, and restricts the usability of the web as a whole.
One particular victim of this problem has been DHTML. MSIE and competing browsers lack a common Document Object Model (DOM)[7] which allows scripting languages (such as JavaScript) to refer to elements of an HTML page in a uniform way. This means that advanced interactive effects are almost always browser-specific unless significant extra effort is expended to include others, often on a per-browser basis. This has severely hampered the use of DHTML as a medium for applications, as it forces processing onto the server side to guarantee reliability, which comes at significant costs in efficiency and latency.
The W3C and the future of WWW
These obvious problems with conventional web technology have not been ignored. To address problems of cross-browser incompatibility, the W3C has been working on an array of new standards for web technology. The slower pace of innovation since the end of the browser wars has allowed it to finally overtake the current status quo of web technology, and begin to propose new standards before any major implementations of those standards occurs. This will hopefully have the benefit of ensuring that future implementations by various browsers will be compatible, and compete on the basis of the quality of their implementations.
However, the drawback may be the recommendation of an undesirable new standard. This has happened before, with HTML 3.0, which the W3C intended 3rd generation browsers to support. Produced late in 1995, the spec for HTML 3.0 was viewed as being unoriginal and infeasible to implement, and the W3C was criticised for taking a long time to produce it. Neither MSIE nor NS supported it, and HTML 3.2, a standardised amalgam of the independent HTML standards devised by the two companies, eventually superseded it. The debacle of HTML 3.0 was a major blow to the credibility of the W3C, and their continuing slow pace of developing standards, including the DOM, meant that 4th-generation browsers had no common standards to support for the introduction of DHTML features in HTML 4.0.
The W3C’s newest adopted standard, XHTML, was released in advance of any commercial support. However, as it is largely no more than HTML 4.0 with stricter syntactical rules and is fully backwards compatible with HTML 4.0, its adoption has not caused major comment. However, the W3C is currently in the process of approving a raft of new standards, including XHTML 2.0, with a clear direction and design principles. However, the W3C is risking making the same mistakes with the new WWW as it did with HTML 3.
The future of the WWW is merely the future of a particular set of technologies; it is not the same as the future of the web itself; this is the problem that web2 is an attempt to solve.
Theory of web design
In order to discuss the design of the web, one needs form a proper model of web architecture independent of any particular implementation. However, the web is a set of quite different technologies, so having a unified model is difficult. The web can be viewed as operating on two levels:
Ø The network level: this relates to how users locate and retrieve web documents and interact with web servers. In the current WWW, this is simply HTTP as the network protocol, with URLs the method by which content is accessed.
Ø The content level: this concerns itself with the internals of web documents themselves, including their structure and appearance, and the relationships between them.
Diagram 2: Generalised network-level
architecture
Diagram 3: The network level in WWW
The Network Level
At the network level, the user uses a protocol such as HTTP to access content through an interface. This interface in WWW is the URL. However, it is important to note that HTTP can and is used entirely independently of HTTP.
The Content Level
The content level is split up into five progressively more concrete levels, each adding further information to the final rendered document seen by a user.
Ø STRUCTURE: relationships between documents
Ø CONTENT: the logical flow of data within a document, if any
Ø SKELETON: layout of elements within a document
Ø BEHAVIOUR: the consequences of user interaction with various elements of the document
Ø APPEARANCE: the sizes, shapes and colours of visual elements, and equivalent characteristics in non-visual browsers.
These levels have parallels in application design and information systems design. The W3C’s web technologies fit fairly clearly into these levels.
Diagram 4: Levels of abstraction at the content level
The need for Web2
This section describes the need for web2 by explaining where current web technology is not meeting current and probable future needs.
Five problems of an ever-growing web
One of the biggest problems in deciding how to direct WWW development is that there is no particular application that the web is heading towards that features need to be provided for. The web is going everywhere, and fast. Indeed, the enormous popularity of the web over all other Internet technologies leads most new users to conflate the two terms[8]. The convenience of the browser as the “universal platform” for any Internet application has led to the creation of HTTP gateways to or emulations of nearly every other Internet application, including e-mail, newsgroups and IRC.
However, expansion is in itself a goal that needs to be supported, and the web is expanding in a number of dimensions:
Ø Breadth: the number of web sites online increases constantly
Ø Depth: individual sites are become larger and more complex
Ø Complexity: pages are becoming more and more complex in terms of their structure, presentation, and the functions they are to perform; they are no longer just a static, linear flow of text.
Ø Traffic: the ever-expanding audience of the web means that even the least popular pages are visited by surprisingly large number of people; the most popular sites are already beginning to stretch the boundaries of WWW to cope with traffic levels.
Ø Audience: not only is the number of users growing, but so is their diversity and the diversity of the devices they use to access the web.
The current WWW is not adequate to handle these issues, and a number of common problems stem from its failures to handle these types of expansion. Many of these problems stem from the document-centric nature of HTML.
Breadth
Ø Web infrastructure provides few semantic clues and is complex to index accurately
The sheer size of the web means that information is hard to find. In addition, the decentralised nature of the web is a massive strength in terms of robustness, but poses a problem for navigation: not everywhere is linked to everywhere else, so if you don’t know in advance where something is, you cannot find it.
The answer to this problem has been search engines. First came the manually maintained Yahoo!, followed by the technical marvel that is Google. However, as the example of Yahoo has shown, Google cannot be expected to cope with the size of the web forever. To allow for the effective location of information on the web, better semantic data is required: there has to be more of an indication of the meaning of data, and its relation to other information than merely links. This is an issue already recognized by the W3C.
Ø Web links "break" too easily, leading to massive link-rot
The very cornerstone of the web’s functionality is the hyperlink. The interactions between pages written for a single purpose depend on it, but more importantly the ability for any page on the web to lead to any other, without any form of cooperation at the destination end of the link, is what has helped the web to grow so fast.
The issue lies not with the nature of hyperlinks, but of their destinations. URLs point to precise locations on remote servers, under no control of the page that links to them. As a result, if the URL of that resource changes even slightly, that link is “broken”. Links breaking in this way mean that a great deal of valuable information is hard to find or simply inaccessible because users cannot locate it and search engines (which work by following links) are unable to discover it. An ongoing study[9] at the time of writing estimates that the “half-life” of links (the time taken for half of the links in a random sample to break) is, on average, 55 months.
While this weakness has obviously not been fatal to the web, it is another complication in the process of accessing information on the web. A way of linking that had the ease of use of the current system but produced more robust links would be beneficial.
Depth
Ø Sites do not grow easily; HTML does not scale
At first glance, it seems nonsensical to claim that a language that has produced a worldwide collection of billions of pages does not scale well. It is a question of the level at which one looks. At the inter-site level, links allow HTML to scale perfectly. At the document level, HTML documents can be extremely large before becoming unwieldy. However, in between these levels, at the level of an individual website, HTML has a problem in that it is document-centric.
HTML is designed for the express purpose of describing a single document: it is structured to define the head and body of the document, and sections within the document. TBL designed HTML as a way of linking related documents, but did not foresee the advent of the “site” as a logical unit in the structure of the WWW. Sites are collections of pages closely related to each other, with some kind of overall control. Sites frequently have a common “look and feel” and feature repetitions of similar elements on every page to aid in navigation and often as a form of “branding”, another now common need not taken into account at the time, when the commercial aspect of the web was a minor consideration.
This document-centric approach produces, in even fairly small sites, a large amount of repetition. As individual websites grow, the task of maintaining a coherent “look and feel” across hundreds of similar pages manually becomes laborious and time-consuming. To solve this problem, an entire industry has sprung up: the field of Content Management Systems (CMS), also called Information Management Systems (IMS)[10], deals with the central management of HTML and other types of documents. In general, this is done by abstracting the look and feel to a series of templates and storing the actual data centrally, in a database. When information is required, the data is combined with the current appropriate template and supplied to the user. However, this is just a solution to the symptom: the problem itself lies in HTML.
Complexity
Ø Web applications provide a poor user experience
There is increasing use of the web not as merely an information repository and publishing medium, but as an applications platform. The web is not designed to provide the level of user experience required for desktop-like applications: these require almost immediate response to user action. This is another effect of the document-centric nature of HTML. Applications generally provide a largely stable user interface with minor changes as a result of user action. HTML is not equipped to provide incremental changes to an interface; to change an HTML page it is necessary to re-describe (and hence re-transmit) the entire HTML document. This is both slow and inefficient.
Client-side scripting goes some way to addressing this problem, but it is again a fix to the symptom of a more fundamental problem.
Ø Web documents are not efficiently modular
One of the key advances in the production of increasingly complex desktop applications has been the concept of reusability in code: the module that formats dates, or the dialog box that gives a notice of an event, can be endlessly reused with different parameters. HTML has no concept of modularity of this type. Again, the document-centric nature of HTML is to blame. The result is HTML’s inability to scale up to large sites efficiently, as already noted. This in turn limits the complexity of the interface that a site can reasonably maintain, as common functionality cannot be encapsulated and included using parameters, as functions in a programming language can, but must be rewritten every time, often with minor changes to suit the structure of the page.
Traffic
Ø The client-server model is vulnerable to traffic surges, but WWW does not take into account other possible network structures
“Flash crowds” is a term coined by Larry Niven to describe the phenomenon of instantaneous crowds of people appearing at the scene of any interesting event, in a fictional world in which teleportation is widely available. This term (also known as the “Slashdot effect”) can be used to describe the effect on a web server when a link to it is publicised on a more widely read site: a large group of users instantly attempts to simultaneously access the server. This sudden load surge brings the server to a crawl, often bringing it down completely.
Again, a great deal of effort has gone into treating the symptoms of this flaw in WWW architecture with load-balancing, caching and other methods of load distribution.
Ø XML and HTML make inefficient use of network resources
Yet another side effect of HTML’s document-centric nature, already mentioned – changing even the smallest element at the server side requires that the entire document be retransmitted.
Audience
Ø Content, presentation and behaviour are too closely linked; web languages are too closely tied to the PC form-factor
The ad-hoc nature of HTML’s development was done entirely for the benefit of a single form-factor: the conventional visual web browser running on a standard PC. As PCs have developed the screen sizes and presentational abilities expected of web pages have increased, and the brittle methods needed to force current HTML to render accurately often leave pages with minimum (and maximum) sizes to which they can be rendered accurately.
With simpler early versions of HTML, it was a reasonably simple process to translate visual elements into their equivalents for aural browsers or less capable viewers, such as text-only browsers like Lynx. However, features such as scripting are intimately linked to the visual model, with no obvious way of translating visual behaviour directly into non-visual equivalents. This is a severe problem given the growing prevalence of new form factors.
What needs to be done
In addition to mere solutions to existing problems, however, a successor to WWW must provide new and compelling features. An ideal successor to WWW would have all of these qualities:
Ø Finer control over document presentation and behaviour, opening up to web developers features which were previously difficult to access or hidden
Ø Greater robustness and maintainability both of individual pages and sites
Ø Better scalability. This includes maintainability but also management of complexity.
Ø Greater accessibility. to cope with non-visual browsers and new form factors.
Ø Improved semantics. These provide better accessibility and also allow for improved automated processing of web pages.
Diagram 5: The architecture of the WWW with reference to the 5-level model
Capabilities and failures of the WWW
To accomplish this, we will look critically at the web technologies currently existing or under development by the W3C and others, what functions they perform, and how they work together. The diagram above shows the main technologies of WWW. Many of the modules shown have multiple sub-modules that are discussed separately, and there are a number of technologies which do not fit directly into this diagram but nevertheless have some bearing on WWW.
This section also serves as a useful broad-based introduction to the technologies both complementary to and in competition with those of web2.
XML
What it is:
A highly modular SGML based generalised markup language with support for extensions to the language. Also used to refer to XML and all of its extensions collectively.
Assessment:
The development of XML was, in retrospect, a natural consequence of the popularity of HTML. The simplicity of a markup language was appealing, and as a generalised data format XML has the advantage of basic human readability: this allows for easy reverse engineering given only a small sample of the data or an XML template. In effect, all of the fairly arbitrary decisions that must be made in defining a new data format have, for better or worse, already been made in XML, which greatly simplifies the task of handling a new data format, and standard programming libraries can be used to parse, edit and create it. Its solid design also guarantees that an application starting with a very simple data structure will be able to “scale up” to larger XML documents without any design flaws.
Having established its usefulness as the default data format, however, does not automatically mean that it is the ideal language for everything. There is a new tendency at the W3C to respond to every user need with an application of, or an extension to, the XML base – a number are covered below. XML is certainly a useful data format, but it cannot and should not become the universal data format.
XML Namespaces
What it is:
A syntactical extension to XML to allow for the interleaving of multiple XML-based formats in a single document, to avoid clashes between identically named tags.
Assessment:
XML namespaces are one of the “side effects” of the extension and expansion process that produced XML. In order to have maximum flexibility, the additional complexity of XML namespaces was logically required. While a fairly sensible solution to the problem, XML namespaces considerably detract from the readability of XML, and hence hinder the process of writing it, as well. It is desirably to avoid the use of namespaces if at all possible.
XHTML 2.0
What it is:
EXtensible HyperText Markup Language. An application of XML intended to replace HTML as the basic language of web documents. Like XHTML 1.1, XHTML 2.0 is highly modularised to allow differing levels of support by different types of user agent. The XHTML base provides for basic text- and block-level formatting as well as links, lists, tables and image maps as well as allowing the embedding of scripts, style sheets and inline rendered objects (such as images, flash animations and Java applets). It also includes a list of global attributes that allow any element to become a link or be represented by an external source such as an image.
Assessment:
XHTML 2.0 is significantly backwards incompatible with XHTML 1.0 and HTML 4. This is a fairly risky move on the part of the W3C, as the level of incompatibility – the removal of the IMG tag in particular – is such that older browsers attempting to interpret the content will fail to render large parts of it. This will necessarily delay the adoption of XHTML 2.0 as a standard for a long time, since a “critical mass” of users with browsers that are XHTML 2.0-capable must build up before features specific to that version can be used.
Apart from the adoption curve, XHTML 2.0 is a much more logical, rational way of describing documents suitable for the web than HTML. However, although it changes much of the way current web functionality is implemented, it does not add a great deal of new functionality. Indeed, much of the old functionality, such as web forms and frames, has been exported to external modules.
XHTML 2.0 goes some way towards solving the current problems of the web. By favouring semantic over presentational markup, it provides greater opportunities for accessibility and portability. However, it makes little progress in other areas.
XLink
What it is:
A generalised specification for multi-purpose hyperlinks between XML documents, XLink is a generalisation of the concept of hyperlinks in HTML.
Assessment:
XLink is a good example of the W3C’s contributions to WWW technology: the production of a precise information theoretical basis for hyperlinks, and a more general specification of which hyperlinks as used in the WWW are just one application. While providing possibly many uses for general XML, it does not make any clear contribution to the WWW itself, which already had hyperlinks. It is also extremely verbose.
HLink
What it is:
An application of XLink that specifies links as used in web documents, HLink is still under development at the W3C as a working draft. HLink is a small specification that allows the very precise definition of any tag as a form of link, with restrictions on how that link is activated and what the effects of its activation are.
Assessment:
It is not hard to see the use of being able to define objects as links to the level of detail provided by HLink. However, HLink appears to be an unnecessarily verbose and complicated way of achieving this functionality.
XFrames
What it is:
An additional module of XHTML, XFrames is an application of XML to provide functionality equivalent to frames in HTML while resolving many of the usability and security issues created by the initial implementation of frames.
Assessment:
XFrames is a peculiarly unimaginative formalization of the concept of frames, being limited still to combinations of horizontal and vertical panes. While frames are a partial solution to problems of scalability and efficiency of transmission in WWW, this implementation of the concept still leaves much to be desired.
XForms
What it is:
Another application of XML, XForms is an “embedded” (rather than standalone) language type that can be used in XHTML as well as other XML applications. XForms solves one of the basic problems of web forms, which is that two forms on the same page cannot be interlaced. It also provides innovations such as submission of data as XML, divorcing the form’s data model from its presentation, and built-in declarative data-validity checking.
Assessment:
XForms is a very creative addition to web standards; it is both powerful and flexible. It is also extremely verbose, and complex. As with many of the other “new” W3C standards, formalising and rationalizing the standard has complicated it, creating a barrier to entry for newcomers.
CSS3
What it is:
Cascading Style Sheets, level 3 are the latest iteration in a specialised language used for describing the presentation of (originally) HTML or XHTML documents. The major benefit of CSS is that it can be used to declare certain presentational characteristics for an entire class of elements at a stroke, cutting down significantly on duplication as many documents can then share the same style sheet.
Currently a work in progress, CSS3 is not sufficiently developed for conclusions to be drawn about its contribution to the WWW. It includes proposals for functionality such as “Behavioural style sheets” which attach “behaviour” (defined as portions of ECMAScript) to certain elements.
XSL
What it is:
XSL represents the eXtensible Stylesheet Language, which is actually a family of three languages: XSLT, XSL-FO, and XPath.
Diagram 6: The relationship between XML, XSLT and XSL-FO
XSLT
What it is:
XSLT is the XSL Transformation language[11]. It provides the ability to describe the transformation of a generic XML document into any other, and is itself an application of XML.
Assessment:
XSLT is an impressive technical achievement, but is a good example of the principle that just because you can do something doesn’t mean you should. XML is a general data storage format, and a template for transforming XML is indeed a form of data. However, all programming languages are also forms of data, but they are not written in XML because the syntax is simply unsuitable. XSLT is a program for transforming XML, written in XML. This is interesting, but not useful. XSLT is hopelessly verbose and completely unreadable by humans. Since humans must also write XSLT directly, it makes more sense to perform this function using a more concise, dedicated syntax. XSLT is not a language meant to be rendered and displayed, as XML often is, but to be written and then processed in a procedural manner. This is a poor application of XML.
XSLT’s contribution, such as it is, is more to the area of general data storage (the scope of XML) and has little direct application the WWW, except in that it is expected to be the language by which generic XML is converted for the purposes of publication into XHTML 2.0. This process would require an author who had already developed an entire XML document format to learn two new ones – XHTML and XSLT – in order to produce a single document in XHTML. This is a laborious and time-consuming approach.
XSL-FO
What it is:
FO stands for “Formatting Objects”. XSL-FO is an entirely presentational application of XML that extends XML’s role as the universal data format, to allow the storage of purely presentational data. XSL-FO is intended to be an intermediate stage between arbitrary XML and standard rendered formats. The idea is that a document written in XML-FO can than be converted into other document formats such as postscript and PDF.
Assessment:
A model in which all data is stored as entirely semantic XML but also needs to be rendered presentationally naturally requires XML-FO. XSLT is the language for translating between these two extremes. However, both PDF and postscript already make claims to being universally accessible presentational formats, and XSL-FO lacks the ability to embed, for example, visual data, as the other formats do. It is an arguably useful addition to general data formatting. Its contribution to WWW standards is less positive; it is another piece of the unnecessarily complex XML-XSL data model.
XPath
What it is:
An application of XML that defines a syntax for addressing parts of an XML document, independent of the tag vocabulary used by that document. This functionality is used by XPointer for creating links, and XSLT for defining templates.
Assessment:
This sounds like a lot of fuss over a very simple concept, and it is.
XPointer
What it is:
This is a family of specifications that define various capabilities for addressing the structure of XML documents. It is not intended as a standalone language, but is used in other languages such as XLink and subsidiary languages. XPointer is an extension to the functionality provided by XPath.
Assessment:
XPointer is not closely enough related to the actual functions of the WWW to make much difference. It is, however, an absurdly complicated specification for such an ostensibly simple function.
RDF
What it is:
Resource Description Framework is an attempt to add semantic data to the WWW. It provides “metadata” for documents, i.e. information about the information in the document. RDF makes “statements” about “resources” which have “ properties”. The motivation is to provide machine-readable semantic information to web documents than can be used for automatic discovery and interaction, hopefully between semi-autonomous “agents”. To do this, the current specification uses an XML vocabulary. However, its makers state clearly that RDF is not restricted to XML syntax.
Assessment:
The statement of the possibility of a non-XML syntax for RDF is a tacit admission of the troublesome verbosity of the syntax that has been developed. RDF is a very positive step in providing explicit semantic information for and about web pages, but the syntax developed leaves something to be desired: its verbosity also means it is not particularly easy to read, and hence difficult to create manually, which is a problem since metadata must, by definition, be created manually by a human reader who can put summarise the document and put the information in context.
CC/PP
What it is:
Based on RDF, Composite Capability / Preference Profiles is a method of describing and resolving the differences between the requirements of a document for display and the abilities of the user agent or proxies. The aim of CC/PP is to aid portability by allowing a document to be automatically translated from its “full” form to a form that can be both transmitted and interpreted by another device, such as a simple text-mode browser running on a mobile device.
Assessment:
Portability and accessibility are both major goals of CC/PP. However, the standard devised is simply a framework for this process: no real details of how the translation is accomplished are considered. This is a step in the right direction for WWW portability, but it is by no means complete.
ECMAScript / JavaScript
What it is:
A procedural, loosely typed “scripting” language with a relaxed syntax. ECMAScript is the formalised version of Netscape’s JavaScript, and is almost identical bar a few low-level details, such as date and time formats and some function names. ECMAScript generally acts upon a hierarchical tree API representing the structure of the document (called the Document Object Model or DOM). It is used to provide dynamic client-side interactions within web pages, such as basic calculation and changes to the appearance of the document. However, ECMAScript is hindered by the lack of a common DOM between the major browsers.
Assessment:
ECMAScript (henceforth “JavaScript”) was an excellent addition to the functionality of the WWW, although its precise method of integration with HTML was haphazard and illogical, a result of the type of development that took place during the browser wars. As a method of providing interactive functionality to web pages, however, its procedural model is not ideal. Procedural code is necessarily linked very closely to its expected operating environment, such as a standard visual web browser; it is not easily portable to a completely different environment such as an aural browser or mobile device. This limits the type of functionality that JavaScript can portably provide, and since no other methods exist, it also limits the type of interactive behaviour that can be portably designed. This is one of the biggest hindrances to the development of more complex web applications.
XUL
What it is:
XUL stands for eXtensible User interface Language. XUL is an application of XML, but it is not associated with the W3C and is not a standard of any kind. Of the major browsers, XUL is currently only supported by Mozilla and other variants of the gecko engine. XUL allows the definition of a user interface for a desktop-like application.
Assessment:
Although not officially adopted by the W3C, XUL is a well thought-out and usable method of defining user interfaces. It is a good use of XML as a descriptive language to replace the procedural code of programming languages that usually describe interfaces. XUL has useful concepts for producing web applications but cannot be directly applicable, being too focussed on the production of desktop applications: it is not portable, and makes many assumptions that cannot be made of web documents.
What the proposed standards lack
The work the W3C has done in recent years on XML has without a doubt been ground breaking, sensible, and useful. XML is a genuinely usable, versatile format and this has been proven by its rapid and widespread adoption in hundreds of applications: it has, for instance, overnight become the language of configuration files for small applications. However, this development has been clearly direct towards the more general, widely applicable aspects of XML than focussed on improving the WWW. In producing these new standards, the W3C is moving away from three main aspects that made the web popular: simplicity, ease of use, and readability. This is a side effect of making the technology more powerful, but it must be addressed.
While addressing many problems, the new standards also fail to fully address the problems of the growing web previously discussed. A lot of this has to do with the continuing document-centric bias of these standards (see “depth” in the problems of the web). Web2 aims to solve the problems, rectify the deficiencies, and add genuinely new functionality to the web.
The web2 Architecture
Now we will look at the proposed web2 architecture, starting with an overall look at the structure and design goals of web2, then a more detailed look at each of the technologies in turn.
The aims of web2
The purpose of web2 is to make the web easier to use. All other goals in web2 are secondary to this central principle.
The reason the WWW became popular was that it was easy to use: web browsers are intuitive pieces of software, and any user could learn how to write HTML simply by looking at the source of another web page and modifying it. When the web was invented, this ease of use was a side effect of the underlying simplicity of the protocols and technologies. However, there is no reason a powerful technology cannot also be made easy to use. The design of the web is just a user interface problem.
Design goals of web2
The main design goal of web2 is simplicity and ease of use. However, there are a number of secondary goals concerned with solving the current problems of the WWW.
Ø Good control of presentation, already mainly achieved by the WWW.
Ø Better user experience, including producing a better model for interactivity.
Ø Greater control over complexity. This will involve improving maintainability and scalability
Ø Greater robustness at all levels, including more robust linking and better managing of the problems of high load on the client-server model.
Ø Device independence will require better semantics and improve accessibility
Web2 as a disruptive technology
It is not enough
that web2 be a useful technology. To gain widespread adoption, web2 must also
have a compelling reason to use it. Web2 aims to do this by being a disruptive
technology.
A disruptive
technology is a term coined by Clayton Christensen in the frequently quoted article
“The Rules of Innovation”[12].
In it, he sets out many tests that a technology must pass if it is to be
successful. Two of the most important are the following:
- Does the innovation enable less-skilled or less-wealthy customers to do for themselves things that only the wealthy or skilled intermediaries could previously do?
- Does the innovation target customers at the low end of a market who don't need all the functionality of current products? And does the business model enable the disruptive innovator to earn attractive returns at discount prices unattractive to the incumbents?
In the context of web technology, these questions must be slightly modified, but web2 appears to fit.
Many of the current failings of the WWW can be solved by the application of other technologies – such as Content Management Systems for the problems of scalability and complexity. However, by solving these problems and moving many of the features of these solutions into the domain of the architecture itself, web2’s focus on simplicity fits the first criterion. Secondly, by sacrificing some of XML’s flexibility for greater simplicity, web2 targets the “low end market” of the WWW: the web hobbyist.
The hobby web author is one neglected by the developments of the W3C, which has concentrated on improving the feasibility of automated processing of documents. A hobby author maintains a small to medium-sized website in her spare time, perhaps with the assistance of a few friends. The site is produced on low budget or for free, with little or no financial involvement. The technical level of the authors is mixed, but they are not web professionals, and prefer to use pre-made solutions than develop their own. Technologies that require significant technical background or are not available for free are avoided.
Small websites of this kind make up a huge portion of the most useful sites on the web. They are the ones most constrained by the web’s inability to natively scale and its poor maintainability, as they are least able to provide extra resources to their sites. A technology such as web2 would be widely welcomed by this group.
Standards & licensing
In order to be widely adopted, web2 must become an official standard. This will involve submitting web2 to the W3C. As such, web2 will be entirely free. There is simply no scope for non-free web languages given that the majority of web pages are not produced for profit.
Incremental adoption
The integrated nature of web2 is one of its advantages. However, a completely integrated system would require that all give elements of web2 be in place – on the client and server sides – in order to work. This is not feasible from an adoption point of view. The various portions of web2 can operate independently of each other by interacting with existing WWW technology. Each portion comes with its own benefits to spur its adoption in addition to the further advantages that come from using the portions of web2 with each other. The exception to this rule is shape2 and face2, which are too closely linked to be used separately.
Hereafter we will divide into the separate but linked content and network levels of web2.
The content level: shape2 and face2
The syntaxes of web2 and face2 are discussed in the appendix A.
Diagram 7: Web2 at the content level with reference to the 5-level model
A high-level language for the web
The issue of power versus simplicity in computer science is not new, and it has been solved before. The solution is a high-level language.
The W3C’s extensions to the WWW have added more than enough power and flexibility to web languages, at the cost of being verbose and unwieldy in large amounts. High-level languages take common programming structures and idioms and condense them into simple structures. This allows programmers to solve complex problems by tackling the problem at a higher level of abstraction. The advent of high-level languages allowed programmers to create applications that were orders of magnitude more complex than their predecessors in this way. In the same way, a higher-level language for the web will allow web applications to grow beyond their current limitations.
Inheritance and reusability
Another facet of high-level languages is the concept of inheritance. This has been partially used by cascading style sheets, but applying the idea of classes and inheritance to web languages more generally can greatly ease maintainability and improve scalability of web applications. By making possible generic, inheritable “classes” of descriptive code using shape2, it is possible to create complicated designs and functionality simply by using existing code templates and applying new styles, as desired. This allows competent web designers to create “libraries” of commonly used structures and use them across entire sites, as well as re-use them in different sites without any modification.
This functionality is accomplished using a raft of new tag attributes and such as PARENT, LOCID and new tags such as ELEMENT.
Libraries: scalability and maintainability
The ability to produce a single common “library” of content-level functions in shape2 will have cumulative benefits for the web as a whole, as authors will be able to share and improve the best libraries, much like program code is currently shared at programming communities such as SourceForge[13].
Using libraries of templates has other benefits:
Ø Maintainability: authors can centrally control the appearance of complex structures across an entire site without re-coding
Ø Scalability: by significantly reducing the amount of repetitive code between pages on a site, one reduced the amount that a user must download. This will have a minimal positive impact on download times for individual users, but sites as a whole will significantly reduce their bandwidth usage, allowing them to serve large numbers of users on the same server
Diagram 8: A library of templates can be combined with content to produce a complex page structure in an efficient manner
Integrated behaviour and appearance; descriptive behaviour
Web2 also integrates behaviour and presentational markup in face2. This initially simply resolves the integration of JavaScript into HTML in a scaleable way: behaviours can be managed centrally like styles can. Instead of requiring “onClick” and other attributes to be part of markup directly, they can be described using the same CSS-like syntax of face2.
However, face2 goes further than this by using the concept of descriptive behaviour. A great deal of scripting is often necessary to produce even quite simple behaviour, such as changing the appearance of elements in response to user interaction or obscuring or revealing different parts of a user interface. Indeed, a large portion of the scripting currently employed on websites performs only these functions. However, the necessity of learning a procedural scripting language is a major barrier to increasing the complexity of a website, especially for “hobby” web designers. In line with web2’s general goal of simplifying the process of web content creation, face2 provides a simplified method of producing quite complex behaviour using a functional approach: instead of describing what to do, face2 uses CSS syntax to describe what changes.
This functional approach is also semantically sounder. A non-visual user agent can respond to changing values on a web form or other events without trouble. However, how can a visual user agent judge the significance of the movement of a page element by a certain number of pixels? By separating real procedural calculation from mere visual changes, the semantic significance of scripting is improved. Conversely, user agents that are unable to represent complex visual formatting (such as small form-factor devices) will be more easily able to interpret (or ignore) visual changes when they are distinct from semantic ones.
A better alternative to frames: incremental loading
A change to an existing universal attribute, TARGET, and modifications to the behaviour of browsers allows a superior model for handling partially static pages than HTML frames did. A browser that understands shape2, when following links with the TARGET value set to “_replace”, will replace elements of the currently displayed page with specified elements of the page being loaded, via the same mechanism by which data is inserted into shape2 “classes”. This behaviour can also be specified at the side of the incremental page using face2 style sheets.
Diagram 9: A fully described page can be combined with incremental data to produce a similar but different page in a more flexible manner than using frames.
This can provide identical behaviour to HTML frames,
as well as being significantly more flexible than frames, solving such problems
as the need to change two visually distinct elements of the page that do not
naturally fit into a single vertical or horizontal pane – a significant
drawback of frames.
Design principles
The design of shape2 and face2 was made with several guiding principles in mind. Of these, three are of highest importance. Web2 languages must be:
Ø
Simple
Ø
Concise
Ø
Readable
Other benefits, such as being easy to learn and easy to write, both key factors in the initial success of HTML, accrue from these qualities.
Shape2 and face2 were designed primarily to make functionality that is currently complicated or difficult to achieve simple and easily within the reach of the “hobby” author. Theoretical elegance is deemed less important than a usable implementation: web2 is a web language designed for web developers by a web developer to make websites easy to write and powerful to use, in the minimum of time.
A note on accessibility and design focus
Web2 is unabashedly focussed on improving the web experience for the majority of current web users, who use ordinary browsers at standard screen resolutions and sizes to access content that is primarily visual in nature. This is not an oversight, but a conscious design decision. No web standard that improves accessibility is going to be adopted if it complicates development further at no perceptible gain to the user, or if it detracts from the quality of the user experience for the majority of users for the benefit of a minority. Accessibility is not ignored: the guiding principle is that improving semantics will automatically improve accessibility.
Since improved semantics is an expressed development goal of web2:
Ø
Improved accessibility will
be a side effect of good design
Furthermore, since simplicity is an overriding principle of all of web2
Ø
Accessibility must require
minimal or no additional development effort
Web authors are notoriously lazy about using accessibility features. If improving accessibility is difficult, time-consuming or non-obvious, then web authors will not do it.
Ø The best way to ensure that documents in a web language are accessible is to make it easy to write accessible documents
Semantic vs. presentational markup
One of the recurrent issues in the design of web technologies has been the debate of semantic vs. presentational markup. In a very early version of the WWW specification, TBL criticizes one proposed feature for indicating “low level format[ting]”[14]. Later on in the same document, however, he favours the presentational tags B (bold) and I (italic) over “logical” (semantic) tags such as “subscript”, “superscript”, and “emphasis”. His stated reason for rejecting semantic tags at the time is that “there are never enough of them, so people reuse them on the understanding that they will be bold, etc.”.
Subsequently, however, the W3C has refocused on the semantic nature of web documents in response to a need for greater portability and accessibility of web documents. Recent versions of HTML have in fact introduced all three of the semantic tags mentioned above and deprecated presentational tags such as B and I as well as FONT and a range of others.
However, there is still a major demand for presentational control in web languages.
Web2 regards control of presentation and semantics both as high priorities. Web2’s solution to the conflict between these two possibilities is a clearer separation of content, structure, and presentation than is present in HTML. These are useful side effects of the use of inheritable data structure:
Ø By eliminating repetition of structures, the remaining content is semantically more meaningful
Ø Incremental loading means that a software agent indexing a site can be automatically aware of what has changed from page to page within the site. This data is likely to be semantically the most important. This is useful information for an automatic parser.
Ø Even without incremental loading, documents that make extensive use of libraries are simply collections of content. This further eases automatic semantic processing.
By divorcing content from structure, web2 provides fine-grained presentational control without sacrificing semantics and without requiring additional development.
The Network Level
Diagram 10: The WWW network-level architecture
Diagram 11: The web2 network-level architecture
Better URIs
What exactly is wrong with current URIs? Already been mentioned as one of the problems associated with the growth of the web, it is that while links are often permanent, URIs are much too ephemeral. Links need to be made more robust. However, attempting to do so by changing the nature of links themselves would be fatal: the web is founded on the ease with which links are created. To make links more robust, some kind of communication between the source and destination of the link would be required; requiring this kind of communication would significantly hamper the creation of links.
However, there is much that can be done to resolve the situation at the server side. Doing so has a number of beneficial side effects at the client side separate to simply making better links.
Divorcing URIs from physical locations
Currently, a URI takes the following form:
[protocol] :// [domain] / [path]
Path is a location within the file system, relative to the document root of the web server. However, for any particular domain, path can be arbitrary – there is no stipulation that it must represent a real file path, and large numbers of CGI applications make use of this fact to produce aesthetically pleasing URIs for their results. However, because this involves considerable extra effort and software to do, most of the time the URI of a file corresponds directly to its physical locations. Physical locations are often chosen for purposes of administrative simplicity (such as placing a collection of templates in a /templates directory, or images in an /images directory), or in response to quirks in the file system of the server.
There is no reason that this should be so, and it is against the platform-independent foundations of the WWW. Link2 makes use of the path to give documents semantic locations, which has many useful effects.
Packages & semantic locations
Diagram 12: A document with a single
physical location may have more than one semantic location
A semantic location is a location based on the meaning of a document, or its relation to others. Since a document can have many different meanings in different contexts, it therefore follows that a single document can have multiple semantic locations. Link2 groups semantically linked documents into packages, which are organized hierarchically into a “primary tree”. However, documents can belong to an arbitrary number of other packages, allowing “secondary trees” to be produced when desirable.
Link2 allows documents to define their own semantic locations via extensions to the shape2 language contained in the HEAD of a document using the PACKAGE tag; a statement of membership to a package is called a claim. The first claim is known as the primary claim (used to produce the primary tree). The full syntax of the link2 extensions is described in appendix C, including the rules that govern precedence of claims if two physical documents claim the same semantic location.
The various aspects of these extensions confer different advantages to the system:
Ø Queuing and expiring
The BEGINS and ENDS attributes of a claim allows documents to more effectively author time-sensitive documents, behaviour that is currently addressed only in the most basic way using the EXPIRES attribute of the META tag in HTML.
An author can produce a series of documents with a “begins” value set in the future. The server will not serve these documents until the time defined, allowing an author to “queue” a series of documents. This can be enormously useful to sites such as news organizations that prepare large volumes of material in advance. A feature common to many Content Management Systems is an automated system to physically move documents to and from publicly accessible locations at specified correct time to produce this behaviour.
Ø Versioning
A series of documents claiming the same semantic location can be produced with differing values for their “begin” and “end” times. The server applies precedence rules to ensure that it serves the most current document. This allows a server to be used as a storage location for many versions of a document without the risk of serving the wrong one or the labour of constantly “swapping out” older versions for newer ones.
When link2 is used in conjunction with talk2, this functionality improves, as talk2 has the facility to request versioning information about a document. This allows users to view not only the current version of a document, but if desirable any number of previous versions.
Ø Archiving
Another positive effect of multiple semantic claims within a document is gained when the client also gains the ability to read this content. By interpreting the “ends” values of a claim, a client which wishes to make a “book-mark” to a documents (i.e. a static link at the client side) can choose the longest-lasting link. A well-designed site using link2 extensions can thus provide “archive” locations for topical information in a very simple way.
Ø Semantic “book-marking”
The compound nature of web documents means that a single semantic location may have elements drawn from many sources, all of which may have additional separate semantic locations. For example, the front page of a site may contain excerpts from many sub-sections of the site, or if it is a “weblog” contain many articles published in sequence that are semantically separate. The embedded semantic information provided by link2 extensions allows a compatible client to bookmark the content based on longer-lasting semantic locations.
Medium independence
Diagram 13: By abstracting paths to semantic locations, the dependence on the file system is removed
A useful side-effect of link2 separating URIs from the file system is that it provides the possibility for an architecture that can more easily draw web content from any storage medium, rather than just flat files. This is, essentially, the function of CGI, replicated with the addition of semantic data. However, the architecture of link2 is not of primary importance: the key feature is the ability to divorce physical from semantic location.
So far the discussion has focussed on the semantic of shape2 and compatible files stored in the file system. The link2 extensions take into account the semantics of non-markup files; see appendix C for details. The semantic location of information obtained from sources other than the file system is defined by those sources. In databases, it might be a primary key or another identifier. However, the heterogeneous nature of the possible other media means that link2 cannot rigorously define how semantic locations should be determined in every case.
A richer protocol
Talk2 is an extension to HTTP to allow distributed authoring, versioning, archiving and publishing of web content. These extensions cover functionality traditionally provided by code-sharing applications such as CVS, including file locking, check-in and checkout. Talk2 also includes additional functionality that allows it to take advantage of features in the other portions of web2, including link2 and WebTorrent. Much of the functionality required by talk2 absent from HTTP is already present in WebDAV[15] and its successor Delta-V[16].
The requirements of the talk2 protocol are discussed in appendix B.
Distributed authoring
HTTP has traditionally been a “publishing” protocol: it is used by many clients to request documents from the server. Extensions have allowed HTTP to be used to submit the content of web forms to documents and upload files to servers, but these are “patch” solutions rather than a genuine re-evaluation of the purpose of HTTP.
Some of the most popular and innovative new applications of HTTP – weblogs and “wikis”[17] – involve continuous editing and addition to content, rather than the simple publish-once model of the early WWW. However, these currently require a great deal of additional software on the server side to allow this sort of editing. This functionality would be better incorporated into the structure of the protocol itself: this is the purpose of WebDAV.
Distributed authoring is more than simply creating an “edit” function in HTTP. By adding file locks and other familiar functions of CVS, a website can become a genuine tool for collaboration rather than publishing; this is the functionality provided by Delta-V.
Semantic protocol information
The main extensions to the Delta-V specification present in talk2 are concerned with integrating into other aspects of web2 to provide greater semantic content.
Ø Package retrieval
If a server uses link2 as well as talk2, a client should be able to retrieve information about packages as a separate operation. This will allow browsers displaying pages from a certain package to present information about the package to the user in useful way, as a list of “related” links, as well as links to parents and children of the current document.
Ø Versioning information
As already mentioned, talk2 clients will be able to request versioning information about documents, to allow the retrieval of older documents that have since been superseded (assuming the author wishes to allow these documents to be retrievable).
Distributed caching and load balancing
WebTorrent: the peer to peer web
WebTorrent is unique in the components of web2 in that it does not augment or replace an existing technology: WebTorrent adds an additional component to the network model. WebTorrent attempts to solve the problem of overload on the client-server model at peak times using a distributed, peer-to-peer caching system. A side effect of solving this problem is that a WebTorrent client becomes an efficient load-balancing technology that is extremely easy to set up compared to other systems, making it a good candidate for a disruptive technology.
BitTorrent
WebTorrent is based on the concept underlying an existing technology called BitTorrent[18]. The concept is fairly simple: if an individual wishes to share a large file (BitTorrent is mainly used to distribute movies[19]), instead of distributing the location of the file itself, they distribute a link to a BitTorrent file (which ends in the extension ".torrent"). When a user attempts to download a torrent file, if nobody else is downloading the file at that time, the torrent-enabled client downloads the file as a standard HTTP connection from the main server. However, if several users are downloading the file, the .torrent server communicates with client and instructs it to download portions of the file directly from other users who have already downloaded those portions from the main server. The result of this is that the main server is less heavily loaded and so is able to send the complete file to the first users without being swamped by subsequent requests from later users. The end result is a quicker overall download for all users while the server never becomes overloaded.
Extending the concept to the web
One of the problems with the nature of the web is that while the web itself is distributed, individual web sites are not: popular sites are subject to the Slashdot effect[20] and other major events which cause flash crowds and overload the server. Sites try to combat this in a basic way by providing mirror sites and companies such as Akamai[21] provide more systematic, commercial services to combat this problem. Once again, however, these are solutions to symptoms of a problem rather than a solution to the problem itself, which is simply that the client-server model is not adequate for such a highly volatile environment as the web.
Enter WebTorrent
A computer with WebTorrent will access websites by their ordinary URLs. If a user wants a web page that no one has downloaded recently, it is downloaded directly. If lots of users are viewing a website, however, the WebTorrent clients are able to discover each other and download the site from each other using P2P methods. The result is distributed web caching. Particularly interesting is the fact that since this model requires no changes to the website itself, it is not a method of distributing a single website, but it simultaneously solves the same problem for the entire web. The more WebTorrent clients exist on any network, the more useful and robust the network becomes.
WebTorrent Architecture
WebTorrent "clients" are actually localized proxy servers for both talk2 and HTTP: this allows any application that uses HTTP or talk2 to benefit from the WebTorrent network, without requiring redevelopment. Since WebTorrent will not always be effect (see below) it also allows users to bypass the WebTorrent network when this is convenient.
Diagram 14: A portion of a WebTorrent network
However, the basis of the WebTorrent architecture is simple:
Ø Ordinary users browse the web running WebTorrent clients (nodes) that are in communication with supernodes.
Ø As they download websites, nodes cache the content locally and calculate a checksum for that content, based on the date of last modification, the URI of the files, and the content of the files themselves.
Ø Nodes periodically indicate their current inventory of checksums to the supernodes they are aware of.
Ø Supernodes automatically discover other supernodes on the network and supply nodes with a list of these supernodes. This allows nodes to switch to other supernodes in the event of failure or to improve speeds
Ø When retrieving content, nodes also send out duplicate requests for that URI to their local supernodes
Ø If a supernode is aware of a node with the desired URI and a valid recent checksum, it returns this to the requesting node
Ø Based on ping times and the relative speed and availability of the main server, the requesting node can then either rely on the content response of the server, or make a new content request directly to peer node(s) for the content.
Diagram 15: The source of the content is dependent upon the availability and response time of the server
The frequency with which the WebTorrent network is used in favour of the direct client-server model will greatly depend on the efficiency of the implementation and the properties of the resultant P2P network (which are difficult to predict without an implementation). However, the main purpose of WebTorrent is not to increase the speed of the web, but to guarantee the availability of web sites under heavy load.
Not a universal solution
The WebTorrent network is not always the solution. In order for WebTorrent to work, substantial numbers of people must be simultaneously downloading exactly the same web content. Pages that have randomly-generated content such as some types of banner ads, will not get the full benefit of WebTorrent. The increased distinction between static and changing content provided by shape2 will improve the efficiency of this. If shape2 is not available, pages can be modified using IFRAMEs and other existing technology to optimise pages for WebTorrent, if this is desired.
WebTorrent also cannot provide for websites that have lots of personalized server-side processing: a site such as Hotmail or Amazon could not run as-is using WebTorrent, and this would not in any case be desirable, as it would present a significant risk to security and privacy of personal data. Again, the use of shape2’s static libraries and other features would allow these sites to at least partially benefit from WebTorrent, for instance to distribute the main interface libraries for the site, decreasing the load on the main servers.
WebTorrent is suited for user-neutral content, i.e. non-personalized content. This describes by far the majority of the content currently on the web and is expected to remain the case for the foreseeable future. In this arena, WebTorrent would provide significant benefits.
Part 3: Link2Apache
Link2Apache is an implementation of the link2 architecture for the Apache web server. The main purpose of producing link2Apache is as a proof-of-concept implementation of link2.
Implementation goals
Ø To prove that the link2 architecture can be successfully implemented
Ø To show that the rules of claim resolution and package formation are practical
The choice to implement link2Apache was on the basis of simplicity: shape2 and face2 are too closely integrated to implement separately, and implementing the entire content level portion of web2 was deemed infeasible in the time frame, given the amount of time required to develop the language itself.
Choices of hardware and software
Apache
Link2Apache is designed to function as an Apache module. A number of reasons lie behind this decision:
Ø Apache is the world’s most popular server, with close to two-thirds of the worldwide web server market[22]
Ø The Apache API allows third-party programs very close integration with the internals of the server, and Apache was chosen for this reason as the basis for the implementation.
Ø Documentation for the Apache server is well-written and widely available, and online support groups are plentiful
Ø Apache is open-source and free to use; this is convenient for a research project
Win32
Link2Apache is written for the Windows platform, as this was the environment most familiar to the author.
Programming language
Initially link2Apache was to be written in C and use the direct Apache API. Early in the course of implementation, this changed to using Perl and the mod_perl API. For the reasoning behind this decision, see below.
Implementation
Process of implementation
The implementation of link2Apache was not smooth. The author comes from a Java background with significant experience in web scripting languages. From this perspective, the learning curve of mastering the Apache API as well as the C language simultaneously was judged too steep. To speed implementation, mod_perl was chosen instead.
Mod_perl is a module of Apache that provides facilities for Perl scripts to interact with the Apache server through the mod_perl API. The mod_perl API has many of the same features as Apache’s C API but as a result of Perl being a scripting language development can be significantly quicker, as there is no requirement for memory management and other facets of C which can be daunting to a programmer more experienced in Java. No equivalent API exists for Apache in any language other than Perl, so mod_perl was the obvious choice.
However, Perl has its own learning curve. Compounding this issue was the decision that link2Apache must have a clear Object Oriented (OO) design. Perl’s OO support comes in the form of a selection of peculiar programming idioms, and lacks many of the niceties of the Java language, such as the passing objects transparently as methods (Perl requires that references be passed, and has limited support for complex data structures).
These setbacks significantly delayed programming progress, resulting in scaling back the original design of interfacing directly with Apache to providing a standalone proof-of-concept of link2’s extensions to shape2. This was successfully achieved. However, much of the design work for the full program was completed, and this has been included.
Program design
Diagram 16: The arrangement of classes and methods in link2apache
Link2Apache works by creating a model of the packages and claims. This is stored in a database as follows:
Diagram 17: The database structure of link2Apache
When content is requested, link2Apache finds all relevant claims and sorts them in order of relevance. This allows multiple conflicting claims to co-exist. Link2Apache then sends the claim to the relevant application (an App object). If the application returns content, the operation has been successful. If the operation fails, the next most-relevant claim is tested. In this way links are made more robust, as even several failures can still produce relevant content.
The major modules of Link2Apache and their function is as follows:
IndexTree
This class is instantiated at server start-up. IndexTree takes the parameters of a server and examines the document root of that server. It recursively searches the document tree for files containing valid link2 markup.
Link2File
Link2File is instantiated by IndexTree when it encounters valid link2 markup within a file in the document root; the file path is the parameter. Link2File parses the XML code and uses it to create Packages. To these Packages it then attaches Claims. Both these operations are performed by instantiating the relevant objects.
Package, Claim, App
These three similar classes are all subclasses of DataStore. A Package, Claim or App object corresponds to an entry in the equivalently-named databases. The DataStore class provides common methods to all for entering data into and retrieving data from the database.
Server
Another subclass of DataStore. This class is an incomplete portion of an attempt to generalise link2Apache to support virtual servers under Apache, involving multiple simultaneous document roots.
MatchURI
This is instantiated by the sever when a URI is requested. MatchURI searches the database for relevant claims using a path-matching algorithm and produces a ranked list sorted by relevance. It then instantiates the claims and calls the relevant App with the claimed URI. It repeats this until content is returned by an App or its supply of matching claims is exhausted, at which point it returns a “failed” signal to Apache, which will return an “HTTP 404” message to the client.
Important algorithms
The two most intellectually interesting algorithms within link2Apache are the used by Link2File to create claims and packages and by MatchURI to resolve requests. These are performing the proof of the link2 concept: link2 claims can represent an arbitrary fully connected graph in which documents are nodes and claims are vertices.
Project summary and conclusions
Difficulties and experience gained
The web2 project was always ambitious for the given time frame. The volume of research required combined with the open-ended nature of the development effort posed time-management problems; it was very difficult to draw a line under web2 specification and begin the implementation. In future, a clearer definition of the deliverables desired should be made and rigidly adhered to.
The problems caused by attempting to produce object-oriented Perl were many, but unavoidable given the choice of Apache for the test implementation. In future, selecting a less complex server than Apache for a test-bed implementation would be sensible.
Conclusions
The work done on web2 thus far has been both creative and useful. Much more work needs to be done on web2 before it can be submitted for consideration as a standard, but it has genuine potential as a future language of the web.
Further work
Web2 is entirely open-ended. Future work can take any number of directions:
Ø Finalization
The existing specifications need to be refined and clarified
Ø Extension
More work can be done to add further features to the specifications
Ø Formalization
The current specifications are loose and informal. To be submitted as a standard, they need to be made much more rigorous.
Ø Adoption
The standards must be submitted to the W3C for adoption
Ø Implementation
& Testing
These standards are all still theoretical. Particularly in the case of the emergent properties of WebTorrent’s P2P network, real-world implementations must be made and extensively tested.
Acknowledgements
I would like to acknowledge the kind support and assistance of the following people:
Ø Ben Morrow for endless discussions, a host of clever suggestions and for repeatedly affirming that yes, this is a better way to do things.
Ø Rik Ward for constant encouragement, morale support, and supplying badly needed discipline in the form of the immortal acronym “DSFW”.
Ø Stephen Spencer for providing the ultimate venue for online project support at irc.meta8.co.uk
Ø Moz Peachey, Kimberley Tate, Marc Bunyard and Matt Elton for being there.
Ø Sadiq Jaffer for encouragement and stimulating conversation when it was needed most
And finally to my supervisor and tutor Dr. Mike Joy for innumerable helpful suggestions, guidance and understand.
Technical acknowledgements:
- Diagram 4 is based on a concept from “The Elements of User Experience” by Jesse James Garrett.
http://www.jjg.net/elements/elements_ch02.pdf
- Diagram 6 is taken (with permission) from “What Is XSL-FO and When Should I Use It?” by Stephen Deach
http://www.seyboldreports.com/TSR/free/0217/techwatch.html
- Link2Apache makes use of a variety of Perl modules available from search.cpan.org. They are included by reference as-is, without modification, and no code included in the readable portion of the source is the work of anyone else.