THE RICH GET RICHER

Received wisdom, underscored by rhetoric from all quarters, is that the web is the universal key to information access. However, we now know that things are not as straightforward as they appear at first sight. Over the last decade or so, the network theory sketched in Chapter 3 has been developed to explain how structure emerges spontaneously in linked organisms, like the web, that evolve continuously over time; structure that is shared by all scale-free networks. The very fact of growth determines the network’s topological form. This structure, clearly understandable from the logical point of view, implies that when people publish information online, there is scant chance that it will actually be read by others. Indeed, Barabási, one of the pioneers of the study of the web as an enormous scale-free network, went so far as to observe:

How can this be? To probe further, let’s examine a 2003–2004 study that took a snapshot of part of the web and then another one seven months later. A sample of 150 websites was chosen, and all their pages were downloaded, plus the pages they linked to—a total of 15 million of them. The two snapshots had 8 million pages in common. The goal was to examine how the number of inlinks to each page changed over the period.

There were about 12 million new links. But surprisingly, a full 70 percent of these went to the 20 percent of the pages that were originally most popular in terms of their number of inlinks. The next 20 percent of pages received virtually all the remaining new links. In other words, the 60 percent of pages that were least popular in the original sample received essentially no new incoming links whatsoever! Even when the new links were expressed as a proportion of the original number of links, there was a negligible increase for the 60 percent of less-popular pages. This extraordinary result illustrates that, on the web, the rich get richer.

Suppose the increase was measured by PageRank rather than the number of inlinks. It’s still the case that the top 20 percent of the original sample receives the lion’s share of the increase, and the next 20 percent receives all the rest. But the total amount is conserved, for, as we learned in Chapter 4, PageRank reflects the proportion of time that a random surfer will spend on the page. It follows that the remaining 60 percent of pages suffer substantial decreases. Not only do the rich get richer, but the poor get poorer as well. In this sense, the web reflects one of the tragedies of real life.

Of course, you’d expect the rich to get richer, for two reasons. First, pages with good content attract more links. The fact that a page already has many inlinks is one indication that it has good content. If new links are added to the system, it’s natural that more of them will point to pages with good content than to ones with poor content. Second, random surfers will be more likely to encounter pages with many inlinks, and therefore more likely to create new links to them. You cannot create a link to something you have not seen.

But these effects do not explain why the bottom 60 percent of pages in the popularity ranking get virtually no new links. This is extreme. What could account for it? The dragons.

THE EFFECT OF SEARCH ENGINES

In practice, the way we all find things is through web dragons, the search engines. Their role is to bestow visibility on as much information as possible by connecting all relevant content to well-posed queries. They are the web’s gatekeepers: they let people in and help them find what they’re looking for; they let information out so it can reach the right audience. What’s their impact on the evolution of the web? The links to your page determine where it appears in the search results. But people can only link to your page if they find it in the first place. And most things are found using search engines.

At first glance, things look good. A democratic way for the web to evolve would be for random surfers to continually explore it and place links to pages they find interesting. In practice, rather than surfing, search engines are used for exploration. Consider a searcher who issues a random query and clicks on the results randomly, giving preference to ones higher up the list. Typical search engines take PageRank into account when ordering their results, placing higher-ranking pages earlier in the list. As we learned in Chapter 4 (page 116–117), PageRank reflects the experience of a random surfer, and so it seems that the experience of a random searcher will approximately mirror that of a random surfer. If that were the case, it would not matter whether links evolved through the actions of searchers or surfers, and the evolution of the web would be unaffected by the use of search engines.

Not so fast. Although this argument is plausible qualitatively, the devil is in the details. It implicitly assumes that random searchers distribute their clicks in the list of search results in the same quantitative way that random surfers visit pages. And it turns out that for typical click distributions, this is not the case. Buried deep in the list from, say, the 100th item onward, lie a huge number of pages that the searcher will hardly ever glance at. Although random surfers certainly do not frequent these pages, they do stumble upon them occasionally. And since there are so many such pages, surfers spends a fair amount of time in regions that searchers never visit.

To quantify this argument would lead us into realms of theory that are inappropriate for this book. But you can see the point by imagining that searchers never click results beyond, say, the 100th item (when was the last time you clicked your way through more than ten pages of search results?). The millions of pages that, for typical searches, appear farther down the list are completely invisible. They might as well not be there. The rich get richer and the poor do not stand a chance. More detailed analysis with a plausible model of click behavior has shown that a new page takes 60 times longer to become popular under the search-dominant model than it does under the random surfer model.

Google shot from obscurity (less than 5 percent of web users visited once a week) to popularity (more than 30 percent) in two years, from 2001 through early 2003. According to the preceding model, today, in a world where people only access the web through search, this feat would take 60 times as long—more than a century!

Surely, this must be an exaggeration. Well, it is—in a way. It assumes that people only find new pages using search engines, and that they all use the same one so they all see the same ranking. Fortunately, word of mouth still plays a role in information discovery. (How did you find your favorite web dragon?—probably not by searching for it!) And fortunately, we do not all favor the same dragon. Nevertheless, this is a striking illustration of what might happen if we all used the same mechanism to discover our information.

POPULARITY VERSUS AUTHORITY

Do web dragons simply manage goods—namely, information—in a high-tech marketplace? We don’t think so. Information is not like meat or toothpaste: it enjoys special status because it is absolutely central to democracy and freedom. Special attention must be paid to how information circulates because of the extraordinarily wide impact it has on the choices of citizens. When people make decisions that affect social conditions or influence their activities, they must have the opportunity to be fully informed. To exacerbate matters, in the case of web search users are generally blithely unaware not only of just how results are selected (we’re all unaware of that!), but of the very fact that the issue might be controversial.

As explained in Chapter 4, despite the internal complexity of search engines, a basic principle is to interpret a link from one page to another as a vote for it. Links are endorsements by the linker of the information provided by the target. Each page is given one vote that it shares equally among all pages it links to: votes are fractional. But pages do not have equal votes: more weight is given to votes from prestigious pages. The definition of prestige is recursive: the more votes a page receives, the more prestigious it is deemed to be. A page shares its prestige equally among all pages it chooses to vote for.

Search engines base prestige solely on popularity, not on any intrinsic merit. In real life, raw popularity is a questionable metric. The most popular politician is not always the most trustworthy person in the country. And though everyone knows their names, the most popular singers and actors are not considered to dispense authoritative advice on social behavior—despite the fact that the young adulate and emulate them. Why, then, do people accept popularity-based criteria in search engines? Do they do this unconsciously? Or is it simply because, despite its faults, it is the most efficacious method we know?

The way web pages relate to one another leads to the idea of an economy of links. Choosing to create a link has a positive value: it is a meaningful, conscious, and socially engaged act. Links not only imply endorsement within the abstract world of the web, they have a strong impact on commercialization of products and services in the real world. Their metaphysical status is ambiguous, falling somewhere between intellectual endorsement and commercial advertising. As we learned in the last chapter, the economy of links has real monetary value: there is a black market, and you can buy apparently prestigious endorsements from link farms. This offends common sense. It’s a clear abuse of the web, a form of both pollution and prostitution that stems from a failure to respect the web as a common good. Nowadays such practices cannot be prosecuted by law, but they are firmly—and, some would say, arbitrarily—punished by today’s search engines.

Search engines make sense of the universe that they are called upon to interpret. Their operators are strenuously engaged in a struggle against abuse. But despite their goodwill, despite all their efforts on our behalf, one cannot fail to observe that they enjoy a very unusual position of power. Although they play the role of neutral arbiter, in reality they choose a particular criterion which, whatever it is, has the power to define what is relevant and what is insignificant. They did not ask for this awesome responsibility, and they may not relish it. The consequences of their actions are immeasurably complex. No one can possibly be fully aware of them. Of course, we are not compelled to use their service—and we do not pay for it.

Defining the prestige of a web page in terms of its popularity has broad ramifications. Consider minority pages that lie outside the mainstream. They might be couched in a language that few people speak. They might not originate in prominent institutions: government departments, prestigious universities, international agencies, and leading businesses. They might not treat topics of contemporary interest. No matter how relevant they are to our query, we have to acknowledge the possibility that such pages will not be easily located in the list of results that a search engine returns, even though they do not belong to the deep web mentioned in Chapter 3. Without any conscious intent or plan—and certainly without any conspiracy theory—such pages are simply and effectively obliterated, as a by-product of the ranking. This will happen regardless of the particular strategy the dragons adopt. Obviously, no single ranking can protect all minorities. Selective discrimination is an inescapable consequence of the “one size fits all” search model.

There are so many different factors to take into account that it is impossible to evaluate the performance of a search engine fairly. Chapter 4 introduced the twin measures of precision and recall (pages 110–111). Recall concerns the completeness of the search results list in terms of the number of relevant documents it contains, while precision concerns the purity of the list in terms of the number of irrelevant documents it contains. But these metrics apply to a single query, and there is no such thing as a representative query. Anyway, on the web, recall is effectively incomputable because one would need to identify the total number of relevant documents, which is completely impractical. And, of course, the whole notion of “relevance” is contextual—the dilemma of inquiry is that no one can determine what is an effective response for questions from users who do not know exactly what they are seeking. A distinction can be made between relevance—whether a document is “about” the right thing in terms of the question being asked—and pertinence—whether it is useful to the user in that it satisfies their information need. It is very hard to measure pertinence. Add to this the fact that search engines change their coverage, and their ranking methods, in arbitrary ways and at unpredictable times.

This makes it clear that users should interact in a critical way with search engines and employ a broad range of devices and strategies for information retrieval tasks in order to uncover the treasure hidden in the web.

PRIVACY ON THE WEB

When defining privacy, various modalities of protection of personal data can be distinguished, depending on the communication situation. In private situations, of which the extreme is one-on-one face-to-face interaction, we are less protective of the data we distribute but in full control of our audience. In public situations the reverse is true; the audience is outside our control, but we carefully choose what information to reveal. A third communication situation concerns channels for the processing of electronically recorded information.

In this third situation, there are two different principles that govern online privacy. The first, user predictability, delimits the reasonable expectations of a person about how his or her personal data will be processed and the objectives to which the processing will be applied. It is widely accepted that before people make a decision to provide personal information, they have a right to know how it will be used and what it will be used for, what steps will be taken to protect its confidentiality and integrity, what the consequences of supplying or withholding the information are, and any rights of redress they may have. The second principle, social justifiability, holds that some data processing is justifiable as a social activity even when subjects have not expressly consented to it. This does not include the processing of sensitive data, which always needs the owner’s explicit consent. It is clear that these two principles may not always coincide, and for this reason there will never be a single universally applicable privacy protection policy. It is necessary to determine on a case by case basis whether a particular use of personal data is predictable and whether and to what extent it is socially justifiable.

User predictability resembles the medical ethics notion of “informed consent,” which is the legal requirement for patients to be fully informed participants in choices about their health care. This originates from two sources: the physician’s ethical duty to involve patients in their health care, and the patients’ legal right to direct what happens to their body. Informed consent is also the cornerstone of research ethics. While not necessarily a legal requirement, most institutions insist on informed consent prior to any kind of research that involves human subjects. Social justifiability extends informed consent and weakens it in a way that would not be permitted in the ultra-sensitive medical context.

Every user’s network behavior is monitored by many operators: access providers, websites, e-mail services, chat, voice over IP (VoIP) telephone services, and so on. Users who are aware of what is going on (and many aren’t) implicitly consent to the fact that private information about their online habits will be registered and retained for some period of time. We can assess the privacy implications by considering every piece of data in terms of its predictability and justifiability, and verify the system’s legitimacy—both moral and legal—according to these principles.

Our information society involves continual negotiation between the new possibilities for communication that technology presents and the new opportunities it opens up for monitoring user behavior. There is both a freedom and a threat; privacy protection lies on the boundary. As users, we choose to trust service providers because we believe the benefits outweigh the drawbacks. The risk is loss of control of all the personal information we are obliged to divulge when using new tools.

We are never anonymous when surfing the net: the system automatically registers every click. Service providers retain data in accordance with national laws such as the U.S.A. Patriot Act.⁴¹ Although they are not compelled to do so, many individual websites keep track of our activities using the cookie mechanism. As explained in Chapter 3 (pages 73–74), cookies reside on the user’s computer and can be erased at will, either manually or by installing software that periodically removes them, paying special attention to suspected spyware. Technically speaking, users retain control of this privacy leak. But practically speaking, they do not, for most are blithely unaware of what is going on, let alone how to delete cookies (and the risks of doing so). Moreover, the balance between freedom and threat arises again, for cookies allow websites to personalize their services to each individual user—recognizing your profile and preferences, offering recommendations, advice or opportunities, and so on.

PRIVACY AND WEB DRAGONS

It is impossible to draw a clear line between protection of privacy and access to useful services. Certainly users should be aware of what is going on—or at the very least have the opportunity to learn—and how to use whatever controls are available. Every website should disclose its policy on privacy. Many do, but if you do have the patience to find and read them, you may discover that popular websites have policies that allow them to do more than you realized with the personal data you give them.

Here is a lightly paraphrased version of the privacy policy of a major search engine (Google):

This example is typical: the policies of other major search engines are not materially different. Yahoo pledges that it will not share personal information about you except with trusted partners who work on its behalf, and these partners must in turn pledge that they will not use the information for their own business purposes. For instance, an insurance company interested in purchasing personal queries on medical conditions cannot procure them from search engines, either directly or indirectly. However, as you can see from the policy quoted here, under some circumstances personal information can be shared. Governments can access it if it relates to criminal activity, and the company can use it to investigate security threats. A further clause states that if the company is acquired or taken over, users must be informed before the privacy policy is changed. In a sense, the philosophy behind search engine privacy reflects the norm for telephone companies: they must prevent the circulation of personal data.

Most companies routinely track their customers’ behavior. There is enormous added value in being able to identify individual customers’ transaction histories. This has led to a proliferation of discount or “loyalty” cards that allow retailers to identify customers whenever they make a purchase. If individuals’ purchasing history can be analyzed for patterns, precisely targeted special offers can be mailed to prospective customers.

Your web queries relate to all aspects of your life (at least, ours do). People use their favorite search engine for professional activities, for leisure, to learn what their friends are doing, or to get a new date—and check up on his or her background. If you subscribe to their services, web dragons monitor your queries in order to establish your profile so that they can offer you personalized responses. In doing so, they keep track of what you are thinking about. When opting for personalized service, you face an inescapable dilemma: the more facilities the service offers, the more privacy you will have to renounce. However, unless you are working outside the law, your private data is not expected to circulate outside the dragon’s lair.

As an example of unanticipated circulation of personal information, in April 2004, dissident Chinese journalist Shi Tao was convicted of disclosing secret information to foreign websites and condemned to ten years in jail. Vital incriminating evidence was supplied to the Chinese government by Yahoo’s Hong Kong branch, which traced him and revealed his e-mail address, Internet site, and the content of some messages. Commenting on the case during an Internet conference in China, Yahoo’s founder Jerry Yang declared:

CENSORSHIP ON THE WEB

During 2004, Google launched a Chinese-language edition of its news service, but in an emasculated version:

Self-censorship was judged necessary in order to launch the service. In 2002, the company had crossed swords with the Chinese government, and in some areas of the country it seems that Google cooperated by withdrawing selected pages and redirecting to a Chinese search engine from Google’s home page. In summer 2004, Google invested heavily in Baidu, a government-approved Chinese search engine. Early in 2006, after a long period of uncertainty, Google finally announced that it was actively assisting the Chinese government in censoring results from its main search service.

Human rights groups responded in a blaze of outrage whose flames were fanned by the company’s self-proclaimed motto, “Do no evil.” Of course, web dragons cannot impose their service in defiance of a hostile government that is prepared to censor access from any Internet address. They have just two choices: to go along with censorship or to refrain from offering their service. Meanwhile, when Microsoft was criticized in mid-2005 for censoring the content of Chinese blogs that it hosted, it retorted:

China today presents a golden opportunity for profit in the business of web search and related technologies. Major search engines and leading Internet companies are jockeying for a favorable market position. The problem is that local regulations sometimes dictate controversial decisions concerning personal freedom. In parts of the world where free speech is in jeopardy, the enormous power wielded by information filters placed between users and Internet services becomes all too evident, as does the danger of monitoring user behavior. The flip side is that withdrawing search services could cause damage by restricting cultural and business opportunities.

Privacy and censorship are critical problems on the web. Both involve a balance between transparency and protection of individuals from information disclosure. People need to be protected both by keeping private information secret and by censoring certain kinds of information. The problem is to define the boundary between permissible and illicit diffusion of information, balancing the rights of the individual against community norms. For example, criminals sacrifice their right to privacy: when the police demand information about a cyber thief, the community’s right to be protected against illegal appropriation may outweigh the suspect’s right to privacy. Censorship is justified when transparency is deemed to damage a segment of society. Almost everyone would agree that young children should be protected from certain kinds of content: social software helps parents censor their children’s viewing.

These issues cannot be dealt with mechanically because it is impossible to draw a line based on formal principles between the community’s right to know and the individual’s right to privacy. Moreover, any solutions should respect cultural diversity. Decisions can only be made on a case by case basis, taking into account the context and the type of information. These difficult judgments depend on social norms and pertain to society as a whole. They should not be left to automatic filters or to private companies with legitimate business interests to defend. Society must find a way to participate in the decision process, open it up, allow debate and appeal. Online privacy and censorship will remain open questions. There is an inescapable tradeoff between the opportunities offered by new information technologies and the threats that they (and their administrators) pose to our core freedoms.

COPYRIGHT LAW

Possessing a copy of a document certainly does not constitute ownership in terms of copyright law. Though there may be many copies, every work has only one copyright owner. This applies not just to physical copies of books, but to computer files too, whether they have been digitized from a physical work or created electronically in the first place—born digital, you might say. When you buy a copy of a work, you can resell it but you certainly do not buy the right to redistribute it. That right rests with the copyright owner.

Copyright subsists in a work rather than in any particular embodiment of it. A work is an intangible intellectual object, of which a document is a physical manifestation. Lawyers use the word subsists, which in English means “to remain or continue in existence,” because copyright has no separate existence without the work. Copyright protects the way ideas are expressed, not the ideas themselves. Two works that express the same idea in different ways are independent in copyright law.

Who owns a particular work? The creator is the initial copyright owner, unless the work is made for hire. If the work is created by an employee within the scope of her employment, or under a specific contract that explicitly designates it as being made for hire, it is the employer or contracting organization that owns the copyright. Any copyright owner can transfer or “assign” copyright to another party through a specific contract, made in writing and signed. Typically an author sells copyright to a publisher (or grants an exclusive license), who reproduces, markets, and sells the work.

The copyright owner has the exclusive right to do certain things with the work: thus copyright is sometimes referred to as a “bundle” of rights. In general, there are four rights, though details vary from country to country. The reproduction right allows the owner to reproduce the work freely. The distribution right allows him to distribute it. This is a one-time right: once a copy has been distributed, the copyright owner has no right to control its subsequent distribution. For example, if you buy a book, you can do whatever you want with your copy—such as resell it. The public lending right compensates authors for public lending of their work—though an exception is granted for not-for-profit and educational uses, which do not require the copyright holder’s consent. The remaining rights, called other rights, include permitting or refusing public performance of the work, and making derivative works such as plays or movies.

Copyright law is complex, arcane, and varies from one country to another. The British Parliament adopted the first copyright act in 1710; the U.S. Congress followed suit in 1790. Although copyright is national law, most countries today have signed the Berne Convention of 1886, which lays down a basic international framework. According to this Convention, copyright subsists without formality, which means that (unlike patents), it’s not dependent on registering a work with the authorities or depositing a copy in a national library. It applies regardless of whether the document bears the international copyright symbol ©. You automatically have copyright over works you create (unless they are made for hire). Some countries—including the United States—maintain a copyright registry even though they have signed the Berne Convention, which makes it easier for a copyright holder to take legal action against infringers. Nevertheless, copyright still subsists even if the work has not been registered.

The Berne Convention decrees that it is always acceptable to make limited quotations from protected works, with acknowledgement and provided it is done fairly. The United States has a particular copyright principle called “fair use” which allows material to be copied by individuals for research purposes. The U.K. equivalent, which has been adopted by many countries whose laws were inherited from Britain in colonial times, is called “fair dealing” and is slightly more restrictive than fair use.

Making copies of works under copyright for distribution or resale is prohibited. That is the main economic point of the copyright system. The Berne Convention also recognizes certain moral rights. Unlike economic rights, these cannot be assigned or transferred to others; they remain with the author forever. They give authors the right to the acknowledgment of their authorship and to the integrity of their work—which means that they can object to a derogatory treatment of it.

THE PUBLIC DOMAIN

Works not subject to copyright are said to be in the “public domain,” which comprises the cultural and intellectual heritage of humanity that anyone may use or exploit. Often, works produced by the government are automatically placed in the public domain, or else it sets out generous rules for their use by not-for-profit organizations. This applies only in the country of origin: thus, works produced by the U.S. government are in the public domain in the United States but are subject to U.S. copyright rules in other countries.

Copyright does not last forever, but eventually expires. When that happens, the work passes into the public domain, free of all copyright restrictions. No permission is needed to use it in any way, incorporate any of its material into other works, or make any derivative works. You can copy it, sell it, excerpt it—or digitize it and put it on the web. The author’s moral rights still hold, however, so you must provide due attribution.

Today the internationally agreed Berne Convention sets out a minimum copyright term of life plus 50 years, that is, until 50 years after the author dies. This presents a practical difficulty, for it is often hard to find out when the author died. One way is to consult the authors’ association in the appropriate country, which maintains links to databases maintained by authors’ associations around the world.

The duration of copyright has an interesting and controversial history. Many countries specify a longer term than the minimum, and this changes over the years. The original British 1710 act provided a term of 14 years, renewable once if the author was alive; it also decreed that all works already published by 1710 would get a single term of 21 further years. The 1790 U.S. law followed suit, with a 14-year once-renewable term. Again, if an author did not renew copyright, the work automatically passed into the public domain.

In 1831, the U.S. Congress extended the initial period of copyright from 14 to 28 years, and in 1909 it extended the renewal term from 14 to 28 years, giving a maximum term of 56 years. From 1962 onward, it enacted a continual series of copyright extensions, some one or two years, others 19 or 20 years. In 1998, Congress passed the Sonny Bono Copyright Term Extension Act, which extended the term of existing and future copyrights by 20 years.⁴⁶

The motivation behind these moves comes from large, powerful corporations who seek protection for a miniscule number of cultural icons; opponents call them the “Mickey Mouse” copyright extensions. Many parts of the world (notably the United Kingdom) have followed suit by extending their copyright term to life plus 70 years—and in some countries (e.g., Italy), the extension was retroactive so that books belonging to the public domain were suddenly removed from it.

The upshot is that copyright protection ends at different times depending on when the work was created. It also begins at different times. In the United States, older works are protected for 95 years from the date of first publication. Through the 1998 Copyright Extension Act, newer ones are protected from the “moment of their fixation in a tangible medium of expression” until 70 years after the author’s death. Works made for hire—that is, ones belonging to corporations—are protected for 95 years after publication or 120 years after creation, whichever comes first.

The original copyright term was one-time renewable, if the copyright holder wished to do so. In fact, few did. Focusing again on the United States, in 1973 more than 85 percent of copyright holders failed to renew their copyright, which meant that, at the time, the average term of copyright was just 32 years. Today there is no renewal requirement for works created before 1978: copyright is automatically given for a period of 95 years—tripling the average duration of protection.

No copyrights will expire in the twenty-year period between 1998 and 2018. To put this into perspective, one million patents will pass into the public domain during the same period. The effect of this extension is dramatic. Of the 10,000 books published in 1930, only a handful (less than 2 percent) are still in print. If the recent copyright extensions had not occurred, all 10,000 of these books would by now be in the public domain, their copyright having expired in 1958 (after 28 years) if it was not renewed, or 1986 (after a further 28 years) if it was renewed. Unfortunately, that is not the case. If you want to digitize one of these books and make it available on the Internet, you will have to seek the permission of the copyright holder. No doubt 98 percent of them would be perfectly happy to give you permission. Indeed, many of these books are already in the public domain, having expired in 1958. You could find out which, for there is a registry of copyright renewal (though it is not available online) that lists the books that were renewed in 1959. However, for the remaining works, you would have to contact the copyright holders, for which there is no such registry. There will be a record from 1930, and again from 1959, of who registered and extended the copyright. But you’d have to track these people down.

The problem is a large one. It has been estimated that of the 32 million unique titles that are to be found in America’s collective libraries, only 6 percent are still in print, 20 percent have now passed into the public domain, and the remainder—almost 75 percent—are orphan works: out of print but still under copyright. As Lawrence Lessig put it in his book Free Culture (from which much of the preceding information was taken),

RELINQUISHING COPYRIGHT

As we have explained, the Berne Convention makes copyright apply without formality, without registration. Anything you write, every creative act that’s “fixated in a tangible means of expression”—be it a book, an e-mail, or a grocery list—is automatically protected by copyright until 50 years after you die (according to the Berne Convention’s minimum restrictions) or, today in the United States, 70 years after you die (assuming you did not write your grocery list as a work made for hire). People can quote from it under the principle of fair use, but they cannot otherwise use your work until such time as copyright expires, unless you reassign the copyright.

If you wish to relinquish this protection, you must take active steps to do so. In fact, it’s quite difficult. To facilitate this, a nonprofit organization called the Creative Commons has developed licenses that people can attach to the content they create. Each license is expressed in three ways: a legal version, a human-readable description, and a machine-readable tag. Content is marked with the CC mark, which does not mean that copyright is waived but that freedoms are given to others to use the material in ways that would not be permissible under copyright.

These freedoms all go beyond traditional fair use, but their precise nature depends on your choice of license. One license permits any use so long as attribution is given. Another permits only noncommercial use. A third permits any use within developing nations. Or any educational use. Or any use except for the creation of derivative works. Or any use so long as the same freedom is given to other users. Most important, according to the Creative Commons, is that these licenses express what people can do with your work in a way they can understand and rely upon without having to hire a lawyer. The idea is to help reconstruct a viable public domain.

The term “copyleft” is sometimes used to describe a license on a derivative work that imposes the same terms as imposed by the license on the original work. The GNU Free Documentation License is a copyleft license that is a counterpart to the GNU General Public License for software. It was originally designed for manuals and documentation that often accompany GNU-licensed software. The largest project that uses it is Wikipedia, the online encyclopedia mentioned in Chapter 2 (page 65). Its main stipulation is that all copies and derivative works must be made available under the very same license.

COPYRIGHT ON THE WEB

The way that computers work has led people to question whether the notion of a copy is the appropriate foundation for copyright law in the digital age. Legitimate copies of digital information are made so routinely that the act of copying has lost its applicability in regulating and controlling use on behalf of copyright owners. Computers make many internal copies whenever they are used to access information: the fact that a copy has been made says little about the intention behind the behavior. In the digital world, copying is so bound up with the way computers work that controlling it provides unexpectedly broad powers, far beyond those intended by copyright law.

As a practical example, what steps would you need to take if you wanted to digitize some documents and make them available publicly or within your institution? First, you determine whether the work to be digitized is in the public domain or attempts to faithfully reproduce a work in the public domain. If the answer to either question is yes, you may digitize it without securing anyone’s permission. Of course, the result of your own digitization labor will not be protected by copyright either, unless you produce something more than a faithful reproduction of the original. If material has been donated to your institution and the donor is the copyright owner, you can certainly go ahead, provided the donor gave your institution the right to digitize. Even without a written agreement, it may reasonably be assumed that the donor implicitly granted the right to take advantage of new media, provided the work continues to be used for the purpose for which it was donated. You do need to ensure, of course, that the donor is the original copyright owner and has not transferred copyright. You cannot, for example, assume permission to digitize letters written by others.

If you want to digitize documents and the considerations just noted do not apply, you should consider whether you can go ahead under the concept of fair use. This is a difficult judgment to make. You need to reflect on how things look from the copyright owner’s point of view, and address their concerns. Institutional policies about who can access the material, backed up by practices that restrict access appropriately, can help. Finally, if you conclude that fair use does not apply, you will have to obtain permission to digitize the work or acquire access by licensing it.

People who build and distribute digital collections must pay serious attention to the question of copyright. Such projects must be undertaken with a full understanding of ownership rights, and full recognition that permissions are essential to convert materials that are not in the public domain. Because of the potential for legal liability, prudent collection-builders will consider seeking professional advice. A full account of the legal situation is far beyond the scope of this book, but the Notes and Sources section at the end contains some pointers to sources of further practical information about copyright, including information on how to interpret fair use and the issues involved when negotiating copyright permission or licensing.

Looking at the situation from an ethical rather than a legal point of view helps crystallize the underlying issues. It is unethical to steal: deriving profit by distributing a book on which someone else has rightful claim to copyright is wrong. It is unethical to deprive someone of the fruit of their labor: giving away electronic copies of a book on which someone else has rightful claim to copyright is wrong. It is unethical to pass someone else’s work off as your own: making any collection without due acknowledgement is wrong. It is unethical to willfully misrepresent someone else’s point of view: modifying documents before including them in the collection is wrong even if authorship is acknowledged.

WEB SEARCHING AND ARCHIVING

What are the implications of copyright for web searching and archiving? These activities are in a state of rapid transition. It is impractical, and also inappropriate, for legal regulation to try to keep up with a technology in transition. If any legislation is needed, it should be designed to minimize harm to interests affected by technological change while enabling and encouraging effective lines of development. Legislators are adopting a wait-and-see policy, while leading innovators bend over backward to ensure that what they do is reasonable and accords with the spirit—if not necessarily the letter—of copyright law.

As you know, search engines are among the most widely used Internet services. We learned in Chapter 3 (page 71) that websites can safeguard against indiscriminate crawling by using the robot exclusion protocol to prevent their sites from being downloaded and indexed. This protocol is entirely voluntary, though popular search engines certainly comply. But the onus of responsibility has shifted. Previously, to use someone else’s information legitimately, one had to seek permission from the copyright holder. Now—reflecting the spirit of the web from the very beginning—the convention is to assume permission unless the provider has set up a blocking mechanism. This is a key development with wide ramifications. And some websites threaten dire consequences for computers that violate the robot exclusion protocol, such as denial of service attacks that will effectively disable the violating computer. This is law enforcement on the wild web frontier.

Other copyright issues are raised by projects such as the Internet Archive (mentioned in Chapter 2, page 51) that are storing the entire World Wide Web in order to supply documents that are no longer available and maintain a “copy of record” that will form the raw material for historical studies. Creating such an archive raises many interesting issues involving privacy and copyright, issues that are not easily resolved.

Copyright is a complex and slippery business. Providing a pointer to someone else’s document is one thing, whereas in law, serving up a copy of a document is quite a different matter. There have even been lawsuits about whether it is legal to link to a web document, on the basis that you could misappropriate advertising revenue that rightfully belongs to someone else by attaching a link to their work with your own advertising. The web is pushing at the frontiers of society’s norms for dealing with the protection and distribution of intellectual property.

The legal system is gradually coming to grips with the issues raised by web search. In 2004, a writer filed a lawsuit claiming that Google infringed copyright by archiving a posting of his and by providing excerpts from his website in their search results. In a 2006 decision, a U.S. district court judge likened search engines to Internet service providers in that they store and transmit data to users without human intervention, and ruled that these activities do not constitute infringement because “the necessary element of volition is missing.” Earlier that year, a federal judge in Nevada concluded that Google’s cached versions of web pages do not infringe copyright. However, at the same time, a Los Angeles federal judge opined that its image search feature, in displaying thumbnail versions of images found on an adult photo site, does violate U.S. copyright law.

Those who run public Internet information services tell fascinating tales of people’s differing expectations of what it is reasonable for their services to do. Search companies receive calls from irate users who have discovered that some of their documents are indexed which they think shouldn’t be. Sometimes users believe their pages couldn’t possibly have been captured legitimately because there are no links to them—whereas, in fact, at one time a link existed that they have overlooked. You might put confidential documents into a folder that is open to the web, perhaps only momentarily while you change its access permissions, only to have them grabbed by a search engine and published for the entire world to find. Fortunately, major search engines make provision for removing cached pages following a request by the page author.

Search technology makes information readily available that, though public in principle, was previously impossible to find in practice. Usenet is a huge corpus of Internet discussion groups on a wide range of topics dating from the 1980s—well before anyone would have thought Internet searching possible. When a major search engine took over the archives, it received pleas from contributors to retract indiscreet postings they had made in their youth, because, being easily available for anyone to find, they were now causing their middle-aged authors embarrassment.

THE WIPO TREATY

A treaty adopted by the World Intellectual Property Organization (WIPO) in 1996 addresses the copyright issues raised by digital technology and networks in the modern information era. It decrees that computer programs should be protected as literary works and that the arrangement and selection of material in databases is protected. It provides authors of works with more control over their rental and distribution than the Berne Convention does. It introduces an important but controversial requirement that countries must provide effective legal measures against the circumvention of technical protection measures (digital rights management schemes, mentioned in Chapter 2, page 56) and against the removal of electronic rights management information, that is, data identifying the author and other details of the work.

Many countries have not yet implemented the WIPO Treaty into their laws. One of the first pieces of national legislation to do so was the U.S. Digital Millennium Copyright Act (DMCA), one of whose consequences is that it may be unlawful to publish information that exposes the weaknesses of technical protection measures. The European Council has approved the treaty on behalf of the European Community and has issued directives that largely cover its subject matter.

THE CONSEQUENCES OF COMMERCIALIZATION

This lively commercial interest pushes in two directions. On the one hand, websites strive to gain good positions in search engine rankings in order to further their commercial visibility. On the other, search engines strive to outdo one another in the quality of service they offer in order to attract and retain customers. The resulting tension explains the web wars of Chapter 5. There is huge commercial value in being listed early in the results of search engines. After all, there’s no need to waste money on advertising if you are already at the top of the class! It means life or death to many small online businesses.

Countless poignant real-life stories describe casualties of the war between search engines and spammers. You start a small specialist Internet business. Quickly realizing the importance of having high rankings on major search engines, you subscribe to a commercial service that promises to boost them. Little do you know that it utilizes dubious techniques for website promotion of the kind described in Chapter 5. In fact, such considerations are quite outside your experience, knowledge, and interests—your expertise is in your business, not in how the web works. The crash comes suddenly, with no warning whatsoever. Overnight, your ranking drops from top of the search results for relevant queries to the millionth position. Only later do you figure out the problem: the search engine has decided that your promotional efforts amount to spam.

The web is a capricious and erratic place. Meanwhile, users (as we saw earlier) place blind trust in their search results, as though they represented objective reality. They hardly notice the occasional seismic shifts in the world beneath their feet. They feel solidly in touch with their information, blissfully unaware of the instability of the mechanisms that underlie search. For them, the dragons are omniscient.

THE VALUE OF DIVERSITY

You might recall from Chapter 2 that France’s national librarian warned of the risk of “crushing American domination in the definition of how future generations conceive the world” and threatened to build a European version (page 54). Shortly afterward, Jacques Chirac, President of the French Republic, initiated a mega-project: a Franco-German joint venture to create European search engine technology, funded from both public coffers and private enterprise. Quaero, Latin for I search, is a massive effort explicitly aimed at challenging U.S. supremacy in web search. The project involves many research institutes in France and Germany, as well as major businesses. Project leaders understand that they must be creative to succeed in the ultra-competitive market of online search. Quaero will target its services to various market areas. One is general web search; another will be designed for businesses that work with digital content; and a third version is aimed at the mobile phone market.

European leaders believe it is politically unwise to leave the fabulously profitable and influential business of online search to foreign entrepreneurs. In addition to economic concerns, they worry about the cultural value of services that organize and access the information on the web, services that are used to preserve and transmit cultural heritage. The Franco-German project aims to augment perceived American cultural hegemony with a European perspective. But wait, there’s more. Japan has decided to create its own national search engine, through a partnership of major Japanese technical and media companies: NEC, Fujitsu, NTT, Matsushita, and Nippon public television (NHJ).

Large cooperative enterprises, organized on a national or regional scale with a component of public leadership and funding, are a far cry from the lone young geniuses working for love rather than money, who created the search engines we have today and grew into talented entrepreneurs whose dragons are breathing fire at the advertising legends of Madison Avenue. Will the new approach lead to better search? If the object is really to forestall cultural hegemony, will search engines be nationalized and funded from taxes? Does jingoism have a role to play in the international web world? These are interesting questions, and it will be fascinating to watch the results play out—from the safety of the sidelines.

However, we believe these efforts are addressing the wrong problem. The real issue is not one of cultural dominance in a nationalistic sense, but rather cultural dominance through a universal, centralized service, or a small set of services that all operate in essentially the same way. The activity of searching should recognize the multifarious nature of the web, whose unique characteristic is the collective production of an immense panoply of visions, the coexistence of countless communities. What users need is not a competition between a few grand national perspectives, but inbuilt support for the very idea of different perspectives. While it is understandable for them to want a piece of the action, the Europeans and Japanese have declared war on the wrong thing.

In order to truly represent the immense richness of the treasure, preserve it, and provide a basis for its continued development and renewal, we need an equally rich variety of methods to access it, methods that are based on different technologies and administered by different regimes—some distributed and localized, others centralized. Not only is content socially determined; so is the research process itself. Although we talk of research as though it were a single concept, it covers a grand scope of activities that share only a broad family resemblance. We research different kinds of objects, from apples to archangels, from biology to biography, from cattle prods to cathodes. And different objects imply different kinds of search. When we seek an evening’s entertainment, we are not doing the same thing as when we seek precedents for a homicide case. Different activities imply different criteria and involve different elements for success.

Search is a voyage of understanding. In the real world, early European explorers set the scene for what was to become a huge decrease in our planet’s ecological diversity. Today, in the web world, we are concerned for the survival of diversity in our global knowledge base.

PERSONALIZATION AND PROFILING

Sometimes we search for a particular object, such as an article we read long ago or the name of the heroine of Gone with the Wind. More often, what we seek is imprecise; the question is not well formulated, and our goal is unclear even to ourselves. We are looking more for inspiration than for an answer. Wittgenstein, the twentieth-century philosopher we met in Chapter 1, thought that search is more about formulating the right question than finding the answer to it. Once you know the question, you also have the response. When we embark on a quest online, we are looking more for education than for a fact. Too often, what we get is a factoid—a brief item that is factual but trivial. The trouble is, this affects the way we think and what we tend to value.

History and legend are replete with quests that are ostensibly a search for a physical treasure (the Golden Fleece, the Holy Grail, a missing wife) but end up transforming the hero and his friends from callow youths to demigods (Jason and the Argonauts, the medieval knights Parsifal, Galahad, and Bors, D’Artagnan and his trusty musketeers). Are today’s search tools up to supporting epic quests? If only he’d had a search engine, Jason might have ended up as he began, an unknown political refugee from Thessaly.

Everyone who works in the search business knows that to respond properly to a question, it is necessary to take the user, and his or her context, into account. Keyword input is not enough. Even a full statement of the question would not be enough (even if computers could understand it). In order to resolve the inevitable ambiguities of a query, it is essential to know the user’s perspective, where she is coming from, why she is asking the question. We need the ability to represent and refine the user’s profile.

From a search engine’s point of view, personalization has two distinct aspects. First, it is necessary to know the user in order to answer his questions in a way that is both pertinent and helpful. Second, knowing the user allows more precisely targeted advertising that maximizes clickthrough rate and hence profit. Google’s AdWords system, for example, is an advanced artificial intelligence application that applies contextual analysis to connect user keywords with sponsored links in the most effective possible way. MSN demonstrated a demographic targeting tool for its contextual advertising service at the 2005 Search Engine Strategies Conference in Chicago. Perhaps search could be even more effective if the same degree of attention were paid to exploiting context in generating the search results themselves, in order to satisfy the user’s real information need.

User profiles are based on two kinds of information: the user’s “clickstream” and his geographical location. Search engines can analyze each user’s actions in order to get behind individual queries and understand more of the user’s thoughts and interests. It is difficult to encapsulate a request into a few keywords for a query, but by taking history into account, search engines might be able to fulfill your desires more accurately than you can express them. Localization provides a powerful contextual clue: you can be given a response that is personalized to your particular whereabouts—even, with GPS, when you are browsing from a mobile phone. This maximizes advertising revenue by sharply focusing commercial information to geographically relevant customers. It increases clickthrough by not overloading users with irrelevant advertisements. It saves sponsors from wasting their money on pointless advertising that may even aggravate potential customers. It provides an attractive alternative to broadcast media like TV advertising.

Personalization is a major opportunity for search companies. Personal service that disappeared when we abandoned village shops for centralized supermarkets is making a comeback on the web. Leading online marketers like Amazon already provide impressive personal service. A user’s location and clickstream gives deep insight into what she is doing and thinking. The privacy dilemma emerges in a different guise. The tradeoff between personalization and privacy will be one of the most challenging questions to arise in years to come.

Web search is by no means the only field where new technology raises serious privacy concerns. Consider the vast store of call information collected by telecommunication companies, information that is even more sensitive. In principle, search technology could retrieve information from automatic transcriptions created by state-of-the-art speech recognition systems. Policies can change and evolve in different ways, and users should be keenly aware of the risks associated with the use of any information service—particularly ones that are funded from advertising revenue.