Kate Ehrlich[1] and Debra Cash[2]
[1]Lotus Development Corporation, 1 Rogers St, Cambridge
MA 02142
kate_ehrlich@crd.lotus.com
[2] New Century Enterprises, 77 Marlboro Street, Belmont, MA 02178
debracash@igc.apc.org
Abstract
Lotus Notes, a broad-based software environment tailored for access to distributed databases and electronic communications, has been called the "foundation of a virtual library." Its flexibility and ability to incorporate numerous databases across domains make it a contemporary "microcosm" of how electronic resources can be a tool for organizational learning. Lotus Notes is a particularly rich environment for finding and archiving existing information, no matter how ephemeral or "unofficial."
However, offering an interoperable information infrastructure and delivery system does not, by itself, render information useful. Based on the results of a recent ethnographic study of a customer support organization, we learned that customer support personnel help each other find and make use of information through face-to-face collaborations. Collaborative processes such as joint problem-solving, re-registering problems, data interpretation and sifting information are part of the larger enterprise that turns stored information into knowledge. Moreover, we identified a key person in the group whose breadth of knowledge and troubleshooting skills were so well developed that he became an indispensable "information mediator" for the group.
Keywords: Lotus Notes, collaborative work, information retrieval, digital libraries, customer support, information mediator.
1. Introduction
The future of digital libraries promises a world in which vast amounts of information of all types - text, images, photos, sound, video, animation - will be available to people wherever they are. The vision of being able to access news, magazines, professional articles, bulletin boards, catalogs, personal mail and the like from a personal computer at home, in the office, or even on the beach is compelling. We applaud many of the efforts towards that vision. But we also caution researchers and engineers to think carefully about the social implications.
We recently conducted an in-depth study of the work practices developed by a group of customer support professionals that enable them to find and use information stored in Lotus Notes databases. This study illuminates a number of issues we believe developers of digital libraries will soon need to address.
1.1. Corporate memory as part of corporate libraries
Researchers and pundits speak almost interchangeably of "digital libraries," "virtual libraries," "electronic infrastructures" and "information highways." The recent NSF RFP referred to the "ingredients of a digital library" quite concretely -- and in some sense, narrowly -- as any "information sources accessed via the Internet." While our terminology evolves, it is crucial that it not blur important distinctions. A library is not the same thing as an infrastructure. A collection or repository of information is not the same thing as the delivery system through which that information is made available. In fact, the content of a library may not be identical to what we mean when we refer to a library [1]. Moreover, and most important for our discussion here, an interoperable information infrastructure and delivery system does not, by itself, render information useful [2].
Corporate libraries -- that is both print archives and electronic databases -- that support people in the course of doing their jobs are a very specialized form of the libraries we know; a set of institutions that include public libraries, academic libraries, medical libraries and the like. Corporate libraries are characterized by the fact that they are the locus of collection and dissemination for a highly-focused set of materials. These items have been pre-selected for what someone, or some group, sees as their business potential. A corporate library may be well-segmented into, for instance, engineering and marketing categories. It might distinguish public information available to any employee or approved vendor from private information available only with security clearances. But most corporate libraries are generated, whether systematically or on an ad-hoc basis, as a shared corporate resource [1]. Such libraries are expected to transmit at least some aspects of the organization's collective history and knowledge. They function as an important element of corporate memory [3].
Lotus Notes is probably the most popular commercial software environment for corporate information-sharing today. Designed to support workgroups in a business environment, nearly 3,200 companies and more than 750,000 people are using Notes to improve business processes, including customer service, sales and account management, and product development. The excitement it has generated in the market speaks to the fact that it is perceived to fill a gap that exists within business organizations. Organizations needed to begin to consolidate their electronic communications and wanted to be able to share data and documents across functional and geographically dispersed divisions. Notes functions simultaneously as a broad-based applications delivery system and a communication device.
As an exemplar, or partial prototype, of the new paradigm of electronic information access, Notes incorporates many of the features that most observers feel that the "virtual library of the future" must have. These include text handling; information integration or juxtaposition; simultaneous user access to geographically distributed databases; multisite access to identical data, including, for instance, access to Notes over laptops as well as workstations; integrated electronic mail; and a standard and standardizable user interface. Lotus Notes is a rich environment for finding and archiving existing information, no matter how ephemeral or "unofficial." It is particularly strong in its interactive tracking applications and reference applications that can be accessed and updated on-line.
As a software environment, Lotus Notes does not distinguish between "working documents" and "reporting documents" (although of course data can be extrapolated from these documents into reports). This makes the "corporate memory" associated with Notes significantly more robust than electronically-based repositories created as formal archives [4]. Notes was also designed to handle a number of multimedia formats that are not yet in wide use, but can be expected to gain in popularity in coming years. It is this flexibility that has led some to proclaim it as "the foundation of a virtual library" [5].
While software such as Lotus Notes begins to offer an environment for sharing information, research is also underway to help identify the processes involved in finding that information in the first place. Tapestry, an ambitious prototype system developed by a research group at Xerox PARC, embodies the notion of collaborative filtering. Users can annotate documents which become available to their colleagues as a basis for selecting among different sources of information [6]. Instead of having to browse through all the documents in a particular newsgroup, for instance, one can limit a search to just those documents that have been selected as valuable by a respected colleague. In this emerging model, users participate not merely in a "free marketplace of ideas" but a "free marketplace of evaluation" [7].
2. The Study
In this paper, we examine the work practices that emerge when people in the same workgroup find, share and make sense of information as part of their job. Our on-site study, conducted between January and March 1994, was designed in the light of a number of other research projects informed by ethnographic methods [8].
2.1 Methodology
We chose to observe and interview members of a customer support organization that answers customer calls over the phone. This group relies on Lotus Notes as an integral part of its operations. Notes is well-suited for customer support applications, both in the features it offers and its track record of adoption in actual work settings [9].
We based the design of this study on the understanding that work practice is shaped by at least three distinct dimensions: the physical space; the technology and other workplace artifacts available to the workers; and the formal and informal relationships existing within the organization (Figure 1). These dimensions evolve separately, and in many cases, such as the design of the physical plant, may not be planned deliberately at all. Yet each offers resources and constraints that structure the ways people are able to work [10]. While the design and functionality of the Notes software environment was a central focus of our inquiry, it existed in the familiar context of telephones, cubicles, and collegial relationships. We assumed that the impact of some of these dimensions would be well understood by the customer support analysts themselves; others would be transparent to them.
In the course of observing and interviewing skilled customer support personnel, our premise was that they were experts in their own practice. However, instead of analyzing or ranking individual tasks and skill sets, we focused on the patterns of use and interaction between people. In this way, we could identify a number of the interdependencies among team members. We expected that while some of these interactions would be face-to-face, others would be mediated by various types of communication technology. Bolstered by other studies [11] that indicate that most work is far less routinized than most managers or procedure manuals -- and therefore, most software developers -- comprehend, we decided to focus our research on the unstructured, dynamic aspects of work. Ad hoc work activity is usually considered outside the range of efforts such as "business process re-engineering" and workflow analysis.
Figure 1: Dimensions that structure work practice
2.2. Data Gathering
The data was collected in two distinct phases. During the first phase, we observed individual customer support "analysts". We listened into customer calls on headsets and occasionally asked questions during breaks between calls. During the second phase, we conducted semi-structured interviews with analysts we had observed, analysts we had not observed, and with a number of managers and related personnel. During these interviews we encouraged them to share anecdotal information, and we paid special attention to the themes and common issues that emerged from their stories. We also attended staff meetings, and had access to copies of various work-related documents.
2.3. The Notes databases
The group relies on two databases for most of their information needs.
"Call tracking" database. This database was used primarily to keep track of all customer calls and their disposition. At the time of the study, this database was approximately 250 Mb in size and contained nearly 15,000 individual documents (a "document" corresponds to a single message of any length made up of any combination of data types. For instance an email message is a "document"). Ninety percent of these documents had been entered within the past 4 months, indicating a rapidly growing database as well as one that is very large.
When a customer calls, the call is routed to the first available customer support analyst. This person would check the ID number of caller and begins answering the call. Once answered, a customer call can remain open if the support person wants to do further research on the problem; it can be escalated to someone in the organization with known expertise in the pertinent area; or it can be closed if the customer's question has been answered to the satisfaction of both the support person and the customer.
"Knowledge" database. This database was used as a repository for technical information including summaries of error messages, release notes, and updates to information that had been generated since the last set of documentation and technical papers were published. It also contained company information including press releases, product fact sheets, resource catalogs, and supplemental information. Some of this information was also available to customers. At the time of the study, this database was approximately 55 Mb in size and contained nearly 3300 individual documents.
The call tracking and knowledge databases are the primary sources of on-line information for the customer support personnel. These databases are shared by everyone in the group: anyone can add information and anyone in the group can read it although all drafts are reviewed for accuracy before they are widely distributed. Each support person also has additional information sources. These include printed documentation, on-line database of recent bug fixes, privately generated notebooks and lists, and perhaps specs of new features.
Both databases had been indexed and a full text search was often the most expedient way to find information. Support personnel also use their knowledge of the databases and especially knowledge of the other people in the organization, to browse a particular "View" of the database, most commonly to look for similar cases.
3. Results: Work Practices
We first report the results of our study by focusing on the work practices that have evolved in this group. In analyzing this set of interactions, we uncovered a number of myths about information access that may inform the design and development of digital libraries.
Myth: Customers understand their own problems, they just don't know how to "fix" them. Customers often believe that they understand their problem, and may even have ideas about solutions, and just want help with the solution. To the contrary, we found that a significant amount of time during calls is spent refocusing the conversation on the description of concrete symptoms and identification of relevant details. That is, the problem as understood by the customer was frequently not the real problem. Moreover, customers were often unaware which symptoms were critical and which extraneous. As one analyst reported
"I try to get the customer as vanilla as possible. If I get working that way then we can slowly start putting things back together."
"Re-registering" the problem in this way requires a sophisticated set of heuristics, and each analyst developed his or her own repertoire. Analysts used different strategies to get at the "real" problem, including asking factual questions about system configurations, whether certain procedures had been tried, walking through certain situations, and trying to reproduce error messages or system crashes. The analysts would also elicit more information by making hypotheses (e.g. "my hunch is that...") which would yield still more information from the customer. Re-registering a problem is a fundamental feature of problem-solving on the job, and is common in other work practice as well [12].
Myth: Customers would use on-line information if it was available. In our target sites, analysts conservatively estimated that at least 30 percent of their calls were from customers who had the information available to them (either on-line or in printed documentation) to solve their own problems.
In some cases, customers simply find it easier to call an analyst than review the on-line or published documentation especially when there is a large amount of information available. One of the analysts emphasized this point very graphically for us by pulling out four large books one by one and setting them on top of each other to indicate the amount of reading a customer would have to do to solve a given problem. Although the solution may only require a small part of each of those books, there was no way to predict where that information might be found.
We have highlighted these two assertions -- that customers understand their own problems, and that they would use on-line information if it was available -- as myths. Many digital library advocates presume that the primary barrier to easy and efficient information access occurs at the level of network and interface design. Once these concerns have been addressed, people should have no further difficulties in using digital libraries to find answers to their questions.
In contrast, our research indicates that at least in specialized domains, finding information depends on framing a question correctly. Asking the wrong question, or even asking an incomplete question, may lead people to information that has no relevance to the problem at hand. The sought-after information may indeed be somewhere in the database, but under current database design schemes, unless the searcher knows what he or she is looking for, it may be impossible to retrieve that information. This is particularly true of systems that have been designed in such a way to direct users to subcategories of information, and systems that are too large to browse easily.
Myth: Customer support analysts could work from home. This myth pervades many discussions of the shape of future digital information systems. Since the main business of a support analyst is to talk to customers on the phone then why not work from home? However, from the analysts' point of view, working from home would not only create social isolation but would impair the quality of their work. The analysts were very sensitive to each other's whereabouts. One of the analysts deliberately placed her chair so that she could see when a colleague got off the phone. Others used "gopher-net," peeking over the cubicle walls, even standing on chairs to see who was free. If the colleague was busy, an analyst could signal to a colleague that she needed some advice by standing in the office doorway. Even the phone was considered inferior to face-to-face discussions primarily because of the difficulty in reaching the other analyst on the first call.
What these professionals were aware of, and what many researchers miss, is that while an individual can query the system, making use of that information is essentially a collaborative activity. In many cases, people looking for information in work settings rely on the skills and insights of their colleagues to work efficiently and effectively.
In answering our question about what would be lost if he worked from home, one analyst replied that he would lose the ability to brainstorm
"it takes more than one head to walk through [a] problem ... especially in trouble-shooting. "
Even escalating problems is often initiated informally before a formal follow-through:
"I mean, you know, a customer calls up and you're taking calls from your house ... You don't necessarily know what to do. Or you kind of have an inkling and you want to talk to the area expert. You don't necessarily want to escalate the call; you just want to talk to him a little bit about it. What do you do if you're at home? Whereas here, you just walk over."
4. Results: Intra Group Collaboration
Our study provided us with a fascinating insight into the role and importance not only of collaboration but of synchronous collaboration. These face-to-face meetings between people in the group served an important social, organizational and cognitive function. At the social level it allowed people to form personal bonds of mutual trust and respect. This was not only a very supportive group - of each other and of the customers - but one in which there was a high level of shared knowledge, wisdom and culture.
The Lotus Notes technology played an important role in providing people with the basis for the shared knowledge. But it was in the face-to-face meetings, the group meetings and the shared "stories" that the real business of turning shared information into shared knowledge and understanding happened. In this section we examine some of these collaborations in more detail. But first, we introduce the person who played a key role in this process.
4.1. Information Mediator
A formal escalation process is an important resource for any technical organization, and in the group we studied, each analyst was a member of at least one area of expertise within 36 designated areas of specialization. Calls that could not be answered by the analyst could be formally escalated to the group chartered with addressing problems in that particular area.
"Much like there are specialists in an emergency room, always waiting on call, we have an emergency procedure that will deal with these issues. So we would escalate to the appropriate ... expert team."
Informally, analysts kept their paper print-out of expert team members near at hand and often referred to it. They turned to area experts on an ad hoc basis, without formal escalation, even during customer calls. These domain experts are a crucial part of an organization that puts information to use through collaborative problem-solving.
Expertise, however, is usually limited to particular domains. We found that in this organization one expert was relied on primarily because of the breadth of his knowledge and powers of interpretation. Although he had been hired as an analyst with responsibilities to act as the technical expert in a certain area, he quickly emerged as an all-purpose "Information Mediator". This person is much more than just an expert. His contribution is so critical that his absence was considered more serious than a database being temporarily unavailable. His skills included the ability to collect and synthesize information across a wide variety of data types - other analysts, stored information, documentation, specs, customers, personal experimentation - and his ability to sift out the relevant information from irrelevant information.
"My information is really what everyone else's information is, except that I try to step back a little bit."
Personally extremely confident, he is an excellent diagnostician and good at coming up with practical tests of hypotheses and possible solutions. He views trouble-shooting as one of the skills that sets him apart from the others:
"The number one skill that I have, is troubleshooting skills. The number two is organization skills. I can juggle a whole bunch of stuff at once. It doesn't frazzle me. Number three is probably communication skills. That includes verbal and written.... But number one by far is troubleshooting...
It is important to distinguish this kind of "information mediator" from more formally constrained relationships. Technical experts, while important and often turned to for advice, may not have his breadth of knowledge. Librarians often play a formal role in helping to formulate actual queries and conduct searches and acquire information from external sources. Typically however, they don't have the domain expertise necessary to apply the information they help collect. A more similar person may be the "fix-it person" or computer guru in a group who fills in the gaps left by inadequate training and poorly engineered or complex computer systems and helps keep the group technically savvy [13].
4.2. Turning information into knowledge
A central finding of our study is that it is dangerous to blur the concepts of "information" and "knowledge." Information must be put into context in order to be useful, and those contexts must be explicated, experimented with, and confirmed. In the context of a customer support organization, as in many other environments, information retrieval must be recognized as part of a larger enterprise of building up knowledge to "solve problems" not merely "answer questions". Knowledge, moreover, is grounded in collective as well as individual experience.
The professionals we studied were highly skilled and highly motivated. The collaboration they engaged in was not a "work-around" but a central and valuable activity on the job. While collaborative activities took place in an unstructured manner, they can be mapped to the entire cycle of their work process.
Guiding the search criteria. When the analysts engaged in problem-solving, especially with the "information mediator", they would be gently guided into learning how to ask the right questions in order to get the right answers. This was especially important in eliciting symptoms of the customer's problem. For instance, the "information mediator" generally opened a discussion by asking the analyst about a set of key symptoms. If the analyst didn't have this information, he would either call the customer or resurrect the information from the call record.
"Usually they'll [the analyst] come in and they'll look baffled and they're pleading for help and they say I have this customer. And then this strange thing starts happening. I'll ask them a couple of questions about exactly what the problem is, trying to get to the most important details for whatever the problem is. And if they know them, they'll tell me. And if they don't know them, then they go and get them and come back to me and that's fine."
Mapping criteria onto documents. Analysts (and customers) often have a hard time knowing when they have found the right document. Discussion with other analysts helps them make sense of the information they already have found. As the "information mediator" noted:
"Sometimes they [the analysts] just can't find the information. Sometimes they may have found the information but don't know that they have found the information, so they just bypass it. [For example] They read a tech note and for them it doesn't click because it seems very different. They may not have asked the right questions from their customers to get the information that is outlined there in the tech note. And maybe the tech note isn't specific enough or clear enough to say, you know, to completely relay the type of problem it is to them."
... "Sometimes I think that they feel that it is simpler to come to me to get the answer, boom ... rather than searching through knowledge base, which is a good size database ... I won't say huge ... but a good size database and maybe finding 15 or 20 documents that sort of fit what they're talking about and then they have to read through each one of them trying to figure out exactly ... which one out of all of them fits their problem.
One of the roles the "information mediator" plays is in helping people make the right connections by translating and applying information to the problem at hand.
Identifying resources. In group meetings as well as face-to-face discussions, analysts would freely refer to people inside and outside the organization, to information in the "knowledge" database or other databases as sources of solutions to a particular problem. This is one of the ways in which people help each other sift through information and can be thought of as a form of collaborative indexing.
Separating the relevant from the extraneous. It is significant that the "information mediator" perceived his ability to separate the relevant from the extraneous as one of the key skills that set him apart from the others. As he says:
"Good troubleshooting skills are the basis of analyzing and resolving a problem. You have to be able to take the problem apart and break it into different pieces of information and find out what pieces are important to solving the problem, and what pieces are extraneous. Then you focus on those important pieces of information and you narrow down the possibilities. And you usually do that by trying to take out some of the factors."
Establishing relevance relies on, not merely meeting explicit criteria, but on a much less tangible understanding of the domain, including a familiarity with the history of similar problems, the ability to make analogies and to rule out various possibilities. Analysts also told us that they felt their direct experience of the system was critical to problem-solving. The "information mediator", like all the other analysts, spent a good amount of time duplicating their customer's problems. This served both for system diagnosis and as a way to gain first-hand experience of a complex technology.
Joint problem-solving. One of the most common reasons for face-to-face discussions was to address problems together and test hypotheses. Many customers had either unique symptoms or unique solutions. Analysts had to use their powers of deduction and reasoning to arrive at an accurate diagnosis and solution. When making these judgments, the analysts often found it helpful to talk to a colleague. The discussion often took the form of playing out the problem , e.g. "what happened when ...", testing alternative hypotheses, "did you try ...", "what if ...".
Confirming hypotheses. Even when an analyst was confident he had a solution, he might discuss the problem with a colleague. In part this behavior served the psychological function of validating the person's judgment. Because the culture of this group prized accuracy, getting confirmation was also a way to get a second opinion before committing a solution to a customer.
Putting accurate information back into the system. Within this organization, group meetings are used to share "stories" and build up collective knowledge about policies, procedures, and emerging trends. Only after these interactions take place is information (often in the form of meeting minutes) entered into a computer repository. It is often overlooked that "corporate history" databases are themselves collectively-created artifacts. One of the most important aspects of having face-to-face contact with colleagues in the customer support organization is knowing who posted information and call records into the database. This social contact is crucial for understanding why information might be reliable or unreliable, current or out-of-date, and to offer the possibility of gathering other information that might not have been recorded. Analysts become more deeply tied to the collective knowledge of their colleagues by using such databases, not more detached from human contact.
Although the main goal of the group meetings was to discuss some of the difficult calls that had come up in the previous week, the meeting was also used to share general information which might affect the group. In the course of the meeting, new information might come to light. If it was identified as being of value to the group an individual could be given responsibility for following up on it by adding it to the database.
5. Conclusions
As the "information mediator" said to us:
"Notes is wonderful. I mean .. when I came to this job, I never used Notes until I got here. I was amazed at how quickly I could learn things. Just by looking in databases. All by myself.. I didn't have to go talk to people. I didn't have to ask questions. I didn't have to go to meetings. I could read about meetings in databases ... It was like "Wow!" A kid in a candy store!"
We have no question that the availability of on-line information is an exciting and valuable development. The Lotus Notes call tracking and "knowledge" databases used by customer support analysts were the lifeblood of the organization we studied. If the system was down for any reason, analysts struggled with paper and pen. The analysts considered Notes a significant improvement over paper-based logs and less flexible record-keeping schemes.
However, on-line information is only part of the whole picture associated with the creation and dissemination of corporate knowledge. In the asynchronous world of email and large distributed databases such as the Internet, ad hoc face-to-face communication and unplanned social interactions still have a critical role to play [14, 15].
The role of technologies in these complex environments must be to support the ways people sort through, evaluate and act on the information available to them. As one example, it may be better to develop technologies which let people run through a series of options to rule out certain alternatives since the process of ruling out those alternatives has value in and of itself, especially in workgroups where people have diverse skills, experience, and heuristics, and the organization is committed to organizational learning. The speed and power of our computer-based systems can be harnessed to enable individuals and teams to try out as many options as possible, as efficiently and quickly as possible, and to share what they learn with their colleagues [16]
Similarly, human mediators and synthesizers, who now exist as "unsung heroes" in many organizations and corporations, should not be moved aside in favor of "better" information management schemes. Rather, their role can be made more visible to the organization, better supported by appropriate tools and technologies, and rewarded as people who add value across the organization.
We believe that no matter how well designed and engineered digital libraries become, no matter how dense our cross-indexing schemes, how "intelligent" our filters and "Knowbots," it is unlikely that these technological achievements will ever be able to completely take over the creation of knowledge. Human beings are far better suited to the tasks associated with separating relevant from irrelevant information within a document as well as across documents, of integrating different types of information and sources of information, and developing strategies to experiment with and test hypotheses.
If Lotus Notes indeed can be considered the forerunner of tomorrow's digital libraries, our findings point to some exciting times when technologies are used to support rather than supplant the skilled work people bring to the important job of turning information in the digital libraries into real shared working knowledge.
Acknowledgments
We wish to extend special thanks to Tony Adams and his group for their invaluable help to us throughout all phases of the work. We also thank Irene Greif for her critical reading of an earlier version of the paper and her support of the project.
References
[1] Geoffrey Nunberg. "The Place of Books in the Age of Electronic Reproduction," in Representations Special Issue: Future Libraries, Howard Block and Carla Hesse, eds. No. 42, Berkeley, CA Spring 1993, pp. 13-37.
[2] David M. Levy and Debra Cash "The Future of Libraries," forthcoming.
[3] Thomas Davenport and Laurence Prusak. "Blow Up the Corporate Library," Ernst and Young Center for Information Technology and Strategy, Research Note, Boston, 1993.
[4] Lucy M. Berlin, Robin Jeffries, Vicki L. O'Day, Andreas Paepcke, Cathleen Wharton, "Where Do You Put It? Issues in the Design and Use of a Group Memory". In Proceedings of the Conference on Computer Human Interaction (Interchi '93), Amsterdam, 1993, 23-30.
[5] Kristen Liberman and Jane L. Rich, "Lotus Notes Databases: The Foundation of a Virtual Library," Database Vol. 16. No. 3. June 1993 pp.33-46.
[6] David Goldberg, David Nichols, Brian M. Oki, and Douglas Terry "Using Collaborative Filtering to Weave an Information Tapestry," Communications of the ACM, December 1992, Vol. 35 No. 12, p. 61-70.
[7] Gary M.C. Bean, Oberon Software, personal communication, January 20th, 1994.
[8] See particularly Brigitte Jordan "New Research Methods for Looking at Productivity in Knowledge-Intensive Organizations" in Panunak, H. Van Dyke, ed. Industrial Technology Institute, Ann Arbor 1992 and Lucy Suchman "Constituting Shared Workspaces". In D. Middleton and Y. Engestrom, eds., Cognition and Communication at Work, Sage Press, 1992.
[9] Michael Gallivan, Cheng Hian Goh, Lorin M. Hitt and George Wyner, "Incident Tracking at Infocorp: Case Study of a Pilot Notes Implementation" CCS WP #149, Sloan School (MIT) WP #3590-93, July 1993.
[10] John A. Seiler, "Architecture at Work," Harvard Business Review, September-October 1984.
[11] This includes the major study of a Monsanto Chemical plant described in Charles Kukla, Elizabeth Anne Clemens, Robert S. Morse and Debra Cash "Designing Effective Systems: A Tool Approach" in Paul Adler and Terry Winograd, eds., Usability: Turning Technologies into Tools, Oxford University Press: New York, 1992. And of an insurance claims processing organization in Etienne Wenger "Toward A Theory of Cultural Transparency," unpublished doctoral dissertation Department of Information and Computer Science, University of California at Irvine, 1990.
[12] Kukla et al and Wenger, cited above.
[13] William M. Bulkeley, "Study Finds Hidden Costs of Computing" Wall St Journal, Nov 2, 1992, p.B4.
[14] Steve Whittaker, David Frohlich and Owen Daly-Jones, "Informal Workplace Communication: What is it like and how might we support it?" In Proceedings of the Conference on Computer Human Interaction (CHI '94), Boston 1994.
[15] Robert E. Kraut, Carmen Egido and Jolene Galegher "Patterns of Contact and Communication in Scientific Research Collaborations". In Jolene Galegher, Robert E. Kraut and Carmen Egido, Eds., Intellectual Teamwork: Social and Technological Foundations of Cooperative Work, Lawrence Erlbaum 1990, 149-172.
[16] K. Eisenhardt, "Making Fast Strategic Decisions in High Velocity Environments," Academy of Management Journal, 32 (3), 1989.