A Citation and Reference Study[1]
A paper delivered at the Midyear Meeting of the American Society for Information Science, San Diego, CA, May 20-22, 1996. Also available in the conference Proceedings, pp. 299-315.
by
Stephen P. Harter and Hak Joon Kim[2]
School of Library and Information Science
Indiana University, Bloomington, Indiana 47405
ABSTRACT
The journal is fundamental to formal scholarly communication. This research reports highlights and preliminary findings from an empirical study of scholarly electronic journals. The purpose of the research is to assess the impact of electronic journals (ejournals) on scholarly communication, by measuring the extent to which they are being cited in the literature, both print and electronic. The intent is to provide a snapshot of the impact ejournals were having on scholarly communication at a given point in time, roughly the end of 1995. This study provides one measure of that impact, specifically on the formal, as opposed to informal, communication process. The study also examines the forms in which scholars cite ejournals, the accuracy and completeness of citations to ejournals, and practical difficulties faced by scholars and researchers who wish to retrieve ejournals through the networks.
INTRODUCTION
The first scholarly journal, Journal des Scavans, was published as a new medium of communication in 1665, and was soon followed by the Philosophical Transactions of the Royal Society (Osburn, 1984). For more than three centuries the journal has played a pivotal role in the creation and transmission of knowledge by serving as the primary medium of scholarly communication, and has remained essentially unchanged in form and function over its lifetime. Science as we know it is scarcely imaginable without the scholarly journal.
Despite its benefits to science and scholarship, the paper journal system has been subject to much criticism. Deficiencies noted by some authors include perceived problems with the peer review process (that it suppresses new ideas, favors authors from prestigious institutions, and causes undue delays in the publication process), high costs that are escalating faster than the rate of inflation, and lack of selectivity. Spiraling costs and long publication delays are perhaps the most serious of these criticisms.
At the same time that the costs of producing the paper journal have increased sharply, developments in computer and communications technology have accelerated. And of course we now have the dramatic explosion of the World Wide Web. Technology increasingly offers the possibility of using computers and communication networks to create alternative electronic forms of the conventional paper journal. It is possible that these new forms of computer-based communication will transform the scholarly communication system. There is much debate and discussion concerning how the roles of the various participants in the scholarly communication process, including libraries, may be redefined in the process (Meyer, 1993).
Although electronic journals (or ejournals) have been under development since 1976 (Turoff and Hiltz, 1982), ejournals in their non-experimental phase did not begin until the 1990s, with a few exceptions. The first peer-reviewed electronic, full-text ejournal including graphics was Online Journal of Current Clinical Trials (OJCCT) (Keyhani, 1993). In the most recently published list, and one of the few research projects involving ejournals, Hitchcock, Carr, & Hall (1996) identified 115 scholarly, peer-reviewed ejournals in science and technology, and there are many more in the social science and humanities. These constitute the first wave of what are likely to be many more scholarly ejournals to come.
Ann Okerson wrote, "One can fantasize endlessly about electronic 'journals,' but without active authorship and readership there is nothing" (quoted in Collins and Berge (1994)). We would add "use" to authorship and readership. If ejournals are not used, they cannot play an important role in scholarly communication. There has been much discussion and speculation in the literature about perceived advantages and disadvantages of ejournals, and about problems and issues related to their development and use. Such speculation is necessary and important. The present study is aimed at gathering hard data regarding the readership, use, and impact of ejournals. What kinds of ejournals are being published, and how often? What impact are ejournals having on the scholarly communities they are serving? To what extent are scholars and researchers aware of, influenced by, using, and building their own work on research published in ejournals?
The present study assesses the impact of scholarly ejournals by examining the artifacts of scholarly communication--the journal article and its references. In bibliometric studies, citation analysis is a well-known technique that has a long history in studies of scholarly communication (Borgman, 1990). As an artifact of scholarly communication, citations reveal formal communication patterns and scholarly impact. The major advantages of citation analysis are its high reliability and unobtrusiveness. Several standards for citing electronic publications have been promulgated (Li & Crane, 1993; Walker, 1995; Brown, 1995).
This research provides hard citation data concerning the impact of ejournals on the conduct of research and scholarship. At another level, publishers of ejournals can gain some information regarding use by recording the number of subscriptions to an ejournal or the number of times articles are accessed or downloaded from host servers. While these kinds of data provide useful indicators of one type of use, they do not measure the extent to which ejournal articles are playing a role in the scholarly and research process, that is, in the advancement of knowledge. While the meaning of citations can certainly be debated, what is clear is that they reflect an influence or impact of some kind on the author of the citing article.
The purposes of this study are to:
The last stated purpose deserves further comment. Although it was not originally part of our study to investigate demographic data, we found as the research progressed that we were learning information of great interest to us that had little to do with the referencing of ejournals. As part of our original research plan, in order to investigate the references of articles in ejournals it was necessary that each ejournal in the population be accessed and articles and other information about the journal to be downloaded and printed for analysis. A useful byproduct of the study has resulted from a study of these data, which revealed several interesting demographic characteristics and access problems associated with the population of ejournals. Because of space limitations, only a few of these can be discussed here.
SUMMARY OF METHODS
The method followed in the first part of this study is one of defining and drawing appropriate samples of ejournals, obtaining data from printed documentation and ejournal articles, conducting descriptive analyses, and reporting the findings in tables and prose. In the second part of the study, tools published by the Institute for Scientific Information (ISI) are used to conduct cited reference searches and other citation analyses. The following section describes how the ejournal samples were drawn for study.
Our interest in this research was on scholarly electronic journals that publish articles reporting scholarship or research that are refereed or peer-reviewed. Other types of publications--newsletters, zines, and the like, were not subjects of study. We chose to use two published directories that include data on electronic journals: Mecklermedia's Internet World's on Internet 94 (Mecklermedia, 1994) [3], and the Association of Research Libraries' Directory of electronic journals, newsletters and academic discussion lists (Okerson, 1995). Ejournals selected from the ARL directory were taken from the section entitled "Electronic journals, magazines, and zines" and identified there as "peer-reviewed." Ejournals selected from the Mecklermedia directory were taken from the section entitled "Electronic journals and newsletters" and identified there as "refereed." All ejournals meeting these criteria became members of the sample. The two lists were combined and duplicates were removed. There were 134 ejournals in the final list. Four titles were eliminated from the sample for various reasons, and one title was added, for a refined initial sample of 131 ejournals.
Next, recent articles and other documentation (journal description, instructions for potential authors, statement of scope, etc.) from each of the ejournals were downloaded or otherwise obtained and printed for examination. The access and demographic study was conducted on all 131 ejournals in the refined initial sample.
For the reference study, we had two criteria for selecting the journals to be studied from the 131 ejournals in the refined sample. First, the ejournal must have published one or more articles that reported the results of research or scholarship. Second, the ejournal be peer-reviewed or refereed articles, as described in the ejournal documentation. Many ejournals were eliminated because they failed to meet the first criterion. A few were removed because of the absence of any statement in the ejournal documentation indicating that it was peer-reviewed or refereed. This characteristic, as well as all the other demographic information related to the ejournals, was taken directly from the ejournals. There were several instances where the data in the two directories (both providing self-reported survey data) did not agree with the information provided in the ejournals themselves.
To be included in the reference study, an article must, in our judgment, have reported the results of research or scholarship and must have cited at least one reference. Our sample of articles for the reference study was defined as the last four available scholarly, peer-reviewed articles of each of the 77 scholarly ejournals. If one or more issues or articles of the year 1995 were not available, one or more issues from 1994 and, if necessary, 1993, were used.
For the cited work study, the population of seventy-seven ejournals used in the reference study defined the initial sample. However, in order for an ejournal article to be cited, especially in a print publication, the lag time in conventional print publishing must be considered. One must provide some time for articles to be read, to influence a researcher or scholar in some way and thus become part of a study in progress, and eventually to be cited in the published article reporting the results of that study. For this reason, the cited work study was done only on scholarly electronic journals that commenced publication in 1993 or earlier. There were thirty-nine such journals. Table 1 summarizes the characteristics of the three ejournal samples used in the study.
Table 1. Ejournal Sample Sizes for the Three Studies
==========================================================
Purpose of Sample Origin of Sample Members Sample Size
----------------------------------------------------------
Access and ARL directory and 131
Demographic Mecklermedia directory epublications
Study
Reference Study Peer-reviewed or refereed 77 scholarly
scholarly journals in the ejournals
initial sample
Cited Work Study Ejournals in the reference 39 scholarly
study that began ejournals
publication in 1993 or
earlier
=============================================================
DEMOGRAPHIC CHARACTERISTICS OF EJOURNALS
We assigned the 131 ejournals in the sample (consisting of all known ejournals at the time the sample was selected) to broad subject categories that coincided in most cases with university departments or professional schools. Subject categories could not be determined for seven ejournals. Although we were fortunate to have access to some fee-based journals, thanks to temporary passwords supplied by OCLC, Inc., we did not have access to three other fee-based ejournals in the sample. Also, three ejournals could not be accessed at all, either by using the information in the two directories or by using other means, such as Internet search tools. Finally, the language of one ejournal was Polish, and we did not attempt to obtain a translation. Since we required all of our demographic data to come from the ejournals themselves, rather than secondary sources, these seven journals were eliminated from the sample. Save for a very few exceptions, then, the remaining demographic data are for the 124 ejournals to which we were able to gain access.
Table 2 shows the subject categories with highest frequencies. Education, literature, and mathematics were the top three categories, with library and information science a somewhat surprising fourth, with seven ejournals. There were five ejournals under the heading of religious studies. We were also somewhat surprised at some of the subject categories with relatively low frequencies -- for example, physics, biology, engineering, and sociology with only one or two representatives each. Of course, since the publication of the ARL and Mecklermedia directories we used to select our sample, many more ejournals in these subject areas have appeared, coinciding with the explosion of the World Wide Web. Our demographic data represent the state of affairs as of June, 1995, and can be taken as a benchmark, against which subsequent studies of electronic journals can be compared.
Many new ejournals have evidently arisen from an introspective consideration of the new communication media themselves, and with a concern with how they will affect society and traditional disciplines and communication patterns. Computer science, communication, and the broad subject we called "information technology, media, and society" are examples.
A final way to characterize the subjects of the ejournals in the sample is to classify them into the traditional four broad areas of scholarship and research in the university: sciences, social sciences, humanities, and professional. Table 3 summarizes a breakdown of subjects into these four main categories. While we might have guessed that the sciences and possibly the professions would be most heavily represented, this was not the case. The four categories are essentially equal in size. Whatever the motivation may be for launching an ejournal, it is apparently held by scientists, social scientists, humanist scholars, and professionals alike.
The major purpose of this research was to study the effects of scholarly, peer-reviewed ejournals on formal scholarly and scientific communication, as measured by cited references. For the reference study and cited work study it was necessary to obtain and examine individual issues of the ejournals in the sample to identify which met our selection criteria. These criteria were that the ejournal must have published at least one article reporting the results of research or scholarship must be refereed or peer-reviewed, as stated in the documentation provided by the ejournal (corresponding to what is usually the front-matter in a print journal).
Table 2. Subject Categories with the Highest Number of Ejournals
====================================================================
Subject Number of Ejournals
--------------------------------------------------------------------
Education 13
Literature 12
Mathematics 10
Library and Information Science 7
Computer Science 6
Communication 5
Information Technology, Media, and Society 5
Medicine 5
Religious Studies 5
===================================================================
Table 3. Distribution of Ejournals in the Sample in the
Sciences, Social Sciences, and Humanities
===========================================================
Broad Subject Category Number of Ejournals in Sample
-----------------------------------------------------------
Sciences 28
Social Sciences 34
Humanities 31
Professional 31
Could not determine 7
Total 131
===========================================================
Table 4 reports the results of our analysis of the scholarly nature of the 131 ejournals in the initial sample. Only 77 ejournals were judged to be scholarly and peer-reviewed. Twenty-five ejournals in the initial sample were in our judgment neither scholarly nor peer-reviewed. These were zines and newsletters for the most part. Six other ejournals did not meet our selection criteria for various other reasons.
Table 4. Scholarly, Peer-Reviewed Ejournals in the Sample
==============================================================
Journal is: Number of Journals
--------------------------------------------------------------
Scholarly, Peer-reviewed 77
Scholarly, Not Peer-reviewed 16
Neither Peer-reviewed nor Scholarly 25
Eliminated for Other Reasons* 13
*Not a journal, publishes data only, publishes
reviews only, or could not determine
==============================================================
Scientific and scholarly journals are today being published in many media: CD-ROM, floppy disk, on computer networks such as the Internet, and of course, print. The literature has tended to use the term ejournal -- short for electronic journal -- to refer to networked journals but not other forms of electronic journals (on floppy disk, etc.). We have used the same terminology in this report, though technically it is at best potentially misleading and at worst incorrect.
Most (nearly three quarters) of the ejournals in the sample started as an original publication -- they began their lives in electronic form. Exactly one-fourth of the sample began and continue as print journals. That is, they exist in two parallel forms, print and electronic. And in three known cases -- Modal Analysis , Essays in History , and Journal of Extension , the print version has been replaced by an electronic version. Table 5 summarizes these findings.
Table 5. Media of Publication*
====================================================================
Media of Publication Number of Ejournals Percentage (%)
--------------------------------------------------------------------
Electronic version only 86 69.4
Print and electronic versions 31 25.0
Print version replaced
by electronic version 3 2.4
Not clear whether print
version continues 4 3.2
*Media of publication could not be determined for 7 ejournals.
Other media such as CD-ROM are not considered in this table.
====================================================================
Several models have been developed for the relationship between print journals and their electronic counterpart. (See Table 6.) The Chicago Journal of Theoretical Computer Science and other ejournals will provide individual articles in paper form, though the journals themselves are not published in a paper version. The Journal of Financial and Strategic Decisions calls its ejournal "secondary to" the print journal, by which is meant that the electronic version is the ASCII format and does not accurately represent the print version. Slavic Review publishes its electronic version a few months after the print version has appeared (called a "postprint edition"). Public-Access Computer Systems Review (PACS Review) has published at least two volumes in book form, many months after the appearance of the electronic version. Several journals publish some but not all of the print journal electronically. Finally, both an electronic and a print version of a journal may exist, but with different pricing arrangement.
Table 6. Models for the Relationship between Print Journals and
their Electronic Counterparts
==================================================================
Ejournal Models
------------------------------------------------------------------
Ejournal replaces print journal
Ejournal coexists with print journal
Journal is in electronic form only, but individual
articles can be ordered in paper form
Ejournal is "secondary to" the print journal
Electronic version is published several months after
the print version
Print version is published several months after the
electronic version
The full print version is not available electronically
Both versions exist but with different pricing arrangements
===================================================================
Whether there exists a charge for a given ejournal is not as simple a question as it may first appear. Questions and distinctions not always made for print journals become important for an interpretation of the meaning of cost figures. For example:
Clearly the answers to these questions will be important information needed to interpret the cost, if any, of an ejournal, as well as the meaning of pricing trends. We identified eight different categories of ejournals in the sample, depending on the answers to these questions. Table 7 condenses the eight categories into just two: ejournals in which all issues are free, and ejournals for which some form of fee exists. For nearly nine out of ten ejournals in the sample, all issues are free. And in some of the remaining ejournals, there is no fee for some kinds of access. For example, JAC Online, a journal of composition theory supported by the Association of Teachers of Advanced Composition and the University of South Florida, charges a fee to view the current issue, but there is no charge to examine archival issues. A chi-square test of the hypothesis that there is no relationship between pricing arrangement and whether the ejournal is peer-reviewed could not be rejected (p=.487).
Table 7. Gross View of Pricing Arrangements for Peer-Reviewed Ejournals in the Sample
====================================================================
Pricing Arrangement Number of Ejournals Percentage(%) of Ejournals
--------------------------------------------------------------------
All issues of 67 88.2
electronic version
are free*
Fee for some or 9 11.8
all issues of
electronic journal
*Membership in a society is required for three ejournals.
====================================================================
REFERENCE STUDY
The purpose of the reference study was to analyze the references from a sample of articles in scholarly ejournals, to measure the extent to which authors of ejournal articles are currently citing ejournals and other online sources. Such authors can be assumed to be more knowledgeable about and sympathetic to the electronic media than typical authors. Thus the results of the reference study should provide a kind of upper bound on the current influence of ejournals and other electronic publications on scholarly communication.
Among the 77 scholarly and peer-reviewed ejournals in the sample, two were fee-based ejournals for which we had no subscription[4]. These ejournals were eliminated from the sample. In addition, Radioscientist ON-LINE, published by the New Zealand National Committee for the International Union of Radio Science, included only one article, which had no reference, so it was also eliminated from the sample. Therefore, the final sample size for the reference study was reduced to 74.
To examine their references, we retrieved and printed the last four available scholarly, peer-reviewed articles published by each of the 74 ejournals. Because the ejournals in the sample were distributed in different ways, a variety of means had to be employed to do this, ranging from email, listserv, and ftp commands to using Web browsers. This part of the study was quite illuminating as to the difficulties involved in accessing and using ejournals. Six of the 74 ejournals had published fewer than four articles as of September, 1995, when the data collection took place. In these cases we studied the articles that were available.
Table 8. Samples in the reference study.
==================================================================
Entities Sample Size
------------------------------------------------------------------
Available electronic journals -- 74 ejournals
scholarly and peer-reviewed
Articles -- the most recently published four, 279 articles
or in six cases, fewer, articles appearing in
the ejournals
References -- the first twenty, or sometimes 4317 references
fewer, references appearing in the articles
==================================================================
Table 8 summarizes the data relating to the samples in the reference study. We took the first twenty references in each ejournal article to build our set of references. If an article had fewer than twenty references, we studied all the references in that article. For this portion of the study there were a total of 4,317 references from the most recent 279 articles published in 74 scholarly and peer-reviewed ejournals.
Each of the 4317 references was classified as to its format (book, serial, book chapter, online source, etc.). Online sources were further subclassified (web page, email, ejournal article, etc.). Finally, the accessibility of the references to online sources was tested by actually trying to retrieve the cited item.
Table 9 shows the formats and frequencies of the most frequent reference sources cited by the 279 ejournal articles. Print serial (43.3%) was the format most frequently cited, with books (26.9%) and book chapters and other parts (16.0%) occupying the second and third ranks respectively. This distribution is roughly what is found in reference studies conducted using print journals in various fields, e.g., library and information science (Harter, Nisonger, & Weng, 1993).
References were classified as "online sources" in Table 9 if they were of a variety of electronic subtypes: web pages, electronic personal papers, email messages, ejournal articles, news group postings, listserv postings, and others. Only 83 online sources (1.9% of the total number of references) were cited by the 74 ejournals (279 articles). Among the 74 ejournals, only twelve had one or more reference to an online source and the other 62 ejournals had none at all.
The distribution of online and ejournal references in the twelve ejournals that have at least one reference to an online source is highly skewed. The ejournal Public-Access Computer Systems Review had 34 online references (40.9%) and Electronic Journal of Virtual Culture and Ejournal accounted for a further 25 (30.1%) and 9 (10.8%) respectively from among the total number of 83 online references to online sources. Thus online references are distributed among only a few ejournals; 68 (81.8%) of the total come from these three.
Table 9. Format and Number of Cited References
===================================================================
Formats Number of References Percentage of Total (%)
-------------------------------------------------------------------
Serial 1,871 43.3
Book 1,160 26.9
Book chapter and other parts 689 16.0
Proceedings 140 3.2
Online source 83 1.9
Report 80 1.9
Presented paper 52 1.2
Other* 230 5.3
Subtotal 4,305 99.7
Could not determine 12 0.3
Total 4,317 100.0
* Others include legislation, mimeograph, provision, unpublished raw
data, film, fax, computer file, hearing, offprint, law, treaty,
international covenant, video, committee decision, and many others.
====================================================================
Table 10 shows the distribution of the online references of each four reviewed articles in the top three ejournals with online references. This distribution is also skewed, with most references to online sources appearing in only three articles. Thus the number of online references also depends on the characteristics of individual articles within the same ejournal, although the degree of skewness is not so large as for the overall population of ejournals.
Table 10. Distribution of Online References Among the Twelve
Articles in the Sample for the Ejournals with Most Online
References (PACS-R, EJVC, and Ejournal ).
===================================================================
Number of References to Online Sources Number of Articles
-------------------------------------------------------------------
0 3
1-5 5
6-10 1
11-15 3
===================================================================
Table 11 shows the types and frequencies of references to online sources. "Web Page" was the most frequently cited document type, with "Electronic Personal Paper" ranking second and "Email Messages" and "Ejournals" jointly occupying the third rank. We could not determine the types of twelve of the 83 online references because they contained incomplete bibliographic information and the online sources were not accessible.
One of the important findings of the reference study is that the citation styles of online references are frequently inconsistent, incomplete, and/or are inaccessible -- that is, they do not lead to the wanted online resource, in contrast to the citation styles of print references.
Table 11. Types, Number, and Accessibility of Cited Online Resources
===================================================================
Type of Online Number of References Number and Percentage (%)
Resource of Accessible References
--------------------------------------------------------------------
Web Page 12 10 (83.3)
Electronic Personal Paper 10 7 (70.0)
Email Message 9 2 (22.2)
Ejournal Article 9 6 (66.7)
Newsgroup Posting 7 0 (0.0)
Listserv Posting 5 2 (40.0)
Electronic Directory 5 5 (100.0)
SGML Encoded Document 5 5 (100.0)
Electronic Preprint 3 1 (33.3)
Computer Software 2 2 (100.0)
Electronic Newspaper 2 2 (100.0)
Online Catalog 1 1 (100.0)
Local File 1 0 (0.0)
Type Could Not
be Determine 12 0 (0.0)
Total 83 43 (51.8)
===================================================================
Each of the 83 references to online sources was examined and attempts were made to access the material cited. Obviously one of the potential advantages of citations to online resources is that if the reader can retrieve such resources quickly, the communication process has obviously been enhanced. The third column of Table 11 shows the results of the accessibility tests of the 83 references to online resources. Email messages and newsgroup postings were especially inaccessible, and none of the twelve resources that could not be classified could be accessed. Overall, only about half of the online resources were accessible online.
Table 12. Access Protocols and Accessibility for the 83 Online References
===================================================================
Access Protocol Number of Online Number and Percentage (%)
References of Accessible References
-------------------------------------------------------------------
URL 47 31 (66.0)
Email 13 4 (30.8)
Listserv 9 6 (66.7)
Usenet newsgroup 7 0 (0.0)
Electronic newspaper 2 2 (100.0)
Incomplete access
information provided 5 0 (0.0)
Total 83 43 (51.8)
===================================================================
Table 12 reports accessibility findings in another way, by type of access protocol. Online information resources are accessible in several ways, and the 83 online references we found are analyzed according to these. Over half of the cited references included a Uniform Resource Locator (URL). We tried to access the online resources using a Web browser. However, only two-thirds of these led to the desired resource (see Table 12). Instead, we obtained the browser message "The server does not have a DNS entry" or the infamous server error message "404 URL Not Found". Several types of online references, such as Web pages, electronic personal papers, SGML encoded documents, software programs, and local file references, were mainly cited through URLs, often only with a URL.
Only four of the thirteen references including email addresses led to the desired online source. In seven cases, we received the message "User Unknown" in response to our email message. In two other cases, no error message was received, but there was no response and the online resource could not be obtained.
Nine of the online references were to listserv postings, with six (66.7%) being accessible. Accessible in this case meant that we could, through the given listserv access information such as name and discussion date, we could retrieve the original sources using listserv commands. However, three out of nine listserv references were not accessible. For instance, one listserv reference cited the 1994 file of CYBERIA-L. Although we could access the CYBERIA-L archive, we could not find the original source because the archive of CYBERIA-L did not include archives for 1994.
Seven Newsgroup postings were cited, from two different Newsgroups. We could access the archives of one of them but not the other, obtaining the message, "No such group." The Newsgroup that we could access maintained postings only from 1996. Finally, two newspaper articles that can be accessed through commercial online providers such as Nexis/Lexis were cited, and we were able to access these articles through the same providers.
We labeled five online references as "incomplete access information provided." These were cases in which no information was provided about how to access the source, or the information that was provided was insufficient for us to gain access to the resource.
In summary, except for commercially-provided electronic newspapers (to which not all Internet users will have access), none of the access methods led to perfect retrieval. The overall rate of 51.8% must be considered to be extremely low, considering that the online resources cited were apparently important enough for an author to cite them and yet in just one or two years, nearly half were no longer accessible, given the information provided in the citation.
Recall that the reference study was conducted to fix a kind of upper bound on the current influence of ejournals on scholarly communication. We argued that authors of ejournal articles would probably be more inclined than other authors to cite ejournals and other electronic sources. This still seems like a reasonable assumption to us.
We examined 4317 references in 279 scholarly, peer-reviewed articles appearing in 74 ejournals. Among these, only 83, or 1.9%, were to online sources, and only nine, or 0.2%, were to ejournals. Only twelve of the 74 ejournals in the sample cited one or more online sources. Only eight of the 74 ejournals cited another ejournal even once, and only one ejournal (not surprisingly, it was the ejournal Ejournal) made two or more references to other ejournals. References to online sources are concentrated in a few ejournals, with PACS-R, Electronic Journal on Virtual Culture, and Ejournal accounting for more than 80% of all online references. And among this group, most are concentrated in just a few articles.
It is worth noting the identity of the ejournals in which the great majority of the references to online sources are made. The top three have as their central subjects of interest topics related to electronic communication. Among the twelve ejournals citing at least one online reference, the five ejournals dealing with online topics (PACS-R, Electronic Journal on Virtual Culture (EJVC), Ejournal, Interpersonal Computing and Technology, Journal of Computer-Mediated Communication) account for 71, or 85.5 % of the references to online sources. This suggests that referencing online sources, and ejournals in particular, is a function mostly of academic discipline. One would expect a journal with a name like Electronic Journal on Virtual Culture to cite electronic sources, by virtue of the subject matter treated in the journal. Indeed, as subjects of academic treatment, the ejournals listed above focus on new topical areas that have arisen as a result of the new electronic environment.
On the other hand, the seven ejournals that have as their subject matter traditional academic specializations (sociology, education, library and information science, psychology, cognitive science, and mathematics), cite only twelve online sources between them, and five ejournal articles.
All of this taken together seems clear evidence that ejournals presently play almost no role in scholarly communication, as measured by references cited. Of course there are other ways of measuring impact; we do not claim that counting references and citations should be relied on exclusively. Interviews with scholars, for example, might well lead to a different conclusion -- but we doubt it. Certainly informal communication is being changed in a major way by the Internet -- especially by listservs, email, and web pages -- even if such communications are not being cited in large numbers. But the reference study suggests that a significant effect on the formal communication system is yet to happen. Obviously, this conclusion may well be very different in two or three year's time.
Tables 11 and 12 show that a high percentage (almost 50%) of online resources are not directly accessible from the information provided in references. Direct online accessibility to such resources is obviously potentially very convenient to readers, much more so than for print references. However, if readers cannot gain access to the original sources of the cited material, the references that link the citing and cited works are much less useful than references to print sources. Note that the references we studied appeared in recently published ejournal articles. One might reasonably expect such references to be largely correct and complete and to lead to the full online information source. This was not the case. It is worth contemplating what percentage of these same references will lead to the original information source in two or three years additional time -- say in the spring of 1998 -- it will very likely be much lower than 50%. Clearly, the accessibility of cited online resources is potentially a very serious problem in the conduct of research and scholarship, especially if the percentage of references to such sources increases beyond its current very small size.
CITED WORK STUDY
The reference study reported in the previous section examined the cited references of 74 scholarly, peer-reviewed ejournals. In the cited work study, it is the references of thousands of print (and a very few electronic) journals that are of interest. The Institute of Scientific Information (ISI) publishes three citation indexes -- Science Citation Index (SCI), Social Sciences Citation Index (SSCI), and Arts and Humanities Citation Index (AHCI) -- that allow searches of journal names as cited works. When a cited work search on an ejournal such as PACS Review is conducted on SSCI, one retrieves bibliographic citations to all the articles in journals indexed by SSCI that have cited PACS Review in one or more reference.
Since ISI indexes thousands of scholarly journals, which it represents as the most significant journals in the various fields of scholarship, a cited reference search should retrieve the majority of all citations made in scholarly journals to the cited work. Such a search will not reveal citations to a work made in less significant journals or in books and other materials. In the context of the World Wide Web, a cited work search will not reveal the hypertext links (analogous to citations) to works made in Web pages. However, these limitations notwithstanding, the ISI indexes are widely recognized as providing a valuable measure of the scholarly impact of articles, journals, and authors -- that is, the formal impact. Not the only, or even necessarily the best, measure, but valuable nonetheless.
Eugene Garfield, the founder of the Institute for Scientific Information, addressed the issue of the meaning of citation for individuals in a section of his book entitled "What Do Citation Counts Measure":
Much the same comment applies to the application of citation-based measures to the evaluation of journals. A number of studies have examined the meaning of citation as an evaluation tool for journals (McAllister, Anderson, & Narin, 1980) and research performance (Narin, 1976; Lawani, 1986 ). Bibliometric measures have also been used as methods for the evaluation of departments, universities, and published works.
This portion of the research reports the results of cited work searches of all three ISI citation indexes for the scholarly, peer-reviewed ejournals in the sample that have had a reasonable chance of being cited. Since the time lag in the publication schedules of print journals is substantial, we reasoned that a minimum of two years should be necessary to provide time for an ejournal article to be read, become part of an author's research, and eventually be cited in a print journal. For this reason, we limited the cited work searches to those scholarly, peer-reviewed ejournals that began publishing in 1993 or earlier. There were thirty-nine such ejournals in the sample.
The cited work study is in progress. Results from this portion of the research will be presented at the conference.
Borgman, C. L. (Ed.). (1990). Scholarly communication and bibliometrics. Newbury Park, California: Sage Publications.
Brown, H. (1995). ASCII citation of electronic documents. URL http://library.ccsu.ctstateu.edu/~history/docs/cite.html.
Collins, M. P., & Berge, Z. L. (1994). IPCT Journal: A Case Study of an Electronic Journal on the Internet. Journal of the American Society for Information Science, 45, 771-776.
Garfield, E. E. (1979). Citation indexing: its theory and application in science, technology, and humanities. New York: Wiley.
Harter, S. P., Nisonger, T. E., & Weng, A. (1993). Semantic relationships between cited and citing articles in library and information science journals. Journal of the American Society for Information Science, 44, 543-552.
Hitchcock, S., Carr, L., & Hall, W. (1996). A survey of STM online journals 1990-95: The calm before the storm. January 15, 1996 (Updated February 14, 1996). Available at URL http://journals.ecs.soton.ac.uk/survey/survey.html.
Keyhani, A. (1993). The Online Journal of Current Clinical Trials: An innovation in electronic journal publishing. Database, 16, 14-23.
Lawani, S. M. (1986). Some Bibliometric Correlates of Quality in
Scientific Research. Scientometrics, 9, 13-25.
Li, X., & Crane, N. B. (1993). Electronic style: A guide to citing electronic information. Westport, CT: Meckler.
McAllister, P. R., Anderson, R. C., & Narin, F. (1980). Comparison of peer and citation assessment of the influence of scientific journals. Journal of the American Society for Information Science, 31, 147-152.
Mecklermedia. (1994). Internet World's on Internet 94: An international guide to electronic journals, newsletters, texts, discussion lists, and other resources on the Internet. Westport: Mecklermedia.
Meyer, R. W. (1993). Roles in scholarly communication under an electronic model. [email to Steve Harter], [Online]. Available email: RMEYER@VM1.TUCC.TRINITY.EDU.
Narin, F. (1976). Evaluative bibliometrics: The use of publication and citation analysis in the evaluation of scientific activity. Cherry Hill, New Jersey: Computer Horizons, Inc.
Okerson, A. (Ed.). (1994). Directory of electronic journals, newsletters and academic discussion lists (4th ed.). Washington, D.C.: Association of Research Libraries.
Osburn, C. B. (1984). The place of the journal in the scholarly communications system. Library Resources and Technical Services, 28, 315-324.
Schauder, D. (1994). Electronic publishing of professional articles: Attitudes of academics and implications for the scholarly communication industry. Journal of the American Society for Information Science, 45, 73-100.
Smith, L. C. (1981). Citation analysis. Library Trends, 30, 83-106.
Turoff, M., & Hiltz, S. R. (1982). The electronic journal: A progress report. Journal of the American Society for Information Science, 33, 195-202.
Walker, J. R. (1995). MLA-Style Citations of Electronic Sources. January 1995 (Rev. 4/95). URL http://www.cas.usf.edu/english/walker/mla.html.