Patterns of Citing Korean DOI Journals According to CrossRef’s Cited-by Linking and a Local Journal Citation Database
Patterns of Citing Korean DOI Journals According to CrossRef’s Cited-by Linking and a Local Journal Citation Database
Journal of Information Science Theory and Practice. 2013. Jun, 1(2): 58-68
Copyright ©2013, Korea Institute of Science and Technology Information
All JISTaP content is Open Access, meaning it is accessible online to everyone, without fee and authors’ permission. All JISTaP content is published and distributed under the terms of the Creative Commons Attribution License ( Under this license, authors reserve the copyright for their content; however, they permit anyone to unrestrictedly use, distribute, and reproduce the content in any medium as far as the original authors and source are cited. For any reuse, redistribution, or reproduction of a work, users must clarify the license terms under which the work was produced.
  • Received : February 25, 2013
  • Accepted : June 12, 2013
  • Published : June 30, 2013
Export by style
Cited by
About the Authors
Tae-Sul Seo
Information Service Center, Korea Institute of Science and Technology Information, Daejeon, Korea
Eun-Gyeong Jung
Information Service Center, Korea Institute of Science and Technology Information, Daejeon, Korea
Hwanmin Kim
Information Service Center, Korea Institute of Science and Technology Information, Daejeon, Korea

Citing literature is a very important activity for scholars in writing articles. Many publishers and libraries build citation databases and provide citation reports on scholarly journals. Cited-by linking is a service representing what an article cites and how many times it cites a specific article within a journal database. Recently, information services based on DOIs (Digital Object Identifiers) have been increasing in number. CrossRef, a non-profit organization for the DOI registration agency, maintains the DOI system and provides the cited-by linking service. Recently, the number of Korean journals adopting DOI is also rapidly increasing. The Korea Institute of Science and Technology Information (KISTI) supports Korean learned societies in DOI related activities in collaboration with CrossRef. This study analyzes cited patterns of Korean DOI journal articles using CrossRef’s cited-by linking data and a Korean journal citation database. This analysis has been performed in terms of publication country and the language of journals citing Korean journal articles. The results show that DOI, SCI(E) (Science Citation Index (Expanded)), and English journals are more likely to be cited internationally.
Many publishers and libraries build citation databases and provide citation analysis reports on scholarly journals. However, citation analysis depends on the size of citation databases determined by such variables as number of journals, preferential selection of citable journals, differential values to citations, and time duration. Citations also depend on wider accessibility and time of availability of the articles/journals to the researcher (Bharathi, 2011). Therefore, articles published through unknown journals are rarely cited.
Liu (1997) has reviewed literatures and categorized author citation practices into the following four major factors:
(1) Physical accessibility : If one is not aware of an article or cannot obtain it, it will not be read.
(2) Cognitive accessibility : If one faces difficulties in understanding an article, or cannot read the language in which it was published, identification of its content, quality, and significance will be unreliable.
(3) Perceived quality : If an article is considered to be of poor quality, it may be less likely to be cited.
(4) Perceived significance : If an article is not scientifically significant, it may be less likely to be cited, even though it is physically and cognitively accessible.
Meanwhile, the citation performance of Korean science in terms of the percentage of world share has been stagnant over twenty years while the Korean share of world publication output in the Science Citation Index has jumped from 40th in 1987 to 12th in 2007 (Kim, 2010). One of the reasons might be that most Korean journals are unknown to foreign researchers. Therefore, the Korean government has adopted policies to enhance Korean citation performance. One solution has been to increase the visibility of Korean science journals. However, many Korean journals are hardly known to foreign researchers and even to Koreans. These journals face difficulties in being cited except for citations by authors who not only are acquainted with Korean journals but also can understand articles written in Korean. Another possibility is that foreign authors can find Korean journal articles in the references of articles published in journals which are published in their own countries or are indexed on journal databases they use.
According to Liu’s categorization, most Korean journals have weak points in physical and cognitive accessibilities; Korean journal articles are rarely read by foreign researchers because most of them are not included in famous journal indexes or databases and are published in Korean. This is so even when the quality or significance of the articles is high level.
CrossRef’s DOI (Digital Object Identifier) system may provide Korean journals with an opportunity to increase citation rates through a reference linking service (Seo & Choi, 2011; Kim et al, 2012). CrossRef’s cited-by linking service enables an author to know who is citing his or her article, to follow references, and to cite effectively by linking references of journal articles (Pentz, 2001). The number of DOI journals has been continuously increasing since November 1999 when CrossRef was established and is now even higher than those of Web of Science journals and SCOPUS journals. Therefore, Korean journals can have more opportunities to be cited by DOI journals worldwide. DOI became an ISO standard in 2012 (ISO, 2012).
KISTI, performing the function of a national scientific and technical information center, has been building Korean scientific and technical journal databases. As part of the internationalization of Korean journals KISTI has been participating in CrossRef since 2007 as a sponsoring member, which executes DOI works for Korean journal publishers or learned societies. DOIs of 76,472 journal articles in 191 journals published by 156 learned societies have been deposited by KISTI as of March 2012. In addition, KISTI has joined the CrossRef cited-by linking service, enabling authors to get a list of cited articles since 2009, and has applied it to scholarly information services at KISTI such as KoreaScience, which includes the landing (or response) pages of Korean DOI journal papers.
The number of Korean DOI journals, 191, is about 30% of the total of Korean scientific and technical journals registered with the National Research Foundation of Korea (NRF). The amount can be considered to be enough to estimate the effect of DOI business on Korean journals. Now it might be desirable to prepare materials for future business by analyzing how Korean DOI journals are cited.
The purpose of this study is to show the patterns of citing those Korean journal articles whose references are deposited in CrossRef using CrossRef’s cited-by linking and a local citation database. It conducts the examinations of 1) how many articles cite Korean DOI journal articles in view of language written and country published; 2) comparison between CrossRef citedby linking and the local citation database; and 3) comparison between SCI(E) and non-SCI(E).
Citation analysis, a widely used method in bibliometrics, is the examination of the frequency and patterns of citations in scholarly literatures. Many researchers have conducted research for finding discernible patterns in citation such as preference of information type, information life cycle, collaborations among researchers, and demands for current information in academic fields or by researchers (Choi et al., 2011).
With regard to citing behavior, Bornmann and Daniel (2006) have reviewed studies on the citing behavior of scientists covering research published from the early 1960s up to mid-2005. They showed that the probability of being cited depends on non-scientific reasons such as time-dependent factors, fielddependent factors, journal-dependent factors, articledependent factors, author/reader-dependent factors, availability of publication, and technical problems. Yue and Wilson (2004) developed an integrated conceptual model of journal evaluation from varying perspectives of citation analysis. They took into account in the model four external factors such as journal characteristics, journal accessibility, journal visibility, and journal internationality, together with journal citation impact. The model also includes non-scientific factors such as publication language of journals (Van Leeuwen et al., 2001) and author geographic location (Lancaster et al., 1990), which act as barriers for Korean journals in being cited internationally. However, as both studies predate DOI, neither takes it into account in journal-dependent or journal accessibility factors.
There are some studies on citation patterns of Korean journals; several articles have conducted citation language analysis on Korean journals (Cho, 2010; Choi, 1996; Choi et al., 2011; Kim et al., 2011), while others include citation country analysis on Korean journals (Cho & Cho, 2005; Lancaster et al., 1990; Kim, 2002; Kim, 2003). However, all of the studies have conducted the citation patterns of Korean authors other than cited patterns of Korean journal articles. There are few studies which have conducted the patterns of citing Korean journal articles.
CrossRef provides a reference linking service called ‘cited-by linking.’ 1 Reference linking is the general term for establishing links from one information object to another. Other systems, including Web of Science, SCOPUS, Google Scholar, and PubMed Central, have also been developed for reference linking services to provide citation information.
For enabling reference linking, each article has a unique identifier and URL (Uniform Resource Locator) (Lee & Lee, 2010). The publisher supplies metadata about each article to be stored in the databases (a reference database and a location database) as shown in Figure 1 . Readers access the databases through the interactions shown in the figure. For each article, the reference database contains metadata corresponding to the information in a citation. Clients who wish to find the content associated with a reference send a query to the reference database. This database returns a list of identifiers for articles that match the query. A client sends an identifier to the location database, which returns one or more URLs. The client selects the URL to retrieve the object. This is known as “resolution” of the identifier. Although they fit within the parameters shown in Figure 1 , most systems differ in detail. The NASA Astrophysics Data Service derives
PPT Slide
Lager Image
Reference linking framework (Caplan & Arms, 1999)
references algorithmically, bypassing the reference database lookup. PubMed and the Web of Science combine the citation and location databases. PubMed’s LinkOut experiment permits users to provide URLs in addition to those provided by publishers.
CrossRef uses the Handle System, which is capable of multiple resolutions 2 of DOI (Caplan & Arms, 1999). Publishers who wish to use the cited-by linking service of CrossRef are asked to deposit DOIs for references so that CrossRef can create reference linking. The process of cited-by linking for CrossRef members is as below:
  • 1. Publisher A registers DOI 10.1234/X for article X with the following metadata: Journal= "Good Science", author="John Smith", volume="21", first page="100" year="2007”
  • 2. Publisher B deposits metadata for article Y and assigns it DOI 10.5678/Y. The deposited references for article Y include the following:
PPT Slide
Lager Image
  • 3. The CrossRef system establishes a cited-by relationship between article X and Y
  • 4. Publisher A later sends a query asking who 'cites' article X and is given the DOI for article Y and its metadata.
  • (source :
1For more information on cited-by linking, visit
2When there are several URLs to different copies of the article, the system is faced with selective resolution: the client may wish to select a specific version based on variations of content, different licensing arrangements, or network performance.
Data sets from CrossRef and KISTI were used in this study rather than international citation databases such as Web of Science and SCOPUS.
First of all, the citation information from CrossRef has been regularly collected and processed according to the process as shown in Figure 2 . CrossRef provides XML data containing cited-by relationships among DOIs as requested through a designated URL composed of a user ID, a password, and a DOI. Then, the XML data is parsed to create cited-by linking data which are to be stored in the analysis database. In order to conduct analysis related to SCI(E), language, and publication country, more data on journals, such as whether they are SCI(E) journal articles or not, what languages they are written in, and what countries they are published in, are also added to the analysis database.
PPT Slide
Lager Image
Data processing flow for cited-by linking data
KISTI operates the Korea Science Citation Index (KSCI) and Korea Journal Citation Reports (KJCR) based on the Korea Science Citation Database (KSCD), which covers over seven million references from 376,700 journal articles published in 746 scientific and technical journals. KSCI is a portal site that provides searching and browsing of Korean scientific and technical
Data coverage of KSCD
PPT Slide
Lager Image
Data coverage of KSCD
journal articles together with a variety of information on articles such as bibliography, subject category, and citation relationship, while KJCR reports various indicators such as impact factor, immediacy index, ZIF, self-citing rate, and self-cited rates of journals in the form of tables and charts by which trends can be shown. Figure 3 shows the relationship of KSCD with KSCI and KJCR.
For comparison with KSCD, the scope of journals to be analyzed in this study was determined by the fact that the cited-by linking data of the journals are available from CrossRef in the same time the journals are included in KSCD. Consequently, a total of 53,147 articles from 68 journals published between 2002 and 2010 were selected and analyzed. 18 journals are indexed in Web of Science (8 journals) or SCOPUS (16 journals). 25 journals are published in English while others are in Korean.
For the analysis of publication countries, the number
PPT Slide
Lager Image
Relationship of KSCD with KSCI and KJCR
of articles with the same publication country is counted. In this case, the publication country of an article means the country of the publisher who publishes the article. The analysis of publication language is also conducted through the same fashion of the analysis of publication country. The citation rates, used in comparisons between CrossRef and KJCR and between SCI(E) and non-SCI(E), are calculated based on KSCD including the references of the selected journal articles.
- 5.1. Country and language
The most interesting pattern of citing Korean DOI journal articles must be how many foreign authors cite
PPT Slide
Lager Image
Distribution ratios of publication countries of journal articles citing Korean DOI journal articles
them. Publication country and publication language of journal articles citing Korean DOI journal articles can be indicators for that.
The total citation number reaches to 32,011 based on CrossRef’s cited-by linking database. 74% of the total citation number was cited by articles of SCI(E) and SCOPUS journals. Figure 4 shows the distributions of publication countries of journal articles citing Korean DOI journal articles selected for this study. It shows that 44% (14,159) of the total citations came from Korean journal articles while the rest, 56% (17,852), came from foreign journal articles. In other words, Korean journal articles were cited by foreign journal articles about 12% more than Korean journal articles were (Kim et al., 2012). Among the foreign citation number, 46.4% are from the United Kingdom (15.3%), United States (13.2%), Netherlands (9.3%), and Germany (8.6%), which have large international journal publishers.
The results show that quite a large number of foreign journal articles cite Korean journal articles even though some of the articles are written in Korean. Table 2 shows the top 20 foreign journals citing Korean DOI journal articles. The top journal is ChemInform published by Wiley & Sons, which cites over a thousand Korean DOI journal articles for nine years from 2002 to 2010. The results also show that six famous publishers publish the top 20 journals: Elsevier (10), Wiley (5), Royal Society of Chemistry (2), IOP (1), Taylor & Francis (1), and AIP (1).
There may be several motives for citing Korean journal articles. First of all, most citations of Korean DOI journal articles by foreign journal articles are probably contributed by SCI(E) and SCOPUS journals.
Top 20 foreign journals citing Korean journal articles (2002 - 2010)
PPT Slide
Lager Image
Top 20 foreign journals citing Korean journal articles (2002 - 2010)
The remainder of citations might be from authors who are acquainted with Korean journals. Another possibility is that authors can refer to Korean journal articles in the reference lists of articles published in the journals which are published in their own countries or indexed on journal databases they use. However, more investigation is required to clarify the motives for citing Korean journal articles.
Figure 5 shows the distributions of publication languages of journal articles citing Korean DOI journal articles selected for this study. It shows that approximately 70% of citations are from articles written in English whereas about 30% are from articles written in Korean. This shows that authors publishing their articles in international journals written in English cite Korean DOI journal articles more than authors who are publishing in domestic journals written in Korean do.
PPT Slide
Lager Image
Distribution ratios of publication languages of journal articles citing Korean DOI journal articles
- 5.2. Comparison with KJCR
One of the important aspects of citing Korean DOI journal articles may be the advantage of DOI journal articles compared with other journal articles without DOI. To show this, the citation pattern based on CrossRef cited-by linking is compared with that based on KJCR, even though some journals in KJCR may have DOIs.
Table 3 is a comparison between citations counted based on CrossRef’s cited-by linking and KJCR. It shows that the total citation number from CrossRef’s cited-by linking is lower than that from KJCR. It means that non-DOI journals have more impact than DOI journals in the scope of KJCR. However, as shown in Table 3 , CrossRef’s cited-by linking has an advantage over KJCR in the number of articles cited one or more times (other than citation number). This means that Korean DOI journal articles have more chances to be cited in international journal articles than journal articles without DOIs have, regardless of the citation rate. Therefore, the impact of DOI journals may also increase if the number of DOI journals increases in KJCR.
Comparison between citation analysis results from CrossRef and KJCR
PPT Slide
Lager Image
Comparison between citation analysis results from CrossRef and KJCR
- 5.3. Korean vs. English
As Liu (1997) pointed out, publication language is an important factor when authors cite articles. The languages used in Korean journal articles are Korean and English. In this study which language is dominant and how the impact of journals with Korean and English as the publication language differs are examined.
When considering languages of articles deposited to CrossRef, articles written in Korean were cited more frequently by Korean journal articles (85%) than foreign journal articles (15%) were. Journal articles written in English, however, received more citations from foreign journal articles (80%) than Korean journal articles (20%) did, even those published by Korean learned societies, as shown in Figure 6 . This shows that publication language is an important element for increasing international citation.
- 5.4. SCI(E) vs. non-SCI(E)
Finally, an interesting aspect of patterns of citing Korean journal articles is whether the journal being cited is indexed in a prestigious international database. In this study the SCI(E) journal group and non-SCI(E) journal group in Korean journal articles deposited to CrossRef are compared.
As shown in Table 4 , the first group, SCI(E) journals, has a higher citation rate than the second group, non-SCI(E) journals. A closer look reveals that the first
PPT Slide
Lager Image
Citation patterns with respect to publication language
group has a noticeably lower self-citation rate on average and a far higher citation rate from foreign journal articles and foreign SCI(E) journal articles than the second group has. The result shows that Korean SCI(E) journals are perceived as higher quality over non-SCI(E) journals, as generally expected.
One more interesting detail is that the citation rate from foreign journals of Korean SCI(E) journals is higher than that from SCI(E) journals, while the citation rate from foreign journals of non-SCI(E) journals is smaller than that from SCI(E) journals.
- 5.5. Summary
In this study how Korean journals are cited is investigated quantitatively in view of publication language, publication country, DOI, and SCI(E). First of all, publication
Comparison of citation rates between SCI(E) vs. non-SCI(E)
PPT Slide
Lager Image
Comparison of citation rates between SCI(E) vs. non-SCI(E)
language and country analyses show that articles published by Korean journals are cited worldwide, even though the total citation number of 32,011 is insufficiently high for an international level considering that Korean authors can publish their articles in journals published in foreign countries or in other languages. Regarding publication language, it is clear that articles published in English, which is widely accepted as an international language, have an advantage in receiving citations from international journal articles.
Comparison between Crossref’s cited-by linking data and KJCR shows that KJCR has more citations than DOI journals do. This does not mean that Korean journal articles are cited more by articles in domestic journals than articles in DOI journals are, because citation numbers of DOI journal articles do not include other citations from articles in journals without DOIs, whereas those of KJCR includes all citations. On the contrary, the important conclusion derived in the comparison is that articles published in DOI journals have an advantage over articles published in journals without DOIs in their opportunities for citation.
Finally, through comparison between SCI(E) and non-SCI(E), it is found that SCI(E) journals have more impact than non-SCI(E) journals do as generally known. Going into detail, eight SCI(E) journals among the Korean DOI journals have 4.5 times the impact of 60 non-SCI(E) journals.
In this study we obtained generic patterns of citing Korean DOI journals using CrossRef’s cited-by linking and KJCR. The patterns are related with citation number and rate, and especially countries and languages of journals citing Korean DOI journals.
We can conclude the following:
  • 1) Korean DOI journal articles are being cited by a considerable number of foreign journal articles.
  • 2) Korean DOI journal articles are more likely to be cited in international journal articles than in journal articles without DOIs, regardless of the citation rate.
  • 3) It is an effective strategy to change a journal’s language to English in order to increase international citation of Korean journals.
Consequently, the purpose of this study has been achieved as discussed and concluded above. The results obtained in this study can be useful information for conducting DOI business and future research. But more detailed analysis is needed to obtain more concrete evidence for the effects of DOIs in international citation.
Additionally, we can conclude that the data set based on CrossRef ’s cited-by linking is useful in citation analysis. So it is desirable that more studies be conducted using CrossRef’s data as research material incorporating Web of Science or SCOPUS data.
This study may also have limitations due to its narrow coverage of Korean DOI journals. Therefore, a further study with a much broader scope is desirable in the future in order to obtain more reasonable results and also to clarify the motives for citing Korean journal articles.
This research was performed as a part of the 2013 KISTI project, “S&T Content Construction and Service.”
Bharathi D. Gnana. (2011) Methodology for the evaluation of scientific journals: aggregated citations of cited articles. Scientometrics 86 563 - 574    DOI : 10.1007/s11192-010-0291-8
Bornmann L. , Daniel H.-D. (2006) What do citation counts measure? A review of studies on citing behavior. Journal of Documentation 64 (1) 45 - 80
Caplan P. , Arms W. (1999) Reference linking for journal articles. D-Lib Magazine (online) Retrieved from:
Cho H.-Y. (2010) A comparative study on researchers' language preference for citing documents in different subject fields. Journal of the Korean BIBLIA Society for Library and Information Science 21 (1) 212 - 221
Cho H.-Y. , Cho H.-S. (2005) A comparative study on the citing behavior of scholars in four major engineering fields. Journal of Information Management 36 (2) 1 - 24    DOI : 10.1633/JIM.2005.36.2.001
Choi S.-H. , Kim B.-K. , Kang M.-Y. , You B.-J. , Lee J. , Park J.-W. (2011) A study of citing patterns of Korean scientists on Korean journals. Journal of the Korean Society for information Management 28 (2) 97 - 115    DOI : 10.3743/KOSIM.2011.28.2.097
Choi S.-Ki. (1996) Comparative study on citation analysis in the field of mechanical engineering in Korea and Japan. Journal of the Korean Society for Information Management 13 (2) 121 - 142
International Organization for Standardization (2012) ISO 26324:2012 Information and documentation -- Digital object identifier system.
Kim B.-K. , Choi S.-H. , Kim S.-Y. , Seo T.-S. (2012) Citation analysis of Korean journal articles based on Digital Object Identifiers. KISTI Proceedings of the 8th International Conference on WIS Seoul 395 - 399
Kim B.-K. , Kang M.-Y. , Choi S.-H. , Kim S.-Y. , You B.-J. , Shin J.-D. (2011) Citing behavior of Korean scientists on foreign journals in KSCD. Journal of the Korean Society for information Management 28 (2) 117 - 133    DOI : 10.3743/KOSIM.2011.28.2.117
Kim H.-R. (2003) A study on the citation analysis of the information resources on science & technology. Journal of the Korean Society for Information Management 20 (4) 1 - 21
Kim K. (2002) The co-occurrence phenomenon of both Korean and Non-Korean literatures within the Korean References: An analysis on the citation motivations and references by social scientists. Journal of KLISS 36 (4) 21 - 47
Kim M.-J. (2010) Visibility of Korean science journals: an analysis between citation measures among international composition of editorial board and foreign authorship. Scientometrics 84 505 - 522    DOI : 10.1007/s11192-010-0168-x
Lancaster F. W. , Kim S.-Y. , Diluvio C. (1990) Does place of publication influence citation behavior? Scientometrics 19 (3-4) 239 - 244
Lee Y.-S. , Lee S.-G. (2010) The Reference Identifier Matching System for Developing Reference Linking Service. Journal of Information Management 41 (3) 191 - 209    DOI : 10.1633/JIM.2010.41.3.191
Liu Z. (1997) Citation theory in the framework of international flow of information: new evidence with translation analysis. Journal of the American society for information science 48 (1) 80 - 87    DOI : 10.1002/(SICI)1097-4571(199701)48:1<80::AID-ASI9>3.0.CO;2-Y
Pentz E. (2001) Brief communication: reference liking with CrossRef. Interlending & Document Supply 29 (1) 20 - 23    DOI : 10.1108/02641610110694720
Seo T.-S. , Choi H.-Y. (2011) Global dissemination of domestic scholarly journals using DOI and open access. Journal of Information Management 42 (4) 1 - 15    DOI : 10.1633/JIM.2011.42.4.001
Van Leeuwen T. N. , Moed H. F. , Tijssen R. J. W. , Visser M. S. , Van Raan A. F. J. (2001) Language biases in the coverage of the science citation index and its consequences for international comparisons of national research performance. Scientometrics 51 (1) 335 - 346    DOI : 10.1023/A:1010549719484
Yue W. , Wilson C. S. (2004) Measuring the citation impact of research journals in clinical neurology: a structural equation modelling analysis. Scientometrics 60 (3) 317 - 332    DOI : 10.1023/B:SCIE.0000034377.93437.18