Opening academic publishing - Development and application of systematic evaluation criteria

Abstract

This report summarizes the development of a standardized scorecard for evaluating the openness of academic publishers. The assessment was completed in January 2018 as part of the Open Science and Research Initiative of the Finnish Ministry of Education and Culture. The project complements the previous reports published by the Open Science and Research Initiative and the Finnish Ministry of Education and Culture, which have covered (i) the openness of universities and polytechnics, (ii) the overall situation of OA publishing costs in Finland, and (iii) research organization and research funding organizations, including selected European research funders. The project mapped and evaluated the openness of selected major academic publishers: Association for Computing Machinery (ACM), American Chemical Society (ACS), Elsevier, Institute of Electrical and Electronic Engineering (IEEE), Lippincott, Williams & Wilkins (LWW), Sage, Springer Nature, Taylor & Francis, and Wiley-Blackwell. The dimensions of publisher openness were summarized in a scorecard of seven key factors, providing a new tool for systematic and standardized evaluation. We used data from the publisher websites to compare the key factors of openness, and the publishers were given a chance to provide comments on the collected information. As complementary sources, we utilized data from commonly acknowledged, open databases: Directory of OA Journals (DOAJ), Gold OA Journals 2011-2016 (GOAJ2), Scopus (title list +Scimago), and Sherpa / Romeo. The main results include the scorecard and the evaluation of openness of the selected major academic publishers. These are based on seven key factors: (i) Fraction of open access (OA) journals and their articles of the total publication output, (ii) costs of OA publishing (article processing charges, APC), (iii) use of Creative Commons (CC) licensing, (iv) self-archiving policies, (v) access to text and data mining (TDM), (vi) openness of citation data, and (vii) accessibility of information relating to OA practices. To take a look beyond the publisher level into journal level practices we also sampled individual journals. We use the samples to discuss the distribution of journals according to APCs, their licensing and three impact metrics (CiteScore 2016, Scimago Journal & Country Ranks (SJR) 2016, and Source Normalized Impact per Paper (SNIP) 2016). The evaluation of the selected publishers with the scorecard indicates, for example, that the fraction of OA journals and their articles of the total publication output runs low within this group. In our sample of journals, the most expensive OA journals also seem to bear the highest impact metrics. A definite view on the matter, however, would require more extensive data and further research. We conclude by discussing key aspects and complexities in quantitative evaluation and in the design of a standardized assessment of publisher openness, and note also further factors that could be included in future versions of the scorecard.