KCSE 편집인 Workshop (2017 년도 ) 일시 : 2017. 11. 30. ( 목 ) 장소 : 한국과학기술회관중회의실 2 주관 : 한국과학기술편집인협의회 최재석교수 대외협력위원장 jschoi@gnu.ac.kr
Contents I. What? (Definition) II. Why? (Purposes) III. How? (Peer Review Process) IV. New Peer Review System V. Difficulties and Problems VI. An Example: Peer Review Process & System VII. Discussion & Design : Future Peer Review System? 2
I. What is peer review? Definition : Review by peers The peer review process란동일한분야에종사하는사람에의하여연구방법및결론등을정밀하게심사받는한과정. 이는학문적연구분야에서필연적인과정임. 이과정은필연적이지만가끔더디고, 비효과적이며오해를불러일으키기도함. Includes: internal review (by editorial staff) external review (by experts in the field) 3
Peer-reviewed journals may also be called refereed or juried journals. Peer review is a process by which manuscripts are submitted for publication in a scholarly or research journal and are reviewed by subject experts for comment, evaluation, and approval. These articles often go through a revision process after the peer review and prior to publication. Many articles are submitted to the journal editors. Of these articles, a select few make it to publication. Peer review process definition adapted from the Concise Dictionary Of Library And Information Science. 4
II. Why? Purpose The process는의미없는실험결과의유포, 비정당한주장 (unwarranted claims), 수용할수없는분석및해석 (unacceptable interpretations), 그리고단순한주관적인관점 (personal views) 등을사전에방지함에목적이있음. 따라서이는서로간의연구를심사하는동료들의능력에의존적임. 이는정당성 (legitimate) 있는정보를갖는결과를도출함에목적이있으며그분야에대토론혹은결론을가져다줄수있음. 5
What do editors want from papers? 중요성 (Importance) 새로움 (Originality) 독자의흥미유발성 독자및연구자들에게대한유익성 진실성 (Truth) 와우 성 (Excitement/ wow factor) 투명성및문장력 (Clear and engaging writing) 6
When did peer review start? Some would say that Peer Review goes back as far as the 17 th century, when it was known as The Inquisition of the Holy Roman and Catholic Church. Scholars works were examined for any hints of heresy. Galileo Wikipedia (http://en.wikipedia.org/wiki/peer_review) 7
Peer review in Modern times Peer review (known as refereeing in some academic fields) is used in: 1. Publication process 2. Awarding of funding for research 3. Patents 4. Standards Each of these involve slightly different practices, but ultimately colleagues are evaluating each other. 8
What we know about peer review? Research evidence 9
10
Your Manuscript On Peer Review I m a big fan of peer review. Most of the revisions that reviewers suggest are very reasonable and sometimes really improve the manuscript. Other times it doesn t seem to work that way. I ve noticed this is especially true when the manuscript goes through multiple rounds of peer review at different journals. It can become a franken-paper, unloved by the very reviewers who made it. 11
Why do peer review? Filter More papers submitted than could be printed Eliminate bad science, pseudo-science, harmful science... Aura of quality (only the best gets in) Collegial stamp of approval Professional obligation to the principles of one s discipline 12
So, what s the problem? Famous papers that were published and did NOT get peer reviewed: Watson & Crick s 1951 paper on the structure of DNA in Nature Abdus Salam s paper Weak and electromagnetic interactions (1968). Led to Nobel Prize Alan Sokal s Transgressing the Boundaries... in 1996 turned out to be a hoax. Now known as the Sokal Affair. Famous papers that were published and passed peer review that later proved to be fraudulent: Jan Hendrik Schon (Bell Labs) submitted and passed peer review 15 papers published in Science and Nature (1998-2001) found to be fraudulent. Igor and Grichka Bogdanov 1999 & 2002 published papers in theoretical physics believed by many to be jargon-rich nonsense. Famous papers that got rejected that later turned out to be seminal works: Krebs & Johnson s 1937 paper on the role of citric acid on metabolism was rejected by Nature as being of insufficient importance, was eventually published in the Dutch journal Enzymologia. This discovery, now known as the Krebs Cycle, was recognized with a Nobel prize in 1953. Black & Scholes 1973 paper on the pricing of options and corporate liabilities, rejected many times, was eventually published at the intersession of Merton Miller to get it accepted by the Journal of Political Economy. This work led to the Nobel Prize. 13
최근에 Peer review와관련하여제기되는문제는 Peer review circle입니다. - Peer review circle 은 Peer review ring 이라고도하며, 논문의가짜리뷰를위해몰래만들어지는과정을말합니다. 정상적인 Peer review 과정에서논문저자는 Peer reviewer가누구인지알수없습니다 (Single Blind Review Process). 그러나, Peer review circle이비밀스럽게만들어진경우, 저자는 Peer review가누구인지알수있으며, 가짜 Peer review를통해부실한논문의출판이승인될수있습니다. - 2014년 7월, Sage Publications사는 Peer review circle과관련저자를적발하고, Vibration and Control 저널에관련된약 60 편의논문을철회처리했습니다. - 2013년 5월, 저널편집장이저자와가짜 Peer reviewer 간이메일수발신기록을적발할때까지계속되었으며, Peer reviewer는논문의저자와직접적연락을해서는안됨에도불구하고, 가짜 Peer reviewer는투고논문의저자와연락할수있었던것입니다. - 계속적인추적으로 Sage Publications사는 Peer review circle과관련된사건을조사하였으며, 130개의가짜 Peer review 의심이메일계정을보고하였다. 그에따르면, 논문의투고이후수분안에리뷰코멘트가작성되기도하였는데이러한코멘트는모두가짜 Peer reviewer가작성한것으로확인되었다.
Two Recent Articles... Is Peer Review Broken? by Alison McCook The Scientist, vol 20 (2), Feb 2006, pg 26. http://www.the-scientist.com/2006/2/1/26/1/ - Submissions are up, reviewers are overtaxed, and authors are lodging complaint after complaint about the process at top-tier journals. What's wrong with peer review? Journal lays bare remarks from peer reviewers by Emma Marris Nature, vol. 439, 9 February 2006, page 642 http://www.nature.com/nature/journal/v439/n7077/full/439642b.html - Cloak of anonymity shed by new publication. Editors of a journal launched this week are out to revolutionize peer review. By publishing signed reviews alongside papers, they hope to make the process more transparent and improve the quality of the articles. 15
III. Peer Review Process 16
17
Kinds of Review Process Single Blind Review Process Double Blind Review Process Open Review Process - Typically it is a double blind process: the reviewers do not know who the author is and the author does not know who the reviewers are. That way only the merits of the paper are evaluated. Hybrid Review Process 18
Peer Review 유형, 장점및단점 리뷰유형장점단점 싱글블라인드리뷰 더블블라인드리뷰 오픈리뷰 하이브리드리뷰 리뷰어들은심사하는논문에대해압력이나방해를받지않고자유롭게비판할수있습니다. 저자가누구인지아는것만으로도연구주제에대해 파악하고투고된논문을평가하는데필요한많은정 보를얻을수있습니다. 지역이나성별및저자의배경에따른차별요소가현 저하게줄어들수있습니다. 저자와리뷰어둘다개인의공격혹은압력의대상 이되지않습니다. 투명성 (Transparency) 을갖춤으로써조작의위험성 과편견을줄일수있습니다. 투명성을갖춤으로써조작의위험성과편견을줄일 수있습니다. 리뷰어들은장기간에걸쳐논문에대한광범위한코멘트를제공하고저자와소통할수있습니다. 저자들은자신의논문이오픈시스템에공개된날을 자신의논문의우선출판일 (publication priority) 로 지정할수있습니다. 개인적편견 : 리뷰어가저자가누구인지알기때문에, 객관적으로논문을평가하지않을수도있습니다. 성별혹은지역에대한편견과같은다른차별요소가 논문의운명을결정할수있습니다. ( 예를들어, 일 부리뷰어들은개발도상국에서수행된연구에대해 부정적일수도있습니다 ). 저자의배경을아는것이실제로심사하는연구를이 해하는데도움이될수있습니다. 더블블라인드시스템이완벽하게구축되어있지는않 습니다. 따라서, 리뷰어들은연구주제, 문체등을기 반으로하여저자의신원을추측할수도있습니다. 리뷰어들은대중들이만족할만한피드백을제공해야 된다는것에압박감을느끼거나논문에대해비판을해야함에도불구하고이를자제할수도있습니다. 오픈리뷰중에저자들은까다로운질문들에직면할 수있습니다. 하지만이러한이유로투고되는논문의 완성도는높아질수있습니다.
IV. New Peer Review
4.1 What is Author-guided Open Peer Review? It is time to challenge the idea that scientific peer review can only be arranged and handled by journal editors. We propose a research assessment process complementary to journal-handled peer review where authors themselves can invite experts to openly evaluate their work. The Open Peer Review Protocol and our recent article on academic self-publishing describe in detail the requirements for implementing author-guided open peer review. The key features are: Authors invite expert peers to formally evaluate their work posted in any online archive (libraries, repositories, preprint servers, etc). Reviewers who accept submit a detailed qualitative and quantitative assessment of the work. 21
The reviewer s name and any conflict of interest are publicly disclosed. Reviews are published with a creative commons license (or similar) and become publicly available along with the original work. Reviews are subject to commentary and evaluation by the entire community. Author-guided open peer review can be implemented at any stage of an article s lifetime: a) before journal submission, (b) during journal peer review (in agreement with the journal s editor), and (c) after journal publication. Read the Open Peer Review Protocol for more detailed implementation instructions and read and sign the Manifesto of Independent Peer Review. 22
4.2 Why is the Open Peer Review? The current blind peer review model has repeatedly been criticized as slow, opaque and prone to bias or even fraud. In addition, the fact that journals monopolize research evaluation creates an environment of fierce academic competition, opposed to open collaboration for the benefit of science and society. Author-guided open peer review: Offers reviewers incentives to provide good quality and helpful reviews. Helps journal editors make better-informed decisions on manuscripts that merit publication and reduce risk-taking. Creates an alternative evaluation system bypassing journal publication costs, which is especially important for authors with limited resources (see developing countries). Allows the dynamic evaluation of scientific ideas and results, in accordance with the concept of critical rationalism. Encourages collaboration between authors and reviewers. 23
4.3 Portable Peer Review 모든연구자들에게논문제출후에거치는 Peer Review 과정은길고부담스러운과정이다. 평균 Peer Review 기간은 80일이다. 하지만, 보완요청과또한번의리뷰과정을감안하면더길어진다. 여기서의문제는거절당하고다른저널에제출하면동일한과정을반복해서겪어야한다는것이다. 출판계내에서도이런시스템의문제를개선하기위한노력이있어왔고, 그중하나가 Portable Peer Review입니다. Portable Peer Review의핵심은 Peer Review과정을줄이는것입니다. 이미 Peer Review 과정을거쳤으면다른저널에서같은과정을거치지않게하거나, 아예저널과무관한독립기관에서 Peer Review를거치고이를복수의저널에통용되도록하는것입니다. 연구자들에게저널에논문을제출하고수록되는과정에서가장소모적인과정중의하나가 Peer Review입니다. 따라서기존 Peer Review과정의효율성을제고하기위한대안이제시되는것은반가운일입니다. 이러한대안들이잘작동하려면독립 Peer Review 서비스의역할뿐아니라저널과출판사들의적극적인협조가이루어져야할것입니다.
4.4 자가출판 (Self-Publishing) 빠른기술발전과더불어서저자가스스로전자책을출판하는자가출판 (Self-Publishing) 이새로운트렌드로주목받고있다. 출판혹은도태 의경쟁하에놓인연구자는참신한아이디어를담은논문을스스로출판해보고자하는것에서시작하였다. 자가출판의경우에는영국의동화작가베아트릭스포터의 피터래빗의이야기 로잘알려진작가이다. 1901년동화책을출판이거듭거정당하자스스로출판을하게되고, 이는대성공을거둔다. 약 100년전에도이렇듯자가출판을통해서성공한사례가많으며, 오늘날은디지털기술의발전으로더욱접근이용이해졌다. 전통적출판방법을더선호하는학자들은우려를표하기도합니다. 정식적이고신뢰가능한출판방법은아니라는주장이있지만, 자가출판은출판에관한제약이거의없고저자들에의해서블로그나전자책을통해서자유롭게의견을소통할수있으며, 출판결과를빨리접할수있다는장점이있다.
4.5 Results-free Peer review 연구결과가최초가설과다르게나올때면고민에빠지기마련이며, 저널게재에대한두려움이들수도있습니다. 논문평가는사람이하는것이니항상 100% 객관적일수는없으며, 주어진가설의달성여부에대한가중치는평가자마다다를수있습니다. 최근한저널이흥미로운실험을진행했습니다. BMC Psychology라는오픈저널이 result-free Peer review방식을만들었습니다. Peer review 과정에서리뷰어가결론부분을보지못하게하고논문의목적과취지, 방법론에얼마나충실한지를보고논문을평가하는방식입니다. 이방법은두번의단계를거치는데, 첫번째리뷰에서리뷰어는제출된논문의결론을보지않고논문의취지와방법론만가지고평가합니다. 두번째단계에서리뷰어는동일한논문을논문의결론을포함하여평가하게되는데, 결론자체가유효한가가아니라그결론이취지와방법론에정합성을갖는지를평가합니다.
V. Difficulties and Problems Means different things at different journals Slow Expensive Subjective Biased Open to abuse Poor at detecting errors Almost useless at detecting fraud 27
Is peer review reliable? (How often do two reviewers agree?) NEJM (Ingelfinger F 1974) Rates of agreement only moderately better than chance (Kappa = 0.26) Agreement greater for rejection than acceptance Grant review Cole et al, 1981 real vs sham panel, agreed on 75% of decisions Hodgson C, 1997 two real panels reviewing the same grants, 73% agreement Are two reviewers enough? Fletcher and Fletcher 1999 - need at least six reviewers, all favouring rejection or acceptance, to yield a stats significant conclusion (p<0.05) 28
Should we mind if reviewers don t agree? Very high reliability might mean that all reviewers think the same Reviewers may be chosen for differing positions or areas of expertise Peer review decisions are like diagnostic tests: false positives and false negatives are inevitable (Kassirer and Campion, 1994) Larger journals ask reviewers to advise on publication, not to decide 29
Bias Author-related Prestige (author/institution) Gender Where they live and work Paper-related Positive results English language 30
Prestigious institution bias Example: Peters and Ceci, 1982 Resubmitted 12 altered articles to psychology journals that had already published them Changed: title/abstract/introduction - only slightly authors names name of institution, from prestigious to unknown fictitious name (eg. Tri-Valley Center for Human Potential ) 31
Peters and Ceci - results 3 편은재심 : Three articles recognised as resubmissions 1 편게재 : One accepted 8 편게재거절 : Eight rejected (all because of poor study design, inadequate statistical analysis, or poor quality: none on grounds of lack of originality) 32
How easy is it to hide authors identity? Not easy In RCTs of blinded peer review, reviewers correctly identified author or institution in 24~50% of cases 33
Reviewers identified (open review) results of RCTs Asking reviewers to sign their reports in RCTs made no difference to the quality of reviews or recommendations made Godlee et al, 1998 van Rooyen et al, 1998 van Rooyen et al,1999 34
Open review on the web Various experiments and evaluations are underway 35
What makes a good reviewer? results of RCTs Aged under 40 Good institution Methodological training (statistics & epidemiology) 36
What might improve the quality of reviews? Reward/credit/acknowledgement? Careful selection? Training? Greater accountability (open review on web)? Interaction between author and reviewer? (real time open review) 37
VI. An Example: Peer Review Process & System 1. 투고 2. 1 차심사 보통 2~3 인의동료과학자에의해심사및결과통보 accept, minor (major)revision, 또는 reject 3. 수정투고 (revised manuscript + response to reviewer s comments) 4. 2 차심사 minor revision 또는 accept 5. 게재승인통보 6. 교정쇄확인 7. 출판 38
http://home.jeet.or.kr/ 39
Review Process of JEET JEET Peer Review Process (www.jeet.or.kr) 40
A/E-Reviewers Flow Chart (1 st Review Mode) 41
42
EIC & Admin - P/E - A/E Flow Chart 43
VII. Discussion Pros and Cons of Peer Review? Future Peer Review System?... Focused on Customers(Authors) Self Standing & Feedback System Effectiveness On Line Real Time Else etc.. Your Opinion & Idea??? 44
Peer Review? It s not perfect grist for a lot of mills The Web has made it less of a obstacle to access Different disciplines have different perspectives different issues Pedagogical yardstick for students 45
Using email Peer review on the Internet e-mail based online peer review See Peer Review of Scholarly Publications in Health, Online Manuscript Peer Review and Tracking Systems and Physics of Plasmas Online Manuscript Submission and Peer Review Discussion approach better interaction among authors, reviewers and the editorial body JIME Open Peer Review Process Wikis Immense potential to conduct peer review Blogs post publication comments See Article Note: On Blogging as Tool, but Really About Using RSS 46
Global benefits Worldwide increase in access to scientific literature Increased opportunities for collaboration among experts worldwide Increased speed to disseminate scientific literature with electronic communities More informal peer reviews Quality needs to be maintained See Scholarly Electronic Publishing Weblog 47
Design Your Peer Review System 48
References & Acknowledgements 1. JULIA WILSON, Standing up for Science 3 PEER REVIEW; The nuts and bolts, SENSE about SCIENCE, 2012 2. Irene Hames, Peer Review and Manuscript Management in Scientific Journals, Blackwell Publishing in Association with ALPS, 2007 3. Erik Cobo, Albert Selva-O Callagham, Josep-Maria Ribera, Francesc Cardellach, Ruth Dominguez, Miquel Vilardell, Statistical Reviewers Improve Reporting in Biomedical Articles: A Randomized Trial, PLOS ONE, 2007.03 4. Mark Ware, Peer review: benefits, perceptions and alternatives, Publishing Researching Consortium, 2008 5. E Cobo, J Cortés, J M Ribera, F Cardellach, A Selva-O Callaghan, B Kostov, L García, L Cirugeda, D G Altman, J A González, J A Sànchez, F Miras, A Urrutia, V Fonollosa, C Rey-Joly, M Vilardell, Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial, thebmj, 2011.11 6. Sherril Gelmon, Cathy Jordan, Susan Gust, Cathy Burack, Innovations in peer review: Expanding the boundaries for community-engaged scholarship, International Association for Research on Service- Learning and Community Engagement, 2012 7. Peggy Dominy, Jay Bhatt, Peer Review in the Google Age: Is technology changing the way science is done and evaluated?, e-lis, 2006 8. Amy Bourke-Waite, Innovations in scholarly peer review at Nature Publishing Group and Palgrave Macmillan, Insights, 2015.07 9. Enago Academy, Portable Peer Review, 2017 10. http://www.openscholar.org.uk/open-peer-review/ 11. https://www.biomedcentral.com/collections/rfpr 12. http://www.ibric.org/myboard/read.php?board=isori&id=10022 49