Log in


  • 25 Apr 2018 11:16 | Michael Seadle (Administrator)

    Data falsification cases generally take time to discover, and generally require someone who is motivated enough to look for problems. Falsification should theoretically be found in the course of peer review, and sometimes is, but journals do not routinely make public the detailed results of peer review. Data falsification can also be hard to prove with certainty. This column will look at a case from social psychology that arose in the wake of the Diederik Stapel retractions. Stapel admitted his guilt and his name is now routinely part of discussions about data falsification. The 2014 case under discussion here is somewhat different because the author of the retracted papers still insists on his innocence. Since the person’s name is irrelevant to the scholarly discussion, this column will refer to him only as JF. Anyone who really wants to learn his name need only look at the reference.

    The issue in the JF case involves datasets whose results are statistically too perfect. An unnamed whistleblower did an analysis:

    “The chances of this happening were one in 508,000,000,000,000,000,000, he claimed.”(Kolfschooten, 2014)

    The whistleblower is apparently known to the university and to the National Board for Research Integrity (LOWI) in the Netherlands (Kolfschooten, 2014). Maintaining the whistleblower’s anonymity seems legitimate as long as due process is followed and the accused has a reasonable chance to respond. Just how much opportunity JF had to respond is unclear from published sources. He implied that the opportunity was limited in an open letter to Retraction Watch (Amarcus41, 2014):

    The rapid publication of the results of the LOWI and UvA [University of Amsterdam] case happened quite unexpectedly, the negative evaluation came unexpectedly, too. Note that we were all sworn to secrecy by the LOWI, so please understand that I have to write this letter in zero time. Because the LOWI, from my point of view, did not receive much more information than was available for the preliminary, UvA-evaluation, and because I did never did something even vaguely related to questionable research practices, I expected a verdict of not guilty… I do feel like the victim of an incredible witch hunt directed at psychologists after the Stapel-affair.

    JF appears not to have kept the original data, only his summary of the results, which is a lesson to other scholars not to be too ready to clean their files in case the original data are needed. Investigators also raised suspicions about the data in the thesis of one of JF’s doctoral students. The doctoral student was declared innocent of wrongdoing, because the data came from JF. For JF the trouble did not stop:

    A panel of statistical experts from UvA that embarked on a second, more comprehensive investigation found “strong evidence for low veracity” of the results in all three papers, as well as in five others.” (Kolfschooten, 2016)

    And “… as part of a settlement with the German Society for Psychology (DGPs)” JF agreed to further retractions (Palus, 2016). The weight of opinion has been strongly against JF to the point that he left the academic world for private practice. (Stern, 2017)

    In a sense the case is closed, but questions remain. Accusations of fraud tend to come in groups, perhaps because an initial case inspires people to look more carefully, and perhaps because opinion shifts away from a presumption of innocence. After the Stapel case, Uri Simonsohn built a statistical tool to detect the possibility of certain kinds of fraud where the data patterns were too perfect to be believed (Enserink, 2013). There is no evidence that this tool was involved in JF’s case, but the principle appears to be the same: the data were just too perfect, not merely once, but in paper after paper. Of course high quality data are what scholars need to get publications. The push to get perfect data is strong.

    One should not forget how complex the creation of a research data set is, and that experienced researchers learn how to get good results without necessarily faking or directly manipulating the data. Selecting participants is an art in a world where genuine random selection is often impossible. A highly successful scholar might unconsciously seek just the right subjects without obvious tampering, and might learn how to ask exactly the right questions in exactly the right way to elicit exactly the right responses without further manipulation. Perhaps this seems implausible, but highly successful researchers must do something different or they would not be quite so untypical.

    In any particular case, repeated perfect results must seem unlikely, but it may be less unlikely that factors other than outright fraud could play a role. In the case of JF, the investigation seems never to have considered other reasons.

    One of the lessons from this case for researchers young and old is to keep all of the experimental data over a longer period. The lack of original data was a factor in this case that counted strongly against JF.

    References

    Amarcus41. 2014. “Social Psychologist Förster Denies Misconduct, Calls Charge ‘Terrible Misjudgment.’” Retraction Watch. 2014. Available online.

    Enserink, Martin. 2012. “Fraud-Detection Tool Could Shake up Psychology.” Science 337 (6090). American Association for the Advancement of Science: 21–22. Available online.

    Kolfschooten, Frank van. 2014. “Scientific Integrity. Fresh Misconduct Charges Hit Dutch Social Psychology.” Science (New York, N.Y.) 344 (6184). American Association for the Advancement of Science: 566–67. Available online.

    Kolfschooten, Frank van. 2016. “No Tenure for German Social Psychologist Accused of Data Manipulation.” Science, July. Available online.

    Palus, Shannon. 2016. “Psychologist Jens Förster Earns Second and Third Retractions as Part of Settlement.” Retraction Watch. 2016. Available online.

    Stern, Victoria. 2017. “Psychologist under Fire Leaves University to Start Private Practice – Retraction Watch.” Retraction Watch. 2017-12-12. Available online.

  • 19 Apr 2018 11:13 | Michael Seadle (Administrator)

    Problems with data are arguably the most serious issue for information integrity in the research world, because they undermine the ability of scholars to build on past results. These problems come in many variations, including people who make up fake data, people who manipulate data to get specific results, and people who leave out data or sources. Each of these represent some form of misconduct when done deliberately. Nonetheless not everyone is guilty of malicious intent. Ordinary negligence plays a role too. The results remain unreliable and irreproducible, but the persons involved may be innocent of intentional wrongdoing. This column looks at the scholarly literature on “honest” errors.

    Classification Issues

    Resnik (2012) explains that recognizing honest error is important but hard:

    “It is important to distinguish between misconduct and honest error or a difference of scientific opinion to prevent unnecessary and time-consuming misconduct proceedings, protect scientists from harm, and avoid deterring researchers from using novel methods or proposing controversial hypotheses. … the line between misconduct and honest error or a scientific dispute is often unclear,”

    Precisely what constitutes honest error may depend on personal judgment. An older study by Nath (2006) in the Medical Journal of Australia looked at ”[a]ll retractions of English language publications indexed in MEDLINE between 1982 and 2002…” and “[t]wo reviewers categorised the reasons for retraction of each article…”. Nath concluded that:

    “Of the 395 articles retracted between 1982 and 2002, 107 (27.1%) were retracted because of scientific misconduct, 244 (61.8%) because of unintentional errors, and 44 (11.1%) could not be categorised.”

    The percentage of unintentional errors suggests surprisingly high rate of unintentional error. While it is possible that misconduct has increased significantly over time (see below for more recent numbers), the more likely lesson here is that it matters how the classification is made. It is hard to know how accurate the classifications of misconduct are under circumstances where the assumption of innocence is not always strictly observed after an accusation has been made.

    Estimates of Size

    Later studies do not confirm the Nath estimate about the number of unintentional errors. An article by Arturo Casadevall (2014) argues that

    Analysis of the retraction notices for 423 articles indexed in PubMed revealed that the most common causes of error-related retraction are laboratory errors, analytical errors, and irreproducible results. … The database used for this study includes 2047 English language articles identified as retracted articles in PubMed as of May 3, 2012…

    This suggests that the cause of just under 12% of the PubMed retractions are essentially ordinary human error. A different study by Moylan and Kowalczuk (2016) looks at the BioMed Central journals finds a similar percentage:

    “Honest error accounted for 17 retractions (13%) of which 10 articles (7%) were published in error. … A total of 13 articles (10%) of retractions were due to problems with the data. Often these issues occurred through honest error in how the data were handled, for example … although in some cases it is difficult to determine whether honest error or misconduct was the cause. “

    Daniele Fanelli (2016) offers a somewhat higher percentage of honest error:

    However, retractions reliably ascribed to honest error account for less than 20% of the total, and are often a source of dispute among authors and a legal headache for journal editors. The recalcitrance of scientists asked to retract work is not surprising. Even when they are honest and proactive, they have much to lose: a paper, their time and perhaps their reputation. Much reluctance to retract errors would be avoided if we could easily distinguish between ‘good’ and ‘bad’ retractions.

    In this case good retractions are generally ones where the authors recognize their own mistake and ask for the paper to be withdrawn. Fanelli (2016) makes the further argument that:

    Self-retractions should be considered legitimate publications that scientists would treat as evidence of integrity. Self-retractions from prestigious journals would be valued more highly, because they imply that a higher sacrifice was paid for the common good.

    This could, as he notes, be open to abuse, but some abuse could well be tolerable in the interests of providing an incentive for researchers to withdraw misleading results so that they do not mislead other scholars. Considering present publication pressure and the effect of public opinion, researchers may be unwilling to admit honest errors because they will be thought guilty of misconduct. It may be hard to escape censure regardless of the choice.

    Greyscale Measurement

    One of the measurements that can help define honest error is the degree to which errors confirm the desired conclusions. This is not to say that every error in favor of the authors’ arguments is dishonest, but errors that weaken the conclusion are more likely unintentional. There is of course a human tendency to believe confirming results and to doubt disruptive ones, and a part of research training that may need more emphasis is a healthy skepticism toward desired results. Another form of measurement has to do with the frequency of error. Everyone makes some errors. When authors repeatedly make errors, it may be reasonable to think that the errors follow a standard distribution where some are for and some against the conclusions. A pattern that is consistently in favour of the desired conclusion may imply more bias than honesty.

    Those judging integrity should not forget that honest errors exist, and that people under career or social pressure may be more error prone without particular ill intent.

    References

    Casadevall, Arturo, R. Grant Steen, and Ferric C. Fang. 2014. “Sources of Error in the Retracted Scientific Literature.” FASEB Journal 28 (9): 3847–55. Available online.

    Fanelli, Daniele. 2016. “Set up a ‘self-Retraction’ System for Honest Errors.” Nature. Available online.

    Moylan, Elizabeth C., and Maria K. Kowalczuk. 2016. “Why Articles Are Retracted: A Retrospective Cross-Sectional Study of Retraction Notices at BioMed Central.” BMJ Open 6 (11). Available online.

    Nath, Sara B., Steven C. Marcus, and Benjamin G. Druss. 2006. “Retractions in the Research Literature: Misconduct or Mistakes?” Medical Journal of Australia. Available online.

    Resnik, David B., and C. Neal Stewart. 2012. “Misconduct versus Honest Error and Scientific Disagreement.” Accountability in Research. Available online.

  • 11 Apr 2018 11:55 | Melanie Rügenhagen (Administrator)

    The HEADT Centre launches its first column on Information Integrity. It will appear weekly on Wednesdays with contributions primarily by Principal Investigator Prof. Dr. Michael Seadle. Potential other authors include Dr. Thorsten Beck whose expertise is in image manipulation.

    Follow this blog where we announce each new article, if you are interested in scholarly perspectives on Information Integrity. You can also reach the column from the menu of our page (Research ->  Column on Information Integrity).

    Read the first article here!

  • 11 Apr 2018 11:04 | Michael Seadle (Administrator)

    What is Information Integrity?

    Information integrity is fundamentally about what makes information true or false, both at the scholarly level (research integrity) and for public and policy discourse. There are reports about false information almost daily. A recent example involves the BBC, which has long been a model for the integrity of its reporting. (Sweney, 2018) This column will focus mainly on the scholarly aspects of information integrity, but the effect of integrity problems on policy matters (public health issues, for example) will not be ignored.

    The topic includes a broad range of problems, including data falsification, image manipulation, and plagiarism. While plagiarism is perhaps the most prominent issue, it is primarily an ethical and legal issue and generally does not undermine scholarship that builds on it because the results are not necessarily false. This column will discuss all aspects of information integrity, but will focus especially on data problems, since no generalized detection tools exist, though a few disciplines (such as psychology) are working on them.

    A core concept in my book on “Quantifying Research Integrity” (Seadle, 2017) is the greyscale approach: integrity issues rarely separate neatly into simple black and white, guilty or innocent, categories. Many scholarly works have imperfections, and problematic works may still contain valid information. From the viewpoint of a university or a publisher, formal decision-making processes involving punishments and retractions may make black-and-white decisions about integrity problems preferable, but such black-and-white decisions can themselves be an integrity issue, since an overly simplistic label is at least partly untrue.

    Scholarly literature contains a wealth of examples of integrity problems going well back in historical time. Today there are tools for investigating plagiarism and for examining some kinds of image manipulation. Data falsification presents more of a challenge because of its variety and complexity. Simple cases such as that of Diederik Stapel, who admitted manufacturing his results, are rarer than scholars who make poor choices about data or its interpretation. (Bhattacharjee, 2013) Unintentional error is also an information problem, even if it is not falsification.

    Selection Bias

    Selecting problematic research may have lasting effects on political discourse as well as on scholarship. While the evidence for climate change appears to be overwhelming, studies by a small number of skeptics have given oil and coal lobbies in the US a tool for opposing effective measures to reduce hydrocarbons in the atmosphere. Natural science builds on the ability to reproduce results, and when many scientists produce the same results based on a wide range of measures, the conclusions are normally accepted as valid. Lay persons unfamiliar with the scholarly literature sometimes select flawed studies that confirm their own personal preferences.

    Other more historical examples of selection bias can be found in claims about the inferiority of people in the US who were not of northern European descent — not merely those from Africa, but also from Italy, Ireland, and eastern Europe. Such claims were popular among the right wing in many European countries in the Nazi era, and are still popular among some groups today. A basis for them reaches back to Christoph Meiners (Grundriß der Geschichte der Menschheit, 1785) in the 18th century and is as modern as “The Bell Curve” by Richard Herrnstein and Charles Murray (1994). These studies did not fake their data and used scientific methods that seemed appropriate at the time, but they were selective about what evidence they included, and today it is widely accepted that the exclusions skewed results in a particular direction.

    Selection bias may have social and cultural origins that can change over time. For those who believe in the inerrancy of Holy Scripture, the data confirming evolution is invalid. A scholar of research integrity needs in some sense to be an historian, in order to understand the research in time and place, and to be an ethnographer, in order to understand integrity violations across cultures and disciplines. No one should imagine that integrity research involves simple labels.

    The Research Integrity Literature

    This column will focus on discussing papers about research integrity and will look at specific cases, whose complexity gives opportunities to apply a greyscale analysis. There are many good sources of information, not the least of which is Retraction Watch (Oransky, 2018), which provides an excellent news feed and classifies cases of retractions by type and field. Retractions may represent only part of the problem, simply because discovering problems is hard and because false positives may distract from more important issues. The ability to reproduce results is a classic hallmark of good science, but there is good evidence that results in behavioral and social science studies are harder to reproduce than natural-science results for the simple reason that social circumstances change.

    The goal of this column is scholarly, not investigative. It does not actively seek out new cases where research integrity may have been violated, but seeks to examine existing cases in order to apply a greyscale understanding of what happened and what the consequences are. As Principal Investigator for the research integrity part of the HEADT Centre, I will be the primary columnist, but others will likely contribute as well, including Dr. Thorsten Beck, who specializes in image manipulation.

    References

    Bhattacharjee, Yuduit. 2013. “The Mind of a Con Man.” New York Times, April 26, 2013. Available online.

    Seadle, Michael. 2017. Quantifying Research Integrity. Morgan Claypool: Synthesis Lectures on Information Concepts, Retrieval, and Services. Available online.

    Sweney, Mark. 2018. “No Title.” New York Times, April 4, 2018. Available online.

    Oransky, Ivan, and Adam Marcus. 2018. “Retraction Watch.” 2018. Available online.

  • 28 Feb 2018 11:52 | Melanie Rügenhagen (Administrator)

    Michael Seadle (HEADT Centre) and David Neal (Elsevier) are on the steering committee of the NetDiploma project, which is currently giving a two-day workshop at Northumbria University. Read all about the NetDiploma project on their project website.


    Photo: Courtesy of the NetDiploma Project

  • 28 Feb 2018 11:45 | Melanie Rügenhagen (Administrator)

    The Berlin School of Library and Information Science at Humboldt-Universität zu Berlin will launch a new one-year postgraduate certificate programme in Digital Information Stewardship in autumn 2018. It is taught entirely in English and offers an option to take courses at University College Dublin.

    This programme blends distance learning with video-based discussions and three brief face-to-face meetings. The goal is to enable people to continue their careers and simultaneously to open new job opportunities.

    Find further details such as the admission requirements on our website: https://www.ibi.hu-berlin.de/en/teaching/postgraduate-certificate-programme/postgraduate-certificate-DIS

    You are welcome to apply. Contact the programme coordinator, Melanie Rügenhagen (melanie.ruegenhagen [at] hu-berlin.de), if you are interested.


  • 26 Feb 2018 11:43 | Melanie Rügenhagen (Administrator)

    The Faculty of Humanities, Berlin School of Library and Information Science, invites applications for a:

    S-Junior Professorship for Information Management

    (Elsevier-HEADT Centre – Einstein-Stiftung professorship)

    Applications will be considered until 14 March 2018. Find all details about this position on Humboldt-Universtität zu Berlin’s website.



  • 9 Feb 2018 13:44 | Thorsten Beck (Administrator)

    Research Integrity is a subject that not only concerns journals and editors, but experts from many fields and disciplines, across Europe and worldwide.

    The Printeger Conference on Research Integrity in Bonn in early February 2018 provided a platform for understanding and discussion of the many facets and aspects that play a role when trying to deal with integrity related issues. (https://printeger.eu/conference2018/) Researchers, commission members, entrepreneurs and experts from a wide variety of fields gathered at Bonn University and discussed integrity as a cultural and social phenomenon and as a threat to scholarly knowledge production. Moreover significant steps were made to develop an updated version of the ALLEA code of conduct 2011.

    In many of the conference sessions it was debated how training, workflows, standards and policies could be transformed and used to keep science clean from error or misbehavior: Rachel Douglas-Jones for example explained in her lecture how it was important to translate a national code of conduct into universities in Denmark. In her eyes it were foremost PhD students that in the future will function as agents of reform. Her research group analyzed how doctoral students eventually change their concepts of integrity over time and how the practical demands of scholarly careers may have an effect on research practices.

    Hugh Desmond from KU Leuven gave a talk in which he elaborated on how to actually distinguish between research misconduct and incompetent research – a line that is not always easy to draw. He pointed out that existing codes of conduct are still not consistent when it comes to defining these boundaries and that there are no clear and unequivocal criteria for when misconduct must be considered a criminal act. He claimed that it requires a clear terminology for incompetence in research and that it is necessary to either follow through on strengthening criminal statutes or to reconceive how to define misconduct.

    Jennifer Gewinner from ETH Zürich presented the results of a study in which she showed how retraction notices in major journals rather conceal fraud and indicate error instead rather than presenting the real nature and dimension of misconduct. In her eyes it is desirable to establish clear standards for how retraction notices should be designed and of the information that need to be presented to the wider academic public.

    Nicholas Fox from Rutgers University analyzed questionable research behaviors amongst tenure track professors in the fields of psychology. His creative approach used social networks as a source for estimating the size of the population, which engaged in questionable practices. He found that 18 percent of the analyzed population admitted to have engaged in such practices over the last 12 month, which is – given the fact that this number might only represent the tip of the iceberg – must be considered quite alarming.

    Mario Malicki from the University of Amsterdam looked at information policies and instructions to authors in a wide variety of journals. Less than a third of all journals, he reported, actually mentions research integrity explicitly and 92 journals have no instructions for authors at all.

    These are only a few spotlights and insights from this inspiring conference (for more information see: https://printeger.eu/agenda/), and it is worth mentioning that much more attention is required to better understand the phenomenon of research integrity and its consequences for the scholarly community at large. It often has been claimed that scholarly work bases on mutual trust, but as long as the research community is not fully aware of the many aspects that may lead to mistakes or how sloppiness can lead to poor results which in the end harm the health and lives of people there need to be many more event like the Printeger Conference 2018.

    Feel free to share or comment this article and please visit the HEADT Centre website again soon!

  • 7 Feb 2018 13:41 | Thorsten Beck (Administrator)

    Dealing with research integrity related issues requires not only a thorough understanding of the fundamental logics of scholarly work, but of the multiple reasons why misconduct is happening or why errors occur. Thus it is equally important to understand how fabrications, falsifications or plagiarism happen as it is to initiate discussion and exchange inside the scholarly community to raise awareness and find appropriate solutions of how errors may be prevented.

    Please watch the latest HEADT Centre video for an introduction to the research integrity initiative at Humboldt Universität zu Berlin and learn about how we plan to make our contribution to preserving excellence in science.


    Dealing with research integrity related issues requires not only a thorough understanding of the fundamental logics of scholarly work, but of the multiple reasons why misconduct is happening or why errors occur. Thus it is equally important to understand how fabrications, falsifications or plagiarism happen as it is to initiate discussion and exchange inside the scholarly community to raise awareness and find appropriate solutions of how errors may be prevented.
  • 29 Jan 2018 13:39 | Thorsten Beck (Administrator)

    The HEADT Centre held its biannual board meeting on 11 January 2018 in Berlin. This time the meeting took place at the Einstein Center Digital Future (ECDF) on Wilhelmstr. opposite the historic Reichstag building in Berlin. The HEADT Centre is a member of ECDF thanks to a new “Digital Berlin” professorship financed by both Elsevier and the city of Berlin. The HEADT Centre’s research integrity project, and its general focus on issues involving integrity and quality, complement a broad range of ECDF projects, as does the second more technical HEADT Centre project on Multi-Modal Similarity Search on Web-Scale Data.

    The Board reviewed progress during the last months with particular emphasis on the Image Integrity Database (IIDB), which serves as a joint effort of the two core projects. (See the separate blog post on this.) The IIDB has potential to become one of the most visible attributes of the HEADT Centre work, both because scholars need a testbed where they can learn how to use tools for detecting image manipulation, and because the IIDB demonstrates the range of problems involving scholarly images. At this point the focus of the IIDB will remain on content in the biological and medical sciences, but the possibility of future expansion remains open. IIDB staff will also help scholars, publishers, and institutions trying to determine whether manipulation took place.

    Another important initiative of the HEADT Centre are the seminars that are being offered internationally online to members of the iSchool group. The focus of these seminars is not the traditional about what not to do, but gives doctoral students and others a chance to ask questions about potential grey-area issues. As part of the Berlin School of Library and Information Science, the HEADT Centre will also offer a certificate programme on Digital Information Stewardship in cooperation with University College Dublin. Research integrity will be a significant element in the programme.

    Marketing played a role in the Board Meeting as well. The Board saw excerpts from promotional videos that were made with the help of Elsevier on research integrity topics. The videos are available on YouTube.

Powered by Wild Apricot Membership Software