This new video introduces to aspects of measurement when it comes to research integrity violations.
Here is the HEADT Centre’s new YouTube infomercial that introduces to aspects of plagiarism.
Elsevier and CWTS published a report on data sharing with the title “73% of Academics Say Access to Research Data Helps Them in Their Work; 34% Do Not Publish Their Data”. The article is available here.
Prof. Dr. Michael Seadle led a panel discussion on “Examining Research Integrity” at the International Symposium of Information Science (ISI) 2017 in Berlin. Among the panelists were Dr. Thorsten Beck (HEADT Centre), Prof. Dr. Gerhard Dannemann (Humboldt-Universität zu Berlin), Prof. Dr. Wolfram Horstmann (Niedersächsische Staats- und Universitätsbibliothek Göttingen), and Prof. Dr. Debora Weber-Wulff (Hochschule für Technik und Wirtschaft Berlin).
Photo - From left to right: Prof. Dr. Michael Seadle, Dr. Thorsten Beck, Prof. Dr. Gerhard Dannemann
Watch our new infotorial on image manipulation and research integrity:
ImageJ is a public domain image-processing program developed under the auspices of the United States National Institutes of Health and especially popular among biologists. It was first published in 1997 and is designed to support a large range of image processing and analysis tasks and the detection of image manipulations in many different types of images, such as „three-dimensional live-cell imaging to radiological image processing, multiple imaging system data (and) comparisons to automated hematology systems.“ (Wikipedia: https://en.wikipedia.org/wiki/ImageJ)
One of its advantages is that ImageJ is able to read many different image file formats, such as TIFF, PNG, JPEG, BMP and many more. The program supports operations like edge detection, contrast manipulation, the measuring of distances and angles, sharpening, smoothing, rotation and many other (for more details see https://imagej.nih.gov/ij) and it allows the processing of stacks.
Downloading and installing the program and the associated InspectJ plugin is fairly simple. Just follow the instructions on the ImageJ website or watch one of the many tutorials on Youtube, for example a series provided by the Zentrum für Molekulare Biologie at Universität Heidelberg:
All you then have to do is to drag and drop an image for analysis onto the ImageJ tool bar in order to start the InspectJ plugin and follow the instructions of the program. ImageJ asks the user to decide whether an image has a dark or light background and then automatically produces an analysis image for each of the three RGB color schemes (green, blue, red). The user is free to decide which one of the layers should be analyzed – or whether all of the layers should be included in the analysis.
Fig. 1: Analysis images for each of the three RGB color schemes (red, blue, green) are generated. In this example most of the information appears to be found in the red and green schemes (the analysis of the blue scheme does not generate any useful results).
Once a layer for analysis is chosen, the program starts to process a set of predefined actions and a video sequence can be watched, which may reveal some of the otherwise hidden features of an image (for example edges).
Example: Gel Electrophoresis Analysis
For this evaluation, I chose an electrophoresis image that I retrieved from Wikimedia Commons (https://commons.wikimedia.org/wiki/File:Gel_electrophoresis_2.jpg).
This is the original (still unaltered) image:
Fig. 2: Gel electrophoresis: 6 “DNA-tracks”. In the first row (left), DNA with known fragment sizes was used as a reference. Different bands indicate different fragment sizes (the smaller, the faster it travels, the lower it is in the image); different intensities indicate different concentrations (the brighter, the more DNA). DNA was made visible using ethidium bromide and ultraviolet light. Author: Mnolf, Innsbruck, Austria, 2006
Creative Commons Attribution-ShareAlike 3.0 Unported https://creativecommons.org/licenses/by-sa/3.0/
For the purpose of testing I decided to copy and paste some areas of the bands and to erase some unwanted areas:
Fig. 3: Manipulated electrophoresis image.
Figure 4 specifies the details of the manipulations:
Fig. 4: Copied and pasted areas (rectangular shapes) and erased areas (oval shapes).
Here is a short video in which the subsequent steps of the analysis procedure are shown:
This is what the result of the analysis looks like once it is completed:
Fig. 5: Result of analysis (red color scheme): The ImageJ program with InspectJ plugin produces an analysis image with highlighted ROI’s (regions of interest), a log chart that presents the analysis results (in this case for example „identical areas“) and a ROI manager that allows to inspect each one of the results separately.
Discussion of results (Red Color Scheme):
The software specified regions of interest that are suspect of inappropriate image manipulation. The red color scheme analysis correctly reveals identical items at area 1 and 4 (with a 2-5 % tolerance setting) – which are actually those areas which were copied and moved, but it does not reveal the similarities between 1 and 3 or 3 and 4. Moreover, with a 10% tolerance setting it indicates a similarity between area 7 and 11, which was not part of the carried out manipulations.
Fig. 6: Result Analysis: Red Scheme
Fig. 7: Result Analysis: Green Scheme
Overview: Analysis results at different tolerance levels for the analysis of red and green color schemes:
Red Color Scheme Analysis:
Green Color Scheme Analysis:
Here is an example for how identical areas in an image are summarized in the log file:
Fig. 8: Log File for Red Scheme Analysis: Identical Items found at 1 & 4; 7 & 11.
This evaluation revealed how ImageJ helps with the identification of identical regions within an image. Still, some areas that have been highlighted in the process of analysis do not appear to be identical, but the program has found and indicated most of the copied regions – which might be due to inappropriate tolerance values.
It should be mentioned that the results of the green and red scheme analysis are not fully identical and that both of them have to be evaluated in order to derive a correct conclusion. Moreover, processing the JPEG image file with the InspectJ plugin indicated identical regions, but did not identify the erased areas (the oval forms, see Fig. 4). In one of the next blog posts, I am going to test the ImageJ solution for detecting erased areas.
More tools will be evaluated soon – please visit HEADT.EU for upcoming posts.
© HEADT CENTRE 2017
We set up a YouTube channel for the HEADT Centre where we make videos available that we produce to introduce our research as well as to share our research progress. We published our very first video yesterday: An Introduction to Research Integrity.
Our team working on Research Integrity at the HEADT Centre is currently producing short videos to introduce the core concepts that play a role in this field of research. Michael Seadle, Thorsten Beck, and I visited the recording studio at the Erwin-Schrödinger-Zentrum in Berlin-Adlershof, which belongs to Humboldt-Universität zu Berlin (HU). Both HU’s natural science library and the Computer and Media Service (CMS) are located there.
Michael Seadle (left) and Melanie Rügenhagen (right) at the recording studio of HU's Erwin-Schrödinger-Zentrum.
The CMS has professional staff dedicated to assist with the entire recording procedure that is done in addition to providing voice. They helped us a lot in getting the sound right, detecting reading mistakes and redo passages that sounded unclear. We spent about half a day reading the scripts we had written, edited and proofread in advance. The first results should be ready soon, and we will publish the videos on our website along with a note on this blog.
The US Office of Research Integrity (ORI) in the United States has been occupied with research integrity related issues for well over two decades. The office was established in 1992 by consolidating the „Office of Scientific Integrity“ and the „Office of Scientific Integrity Review.“ Today the Office of Research Integrity oversees many of the „Public Health Service (PHS) research integrity activities on behalf of the Secretary of Health and Human Services.“
For more information on the history and agenda of ORI, please visit: https://ori.hhs.gov/
Among its many aims are the development of “policies, procedures and regulations related to the detection, investigation, and prevention of research misconduct and the responsible conduct of research.” Over the years the office developed a wide spectrum of activities and services, like interactive videos, case studies, web modules, and many other forms of outreach and tools.
One of the resources the office provides for the wider academic audience are the so-called “Forensic Tools” for image analysis and manipulation detection, which provide customized standard operations and recordings of standard routines in Adobe Photoshop. (https://ori.hhs.gov/forensic-tools)
Forensic Droplets and Actions
ORI started with the development of such tools for Photoshop CS2-CS3, which leads to the fact that many of these routines can not be assessed with later versions of Photoshop. The office offers droplets and actions: droplets are „small desktop applications (…) that automatically process image files that are dragged onto their icon” whereas an action “is the sequence of steps that was pre-recorded in Photoshop” and is started from inside the program.
The “Forensic Tools” package comprises routines for the enhancement of certain image qualities and features, like droplets for the lighting up of areas or the adjustment of gradients. Other droplets allow for the comparing of two images, or provide overlays for images with dark or light backgrounds. Basically, the aim is to enhance features to support visibility, so that hidden details in an image (like obscured background information, artifacts or edges from copy-paste operations) may be revealed.
Other than the droplets the actions for Photoshop are claimed to be compatible with later versions of the program. However, the office provides many operations – too many to deal with all of them in detail in this blog post – therefore I am first going to first focus on the “Advanced Gradient Map” and thereby demonstrate some of the underlying logics of working with such actions.
For the purpose of testing I composed two different forms of manipulations of an image of a wallpaper to demonstrate the capacities or limitations of such standard operations.
Fig. 1: The original image: Wallpaper with decorative floral design. Photo was taken at Mirow Castle, Mecklenburg, Western Pomerania with a Sony Alpha 6000 ILS Camera.
Manipulation 1: Copy-Move Forgery
Fig. 2: Image detail of the original image without manipulation.
Fig. 3: Details obscured through background cloning.
Fig. 4: Copying and moving floral element with magic wand tool.
Fig. 5: Resulting manipulated image with copied floral elements (highlighted).
Manipulation 2: Cut & Paste Manipulation
The second manipulation is less subtle and easier to spot even with the naked eye.
Fig. 6: Copied and pasted area (highlighted).
Fig. 7: Detail of copied and pasted area. The cutting edge is clearly visible even without further enhancements.
Image Analysis with Forensic Photoshop Action: Advanced Gradient Map
Processing an image with the Advanced Gradient Map action simply requires to drag and drop the image onto the action item. This automatically initiates a number of steps: first the color image is reduced to grey scales, then the user is free to manually adjust curves and finally the action offers a set of color tables that may be helpful for further enhancing contrasts.
Analysis: Manipulation 1
Here are some screenshots I took while executing the action:
Fig. 8: Adjusting curves.
Fig. 9: Choosing a suitable color scheme for contrast enhancement
This is how the resulting image looks like:
Fig. 10: Result of Analysis
Result: The Advanced Gradient Map action allows for new perspectives when analyzing image content. Still, the action did not help to reveal the subtle changes that were carried out with the clone stamp and the magic wand – they simply remain invisible. Note: the result could have looked completely different with another color scheme. There is not just one option.
Fig. 11: Detail from resulting image: No clear evidence of a manipulation.
Analysis: Manipulation 2
Now let’s have a look at the second manipulated image in which the pasted region appears to be easier detectable. Again, the analysis is carried out applying the automatized steps of the action, but with different curve adjustments and color scheme. This is how the resulting image looks like:
Fig. 12: Advanced Gradient Map action: result with cut and paste manipulation
Clearly in this case the Advanced Gradient Map supports the identification of the manipulation. The edges around the pasted area are enhanced through the “light rainbow” effect and thus clearly visible.
Fig. 13: Detail from Advanced Gradient Map action
As has been shown in this quick evaluation the ORI action palette opens up a wide range of opportunities when it comes to image manipulation detection – respectively the visual examination of images or the visualization of hidden features. However, it should be mentioned that the ORI customized the action palette for analyzing specific images, especially for Western blots and other images from the field of biomedicine. Nevertheless, the manipulations carried out on the wallpaper images above are in some way comparable to manipulations of blot images. Duplicating image areas or obscuring certain aspects are a serious concern in many biomedicine research integrity cases.
Other than the “Forensically” tools that we have discussed earlier on this blog Photoshop actions support the enhancement of certain features within an image, but can not be understood as a way of automatically detecting image manipulations. On the contrary, applying the actions requires a “trial and error”-attitude and the user has to invest time and effort to produce satisfying results. As it seems there are manipulations that cannot easily be traced with the Advanced Gradient Map action tool, but which require other tools – tools which will be discussed on this blog soon.
The HEADT Centre 2017
The Clone Detection Tool
Another tool available on the website “Forensically” (https://29a.ch/photo-forensics/#clone-detection) helps with the detection of clones. Like the Error Analysis Tool, which was discussed earlier on this blog, the Clone Detection Tool includes a set of levels that enable the user to identify manipulated areas in images.
Before discussing this tool, we need to have a closer look at another instrument – the “clone stamp” in Adobe Photoshop. This instrument makes it easy to significantly alter images in a way that the average viewer tends not to recognize. The clone stamp makes it possible to reproduce and to overwrite the “natural” texture of a certain area in an image. Detecting image manipulations in all kinds of images (e.g., artistic or scientific) not only requires a basic understanding of the tools that are designed for detection, but of the tools and practices that made the manipulation possible. In other words, whoever wants to detect manipulations must be able to produce them.
This is why I include three examples of how to work with the clone stamp:
Erasing Artifacts with the Clone Stamp in Adobe Photoshop
Fig. 1: Blossom and Bee
The idea of the first experiment is to make this picture a little more dramatic by erasing elements surrounding the blossom on the left and on the right.
Fig. 2: Schaffhausen Boat Scene
The second image shows a boat near the Rheinfall in Schaffhausen. I decided to let the boat disappear completely using the clone tool.
Fig. 3: Outdoor Scene with Shadows
The third image manipulation is a little more complicated, since it is not trivial to reproduce consistent color flows. The plan is to hide two of the shadows in the foreground (keeping only the shadow in the middle), and to erase the figure in the background.
Before I present and discuss the final results of the manipulations, here are some screenshots that document how I altered the images.
Overview of manipulations:
1_Blossom and Bee
Fig. 4: Erasing of background texture
Fig. 5: Reproducing flower petals.
2_Schaffhausen Boat Scene
Fig. 6: Cloning the water surface gives the impression as if the boat were sinking.
Fig. 7: Half of the boat vanished.
Fig. 8: Image with replaced shadows in the foreground.
Figs. 9-11: Cloning the background figure. Because the figure is located in front of different background textures (like the tree, the fence, the rocks and the path), it is crucial to copy information from each of these elements to produce a somewhat natural impression.
THE CLONE DETECTION TOOL
Now, here are the three images after the manipulations:
Fig. 12: The image has a little less background texture, and duplicating the petals highlights the blossom.
Fig. 13: The boat is gone and the manipulation not easy to trace.
Fig 14: A closer look makes it possible to see the manipulations in the foreground. The lighting on the right appears unnatural and there are visible traces of the clone tool. The erasing of the figure in the background seems less easy to trace.
Let us now see whether the Clone Detection Analysis Tool can reveal these manipulations. For the purpose of comparison, I analyzed both the cloned and the original image to give an impression of how the results differ.
Example 1: Blossom and Bee
Fig. 15: It turns out that the clone detection tool covers most of the manipulations. The tool highlights the duplicated/cloned flower petals and it reveals some other similarities in background structures (not all of them represent manipulated areas). It does not, however, highlight any of the rather drastic background manipulations (I erased parts of the background on the left and enlarged the dark area on the right – best compared with the original below).
Here is how the Clone Detection Tool analyzes the unaltered version of the image:
Fig 16: Although the tool traces some similarities, the difference between original and manipulated image remains clearly visible. Overall, the tool helps revealing many clones – except for those in the background texture.
Example 2: Schaffhausen Boat Scene
Fig. 17: In analyzing the second image, the analysis highlights certain areas of the image, but since the texture of the water surface is naturally repetitive, it is not easy to tell the manipulated from the untouched parts.
Fig 18: The comparison is useful in this case, because it shows that the manipulated areas are highlighted in different ways than other highlighted areas. One possible conclusion is: whenever the detection tool highlights a section strongly, this may be interpreted as evidence for cloned areas.
Example 3: Outdoor Scene with Shadows
Fig. 19: The last example shows some of the tool’s limitations. The tool introduces a paradox: it highlights areas in the image that have not been touched, and it does not identify areas that the human eye can easily identify as suspicious.
Fig. 20: Most of the textures and elements in the image are highly repetitive, which may be a reason why the algorithm does not detect the cloned areas. What is obvious: the clone detection tool does not work well in this case.
The help section of the “Forensically” website says: “The clone detector highlights copied regions within an image. These can be a good indicator that a picture has been manipulated.” (https://29a.ch/photo-forensics/#help) As the above experiments have shown, there are clear limitations to the capacities of the tool. Yet none of the tools on “Forensically” claim to reveal all manipulations under all circumstances. They rather promise to make it easier to identify where to look closer.
All in all, out of the three examples discussed in this blog post, there is only one in which the tool clearly highlighted repetitive areas (Example 1), another one in which the tool indicated areas within the image that the algorithm marked as suspicious (Example 2), and one in which the algorithm remained oblivious, if not misleading (Example 3).
More tools will be evaluated soon – visit headt.eu for upcoming posts.
HEADT CENTRE 2017