After two years of virtual conferences (2020, 2021), we are back in person in Porto, Portugal - with the talks of the European Conference on Academic Integrity and Plagiarism being broadcast so that people can attend at a distance. How wonderful it is to see people!! I can't help hugging people I've know for more than a decade and haven't seen in person since 2019 or earlier. And wow, I'm not the only person from Germany here, there is someone from the University of Constance. And wonderful technology: The European wireless network eduroam works seamlessly here! But there are no desks in the auditorium, that makes it a bit hard to organize one's work. I will report on the talks that I attended.
The book of abstracts and the final program are on the general conference web site. I was goint to post this during the conference, but I ended up with no free time and spent Saturday enjoying Porto and Sunday returning home (with sore calf muscles from all those steep streets). So I will try and get this out as soon as possible.
The conference was opened with a keynote speech by Daniele Fanelli, (Fellow, London School of Economics and Political Science) "Research integrity in a complex world". Among other publications, Daniele is the author of "How many scientists fabricate and falsify Research? A systematic review and Meta-Analysis of survey data" from 2009. He dove right into the question of complexity and how we can go about actually measuring complexity. In a nutshell, the more "moving parts" a topic has, the complexer it gets, and the more complex a system is, the more prone it is to questionable research practices (QRP). And the more QRPs, the more there is a possibility of research integrity problems. One of the big questions is the irreproducability crisis, which he tried to boil down to a mathematical formula that I think very few understood. He closed looking at various factors and concluding that we really don't know exactly, but his formula is an attempt to get a handle on it. It was a great start, as discussions of complexity ran throughout the discussions during the rest of the conference. In the first breakout session I attended 3 talks:
- Patrick Juola from Duquense University in Pittsburgh, PA, USA (and that is pronounced "do-cane", I come from that neck of the woods!) created a controlled test corpus for looking at text overlap by having 91 participants write two short texts: One on how to get from A to B on a map and one on how to make lemonade. He then used the Jaccard similarity coefficient to calculate how many words the text had in common pairwise: 0.0 means no words in common, 1.0 all words. The average similarity was only 0.21 +/- 0.07 for Map and 0.19 +/- 0.07 for Lemon (thus the name of the corpus, MapLemon). We giggled at one of the outliers that described making lemonade without using the words "lemon", "sugar", or "water": The author just wrote: Go to the store and buy it. This provides empirical evidence that even people writing on the same topic will use different words, because they are using their own voice. He want to replicate this for other languages and additional topics.
- Tutku Sultan Budak-Özalp from the Canakkale Onsekiz Mart University in Turkey looked at perceptions towards academic integrity in English as a Foreign Language teachers. She interviewed 25 Turkish EFL (we tend to call it ESL, English as a Second Language) instructors in an online survey with 39 questions. Unfortunately, she was presenting at a distance and we hadn't yet figured out how to make the PowerPoint slides larger (Zoom and PowerPoint don't play nicely together), so I was unable to read the results. She said in summary that the teachers are knowledgable about academic integrity, but there is not yet a nationwide policy, although the Higher Education Council in Turkey is working on one.
- Pegi Pavletić and other students from the European Students' Union presented about the work of students in the area of academic integrity. As Teddi Fishman, the chair of the session noted: There should be "no talks about us without us!" It was great to hear that students are getting active! The ESU is an organization of 45 student unions from 40 countries, in Ireland the National Academic Integrity Network (NAIN) has 13 student members among their 92 and has published guidelines, Croatia is dealing with a number of academic freedom issues. These students want to serve as ombuds, go-betweens between students and the various decision-making bodies. But there are so many different forms of decision making bodies at universities that the concept rather becomes an enigma.
Lunch was awesome, the medical school in Porto has a dining room with linens, cutlery, and wine glasses laid out. They had three buffets, salad, main course, and dessert, although without signs it was hard to determine what was vegetarian. The lentil salad turned out to have chicken in it, and there was ham mixed into the noodles, but one managed and the food was delicious. I turned down the wine, however, until the last day. Without a siesta after lunch it would be difficult to not fall asleep during the afternoon sessions, and I didn't want that.
- Zeenath Reza Khan, Sreejith Balasubramanian, and Ajrina Hysaj spoke on "Using the IEPAR Framework - a workshop to build a culture of integrity in higher education". They have started the Centre for Academic Integrity in the United Arab Republic. IEPAR stands for "Inspiration, Education, Pedagogical Consideration, Assessment Design, Response and Restorative Practice". They are focusing on prevention instead of policing and sanctioning.
- One of Thomas Lancaster's students, Pundao Lertratkosum, wrote their thesis on "Contract Cheating Marketing in Thailand". The question was whether or not there is marketing going on in non-English-speaking countries for contract cheating. The answer is a sad and resounding: yes! They looked at various social media and found lots of marketing, even videos that try to make it sound normal to use such services. The transactions themselves are then often conducted by messenger, the essays were asked for in Thai, English or Mandarin. The typical advertised price for a 1000-word essay in Thai: $80 to $140 US dollars. There are serious risks to students, who for example post receipts on Twitter, testamonials. They can be reverse-engineered and blackmailed. Thailand closely mirrors other countries with there being offers even for exam impersonation and admissions fraud. Even though this is a localized market (because most of the papers are written in Thai), we need to look to see how we deal with this normalization of cheating.
- The talk on "Contract Cheating in Lithuania" by Simona Vaškevičiūtė and Eglė Ozolinčiūtė was unfortunately quite short, as Zoom seemed to be misbehaving. We eventually solved the problem: when telling people to "share their slides" they would share only PowerPoint, not their screens. We would only see the last slide they had open, while they saw the current slide on their screens. Orally they spoke about visiting various web sites every day for one month and copying the advertisements found there. In particular they found that the advertising focusses visually on achievements with photos showing people reading, writing, or wearing a cap & gown.
- The session ended with a shocking talk by Anna Abalkina on "Publication and collaboration anomalies in academic papers originating from a paper mill: evidence from a Russia-based paper mill." She, too, had issues with Zoom, but we managed to get it sorted out. The organization "International Publisher LLC" is no longer just Russia-based, they have clients are all over the world. It published papers online, and then sells additional authorships, first author is more expensive than somewhere in the middle. She looked at 1000 paper offers and found 441 that had been published online. There were some 800 authors that could be identified from >300 universities in 39 countries. 152 of the journals appeard to be authentic and 3 so-called hijacked journals. In all, more than 6000 co-authorships slots appear to have been sold. She contacted many editors, only to be brushed off with statements like "We have strict peer review!" For an additional price, a city could be inserted into the abstract to localize the paper. Most of the authors were from Russia, but there were also authors from Kazakhstan, China, Ukraine, UAE, Azerbajan, Uzbekistan, UK, Israel, Vietnam, Egypt, Jordan, Spain, ..... Most of the purchased authorships were first authors. She calculates this to bring the company more than 6 million dollars in 3 years. So there are many problems in publishing that have not yet been discovered. The journals' (non)reactions to information about what appears to have happened is a serious challenge for academic integrity.
The next session I attended was more about technical tools for dealing with academic integrity questions.
- First off was Clare Johnson (working with Ross Davies and Mike Reddy) with her tool Clarify. It can be used to do forensic research on Word documents, as Word stores a lot of metadata in the saved version. There is information about formating, revisions, cropped images and sources and so on that are compressed, as it were, in the document. So this tool decompresses that and looks to see if it appears to have genuinely been written over a period of time, or if just one big copy & paste action put in all the text. She visualizes the text runs, i.e. the text that was inserted at one time. She showed us some examples. I really want to give this a test-drive, but have been unable to find it online. I have written to Clare to ask her if she can please let me have a copy of the tool.
- Evgeny Finogeev from the Russian company Antiplagiat spoke about "Image reuse detection in large-scale document scientific collection." It was made clear that this paper had been accepted before the war, and that sponsorship money that Antiplagiat had paid for the conference was being donated to a Ukrainian relief charity. They were not allowed to advertise at the conference, only present the academic paper. They took 1.9 million papers from the DOAJ, extracted the images, classified them, vectorized them and used a Siamese neural network to try and identify images that had been reused. The neural network identified 43 000 cases, 4051 of these were checked by hand. Most of them were self-reuse, very little correct re-use. Possible plagiarism was found in 8 cases, possible falsification in 11, there was permission to use the images in 4, 93 were paper copies, the rest were no problem. I objected to them using the "Lena" image from Playboy for their presentation, they did not seem to understand that we are trying to convince people to no longer use this image.
- Finally, Christopher Nitta (UC Davis) spoke on "Detecting Potential Academic Misconduct in Canvas Quizzes." The learning management system Canvas has an API that can be used, and a Python library (canvasapi) that makes use of the API. The problem is that the lock-down browser doesn't work with Linux, which many students use. [my solution is to devise exams that use the entire internet - after all, at work they can Google... -- dww]
Their solution tries to identify potential misconduct and highlights these exams for further analysis. How long did they spend away from the quiz? Of course, this could be dealing with a child at home. Formating from copy & pasting is preserved on Canvas, so web links and other formatting are tell-tale signs of misconduct. Large exam time windows permit answers to be shared with others, so the timing of the questions is analyzed for outliers. If students start within seconds of one another on all questions, this could be a sign of collusion. Of course, if they are using secondary devices, this cannot be seen. [See? So just quit with the proctored exams already! -- dww].
UC Davis had 1415 referrals to the academic integrity office in the 4 terms prior to the pandemic, but 3246 in the first 4 terms of the pandemic. They see this as evidence of increased cheating [I see this as evidence of increased looking! We don't know how much cheating was going on prior to this, only how much cheating we found. -- dww]
Almost 20 % of the cases were misconduct on Canvas quizzes. The manual review takes much more time. Of course, the false negative rates unknowable, and the false positive rate seems to also be muddled.
The code is not open source because they don't want the students to figure out how to get around the system. [I wish to remind them of Kerkhoffs's Principle. They are smart, they will figure it out, so make it public anyway! -- dww]
Posts
No comments:
Post a Comment
Please note that I moderate comments. Any comments that I consider unscientific will not be published.