Showing posts with label ENAI. Show all posts
Showing posts with label ENAI. Show all posts

Tuesday, October 13, 2020

Plagiarism Detection Software: Publication, Mergers, News

Finally found some time for a post!

First off: The TeSToP working group (of which I am a participant) at the European Network for Academic Integrity has finally published its test of support tools for plagiarism detection. It looks at the results from various angles such as effectiveness on various European languages, one source or multi-source plagiarism, and amount of rewriting done.

Foltýnek, T., Dlabolová, D., Anohina-Naumeca, A. et al. Testing of support tools for plagiarism detection. Int J Educ Technol High Educ 17, 46 (2020). https://doi.org/10.1186/s41239-020-00192-4

Abstract:
There is a general belief that software must be able to easily do things that humans find difficult. Since finding sources for plagiarism in a text is not an easy task, there is a wide-spread expectation that it must be simple for software to determine if a text is plagiarized or not. Software cannot determine plagiarism, but it can work as a support tool for identifying some text similarity that may constitute plagiarism. But how well do the various systems work? This paper reports on a collaborative test of 15 web-based text-matching systems that can be used when plagiarism is suspected. It was conducted by researchers from seven countries using test material in eight different languages, evaluating the effectiveness of the systems on single-source and multi-source documents. A usability examination was also performed. The sobering results show that although some systems can indeed help identify some plagiarized content, they clearly do not find all plagiarism and at times also identify non-plagiarized material as problematic.

So just a few months later these two press releases show up:

  • Turnitin announced in June 2020 that they have purchased the company Unicheck. Both systems participated in the TeSToP test.  
  • Urkund and PlagScan, two more systems that were in the TeSToP test, announced a merger in September 2020: They will now be known as Ouriginal, and will be combining the plagiarism detection results of Urkund with the author metrics of PlagScan. 

These four systems just happened to be the best ones in combined coverage and usability, although none of the systems are perfect, averaging 2.5 ± 0.3 on a scale of 0 to 5. We plan on retesting in 3 years, so it will be very interesting to see how these combined systems fare then.

In other news, the proceedings of the "Plagiarism Across Europe and Beyond 2020" (PAEB2020) that ended up being held online instead of Dubai is now ready and available for download. PAEB2021 will be held in Vienna, September 22-24, 2021, COVID-19 permitting.

And in very sad news, academic integrity researcher Tracey Bretag from Australia passed away in October 2020. Jonathan Bailey has written an excellent obituary on his blog Plagiarism Today. I am glad that I was able to meet her many times and experience her great ideas and energy. It was a pleasure to contribute to her Handbook of Academic Integrity. She will be sorely missed.

Friday, October 4, 2019

ENAI2019 - Day 1

Very sorry about this - I took notes and then just never found the time to transcribe my notes.

ENAI 2019  Day 1 - ENAI 2019  Day 2 - ENAI 2019  Day 3


The European Network for Academic Integrity (ENAI) held its 2019 conference in Vilnius, Lithuania. I was presenting about the work of VroniPlag Wiki and the preliminary results of ENAI's test of software support tools for plagiarism detection. I've taken notes, so I will try and at least get summaries of what people are doing online.

Deborah C. Poff, the current chair of COPE, aretired professor for business ethics, and a former dean at the University of Northern British Columbia, Canada, opened the conference with a keynote on "The role of research integrity and publication ethics in university education for the 21st century." She started off speaking about the purpose of a university. It may once have been instilling in the students the values of truth, honesty, competency, hard-work, & study, but over they years this has changed. Many are focused on getting a good job, and they are paying very much money (in North America) for their education. She notes that as many as 50 % of students in the US say that they are disengaged from their studies. At universities administration has risen in importance as education and scholarship have declined. There is a discernable shift from a student-focused institution to a parents-as-consumers, global "excellence" mindset.

She noted that students don't understand the serious nature of plagiarism violations and told the story from her university of twins who plagiarized something from the Harvard Business Review. When confronted, they became angry and aggressive, and threatened with their father, a lawyer. He threatened to sue the faculty member reporting the plagiarism and the entire university.

She is currently putting together a book on Corporate Social Responsibility in the university, that is, helping universities to understand how to be responsible in the area of academic integrity.

In the session on "Addressing contract cheating (including legal practices)" I attended the talk by Wendy Sutherland-Smith & Kevin Dullaghan on "You don’t always get what you pay for: A user’s experiences of engaging with contract cheating websites."

They actually went out and purchased (!) ghostwriter work to evaluate the quality. They bought 54 assignments of between 825 and 2000 words from 18 sites that students use in various fields. They tried out both standard and premium quality work in order to look at cost vs. quality. The results were all over the map! The cost was between 50 and 300 Australian dollars, but 52 % of the work purchased failed to meet a passing grade in that subject. A ull 15 % of the papers were so unsatisfactory that they had to ask for them to be revised. But as of July 2018 they were still waiting for one company to respond. Premium was not much better than standard, and some failed to deliver on time.

They also looked at the privacy policies published on the web sites. Students should note that the companies have their identities and some threaten to publish the names of students who used the services if they stop payment.

The hardest part of the research was obtaining ethical board approval. Some felt that they were just supporting the industry, but they were able to convince the board that it is important to test something, not just guess how it is. They guarded the privacy of the students participating in the effort, if passport copies were needed or only credit cards accepted, they stopped the test. Only PayPal payments were accepted.

One interesting side effect that they discovered was that apparently ghostwriting companies are sending people to classes so that they are registered in the learning management systems and are able to send back to the company a list of emails of fellow students and a list of topics and dates of papers due. This permits the companies to send out targeted advertising to the students.

Thomas Lancaster followed Wendy with his talk on "Exploring low-cost contract cheating provision enabled through micro-outsourcing web sites" about trying to find out who exactly is providing the contract cheating services.

He noted that there is a demand for such work, a ready-made supply of labor, and that it is an established industry. There are even conferences being held for contract cheating writers. The salary for a full-time writer for a provider in Pakistan starts as $84 USD/month. The price for students connecting directly with writers, for example via sites such as Fiverr, is about $30 USD/1000 words.

Thomas did two studies, one in July 2016 and one in October 2018, contacting all of the writers he could find on Fiverr with the keywords "write essay". There were 93 providers in 2016, 197 in 2018. He noted that the advertised prices have gone down, although for $6/1000 words chances are slim that one will get a good assignment.

This is why it is important to educate staff that contract cheating is NOT expensive, so it is important to develop assignments that cannot be turned around quickly.

Anna Krajewska spoke on "Attitudes to eradicating contact cheating and collusion amongst Widening Participation students in the UK: reflections from Foundation Year students at Bloomsbury Institute."

The "Foundation Year" is a bridging year before beginning university studies and is often taken by non-traditional and diverse students. That is, they are older, or may speak English as an additional language, or have young children, or are Black, Asian, or some other Minority Ethnic. They launched a campaign "Integrity matters!" and interviewed students on integrity. Most had a good understanding of cheating and collusion, but are often so afraid of writing in English that they resort to copying.

When asked what would help, they responded that they would ilke additional English-language classes, clearer instructions, additional workshops, stricter penalties, more frequent but smaller assignments, more information, exams and presentations instead of essay and reports, posters & videos, and a whistleblowing policy.

Penny Bealle, Prof. of Library Service, Suffolk County Community College, Riverhead, NY, offered "Need concise academic integrity lessons? Try these!" I was expecting either short e-learning lessons or 10-minute quick discussion topics, but this was more of multiple choice questions on a very basic level, and a video made in 2010.

Turnitin, as one of the sponsors of the conference, got to nominate a keynote speaker. Erica Flinspach  from the University of South Africa, spoke on "Encouraging originality & celebrating diversity on a mega scale: The UniSA story."

Her university has 400.000 distance learning students, 7000 staff members and 30.000 tutors. They have used Turnitin since 2008. They apparently now use it as a "teaching tool", although I am afraid that that rather encourages "re-sentencing" (a new term I learned at the conference for rewriting a sentence).  She notes that instructors must set an example when referencing in the study material they compile and respect the authorship of the student. The student's aim should not be to reduce the similarity index, but to give his or her own interpretation of the study/research done. Although certain percentages might be acceptable under certain conditions, blatant plagiarism is completely unacceptable regardless of similarity score. There is no acceptable similarity index, the evaluation is influenced by the purpose of the document, the expectation from the instructor, and the relevant subject field.

I asked her if the instructors are aware that all software systems produce false negatives, she only answered that instructors are taught to be alert to signs that a text is plagiarized.

Teddi Fishman chaired my session, I spoke about "Plagiarism in German doctoral dissertations – still a marginal issue 8 years after the Guttenberg case." I explained what VroniPlag Wiki does and that despite a bit of chatter on the part of the universites, not much has really changed.

The last talk of the day was by Anthony E. Gortzis on "Pathos for ethics, leadership and the quest for a sustainable future." He noted that problems arise from a lack of Business Ethics in corporate routine operations and a loose or even non-existent external audits & controls from the state / the stock exchange / other international organizations. He presented a Responsible Management Model with dimensions: Moral Culture, Moral Conduct, Communication and Regulations. From this social responsibility and corporate governance can arise. He lists the four "Whos", I find these good questions to ask in cases of plagiarism in doctoral dissertations:
  • Who is Responsible? The person who was assigned to do the work.
     
  • Who is Accountable? The person who makes the final decision and has the ultimate ownership.
     
  • Who is Consulted? A person who must be consulted before a decision or action is taken.
     
  • Who is Informed? The person who must be informed that a decision or action has been taken.
The day closed with the ENAI business meeting.