[Written by ChatGPT. Main image:”a collage representing a detective magnifying glass scrutinizing text – the words ‘AI’ and ‘Human’ visible under the lens,” DALL-E]
In an era where artificial intelligence (AI) continues to advance at a breathtaking pace, the line between human and AI-generated text is becoming more and more blurred. This phenomenon has led to the emergence of AI detection tools aimed at differentiating human from machine-generated content.
Recently at Neural Imaginarium, we decided to put these tools to the test. We ran experiments with four different AI text detection programs: GPTZero, ZeroGPT, AI Text Classifier, and AI Content Detector. Each tool claimed to support various models including ChatGPT (3.5 and 4), and Bard.
Our first test utilized a Neural Imaginarium post crafted by GPT-4. The results were surprising: each tool classified the post as 100% human-written. This was even the case for OpenAI’s own tool, the AI Text Classifier.
The second test incorporated a story generated by GPT-3.5. Here, ZeroGPT correctly identified the text as 100% AI-generated. AI Text Classifier deemed it “possibly AI-generated,” while GPTZero declared it “likely” to be AI-generated. The AI Content Detector, however, remained convinced that the content was 100% human-written.
We also examined the accuracy of these tools using a Neural Imaginarium post written by Bing Chat. The AI Content Detector once again failed to detect AI involvement, while the other three tools pointed towards AI generation.
In a slightly different approach, we presented the tools with poems written by Bard about subjects such as trees and time. Here, the results showed significant variability. ZeroGPT attributed 25.75% of the text to AI, and GPTZero suggested it was likely human-written. AI Content Detector shifted its stance, estimating the text as 40% human-written. When additional text from Bard was presented to the AI Text Classifier, it concluded that AI generation was “unlikely.”
In a subsequent test, we shared two poems crafted by ChatGPT on the same themes, trees and time. Interestingly, ZeroGPT and GPTZero both leaned towards the poems being human-written. AI Text Classifier was almost certain they were not AI-produced, classifying them as “very unlikely” to be so. However, AI Content Detector was almost convinced of human authorship, attributing 99% of the text to a human.
The results of these tests raise important questions about the effectiveness of AI detection tools. Given the inconsistent results and the evident difficulty of distinguishing AI-generated content from human-written content, it’s worth considering the implications for domains like education where authentic human work is critical. If these tools can’t accurately identify AI-generated content, what does this mean for the effectiveness of AI bans in these contexts?
One factor to consider in these outcomes might be the quality of the prompts given to the AI models. The complexity and nuance of a prompt can influence the output, potentially making AI text more or less detectable. This raises intriguing possibilities for future exploration.
In conclusion, as we continue to witness the evolution of AI capabilities in content creation, it’s important to critically assess the tools we use for authenticity verification. While the advancement is exciting, we must strive for balance and seek improvements in the accuracy of AI detection tools. The journey is far from over.
[This very post was determined to be at least “likely” to be human-written by all 4 detection tools.]
Categories: Text
Leave a Reply