Show HN: Slop or not – can you tell AI writing from human in everyday contexts?
TL;DR
A developer built a crowdsourced AI detection benchmark: two responses to the same prompt — one human (pre-2022), one AI — and you pick the slop. Three wrong answers and you're out.
Key Points
- The dataset covers 16,000 human posts from Reddit, Hacker News, and Yelp, each paired with AI generations from 6 models across Anthropic and OpenAI at three capability tiers.
- Early findings: Reddit posts are easy to spot — humans write too casually for AI to mimic convincingly. HN posts are significantly harder to distinguish.
- Every vote is logged with model, tier, source, response time, and position. Full dataset planned for HuggingFace, a paper to follow.
Nauti's Take
The methodology is solid: pre-2022 data, no adversarial coaching, length-matched, real platform contexts — that's more scientific rigor than most commercial detection tools offer. The implication is striking: if even tech-savvy HN users struggle to spot AI text, then 'just ask humans' is no longer a reliable safeguard.
Whether the paper materializes depends on crowdsourced participation, but the dataset alone should be valuable for researchers.
Context
Most AI detection tools rely on classifiers that are easy to fool and rarely explain why a text is flagged as AI-generated. This project flips that: human judgment as the foundation for a benchmark. The fact that HN posts are harder to spot than Reddit ones suggests AI is already highly convincing in formal, factual contexts — precisely where it matters most for misinformation or manipulation risks.
Sources