ChatGPT - Yes or no?

Question

We have had our first post from the ChatGPT AI answering bot. Or at least, a post flagged by an AI detecting AI as 99% probable AI generated. The answer has not been deleted - yet, it isn't a very good answer but it doesn't flag up as NAA and nobody has flagged it VLQ.

There is some angst about AI-generated questions across SE at the moment. SO in particular has a ban on it...
Temporary policy: ChatGPT is banned
That post gives some good reasons for the ban. Various other sites are also banning it, and some have decided to cautiously accept it.

So, this is to solicit the community's opinion. Do we wish to ban answers detected as being AI generated? Do we want to treat them like any other answer?

With only one such answer so far, we aren't suffering any dilution of quality at the moment.

Answer 1

I'm opposed. Perhaps this is just a knee-jerk bladerunner anti-AI reaction on my part, but I want to see answers (and questions) from someone who has an actual interest in the subject, not just someone operating an automated internet tool to synthesize a response.

I understand that SE says that the content of Q+A is the thing that matters, but I don't think that's the entire truth for Retrocomputing, which owes a substantial part of its appeal to first-hand experience.

Answer 2

No.

My concern is mostly with fully automated responses, which of course given the costs of GPT vs. human writers, will soon prefer GPT over humans in the sheer volume of text that can be produced.

I have fewer concerns about a response that was partially synthesized by statistical completion, under human supervision. For example, taking the output from this prompt: "Give an introductory paragraph about the typical workflow using a mainframe computer in the 1960s" and reviewing it, and working it into a larger human-originating synthesis, about the shift to CRT terminals in the 1970s.

When I write, I often outline like that. Ah, I will need a paragraph here to develop this idea, and another one to develop that idea, and then I need to link them somehow... automating that is more of a writing aid than an AI written post, IMHO. But since it will be difficult to distinguish machine-aided vs. fully automated sources, until we've had some time to work through the intellectual and social and other implications of this technology, I will apply the precautionary principle and say, at least for now: no.

Answer 3

Dilution of quality isn't the only problem. For some of our questions, the answers are externally-verifiable. For example:

• How do I use the extra commands in the Spanish version of the 128k Spectrum ROM?
• Did any RISC CPU ever take more than one clock cycle per instruction?
• How much of the Program Segment Prefix area can be reused by programs with impunity?
• What was the first device to "beep" in lieu of tactile feedback to button presses?

For other questions, the answers are the sources:

• Why did the stock Amigas not have a battery for keeping the time/date?
• Why was 'echo on' chosen as the default setting within batch scripts in MS-DOS?
• Why did modems have speakers?
• Why Game controllers changed from righthanded to lefthanded?
• cmm's answer to Why plastic cases?

And some questions are in-between:

• Was OS/2 a viable alternative for daily DOS and Windows tasks?
• What was Sinclair's intention with the Microdrive?
• Why was PETSCII based on an obsolete version of ASCII?
• Why were single quotes ('…') chosen for characters, and double quotes ("…") for strings?
• Why did ScanDisk exist?

For the first kind of question, we can tell when answers are wrong, and it's not so bad. For the second and third kinds, an authoritatively-written answer could be taken as authoritative; for these, I think we should ban the use of ChatGPT with extreme prejudice. Which is better?

• No information is available online; or
• Information is available online, but it's wrong.

My proposal

• For questions about retrocomputing in the modern day, we could allow "AI-generated" answers for now.
• For questions about well-known, easily-verifiable history (e.g. "what date was this released?"), we could likewise allow these tools.
• For questions about obscure history, no.
• For questions about people's motivations, no.
• For questions where the AI might hallucinate a historical event, a lost artefact, or a quotation from somebody, no.

ChatGPT, and tools like it, are very good at making things up. This could easily lead to citogenesis. How would we know that this has occurred? It's not just the quality of our site that's at stake, here, but our ability to preserve knowledge at all.

Where Citations Come From:

Citogenesis Step #1

Through a convoluted process, a user's brain generates facts. These are typed into Wikipedia.
[[A guy with short hair sits at a desk, typing on a laptop.]]
Guy: (typing) The "scroll lock" key was designed by future Energy Secretary Steven Chu in a college project.

Step #2

A rushed writer checks Wikipedia for a summary of their subject.
[[A woman with a ponytail sits at a desk, typing on a desktop.]]
Woman: (typing) US Energy Secretary Steven Chu, (Nobel Prizewinner and creator of the ubiquitous "scroll lock" key) testified before Congress today...

Step #3

Surprised readers check Wikipedia, see the claim, and flag it for review. A passing editor finds the piece and adds it as a citation.
[[A man sits on a couch with a laptop in his lap, typing.]]
Man: Google is your friend, people. (typing) <ref>{{cite web|url=

Step #4

Now that other writers have a real source, they repeat the fact.
[[A flow chart, with "Wikipedia citation" in the center. The word "Wikipedia" is in black, the word "citations" is white with a red background. A black arrow leads from "brain" to "Wikipedia."
A black arrow labeled "words" leads from "Wikipedia" to "careless writers," and a red arrow labeled "citations" leads back to "Wikipedia citations."
A black & red arrow leads from "Wikipedia" to "cited facts" which leads to "slightly more careful writers," which leads to "more citations," which leads back to "Wikipedia" (all black & red arrows).]]
References proliferate, completing the citogenesis process.

Title text:

I just read a pop-science book by a respected author. One chapter, and much of the thesis, was based around wildly inaccurate data which traced back to ... Wikipedia. To encourage people to be on their toes, I'm not going to say what book or author.

Answer 4

GPT answers should be banned because that’s modern technology. Retro systems use MBR partitioning.

More seriously, though. Any policy that aims to combat misinformation should target it regardless of whether it was LLM-written or not, as a matter of both fairness and practicality:

• As for the former, it has been pointed out (especially on Stack Overflow) that LLM-written answers don’t create qualitatively different problems. The volume may be higher, but the problems remain fundamentally the same: answers that are incoherent, subtly wrong, misguided, or miss the point of the question entirely, posts that go ‘I don’t know the answer, but here’s something vaguely tangentially related just so I can scoop some Internet points from sympathy/A-for-effort upvotes’. And it’s actually those issues that are what makes LLM-written answers problematic, more than being LLM-written on its own; if an LLM always wrote perfectly accurate and relevant answers, it would be much less problematic, putting aside the question of plagiarism.

  I wouldn’t mind that much, for example, if someone were to use an LLM like a ghostwriter instead of a source of knowledge: that is, knowing an answer yourself, you could prompt the model ‘write an answer explaining that X, Y and Z’, refine the response and then post it; as opposed to asking the model for an answer and then copy-pasting it. It could be a great tool for getting around a writer’s block.

• As for the latter, we don’t really have a reliable way to tell whether an answer was written by ChatGPT or not. False positives are not unheard of (though so far evidence is mostly anecdotal: see https://meta.stackoverflow.com/q/422066/3840170 and https://www.aiweirdness.com/writing-like-a-robot/). If we take the output of a specific LLM-detection tool too seriously, we may end up flushing the baby out with the bathwater.

Even if a policy banning answers generated from language models makes sense as a matter of plagiarism, the lack of reliable methods of detection means it would amount to a symbolic declaration at best, and an excuse for witch-hunts at worst, mired in false positives and true positives claimed to be false positives.

As such, being LLM-written should generally not be a primary reason to delete an answer. So what interventions would I recommend against such answers? I can think of a few:

• Be more generous with downvotes, delete votes and ‘not an answer’ flags (and perhaps even ‘rude or abusive’ flags), and more strict with handling of the latter, especially watching out for:

  • Links that do not resolve to anything relevant
  • References to never-existent products and companies
  • Anachronisms
  • Long-winded, incoherent rambling
  • Posts that answer a different question from what was asked, even if they appear superficially related
  • Answers giving advice that is misguided or subtly (or not-so-subtly) wrong

  In particular: often when a post that misses the point of the question is flagged ‘not an answer’, moderators wash their hands off hiding behind the policy that ‘flags are not for technical inaccuracies’. Even though what was reported was not a mere technical inaccuracy, but a failure of basic reading comprehension. That needs to change.

• Be more stingy with upvotes. Review and fact-check before upvoting. (Even if it’s a human poster!) Don’t upvote answers just because they are long and nicely formatted.

• Raise the reputation threshold needed to upvote answers, and the penalty when receiving a downvote for an answer. This should reduce the impact of careless drive-by upvoting from HNQs and thwart attempts at reputation farming.

• Speak up: leave comments when an answer seems fishy. Demand verifiable sources for claims. Be sceptical (though maybe not entirely dismissive) of personal recollections.

These may be no panacea, but then, I don’t expect any such to appear soon.

Answer 5

Allow these answers only when clearly marked as such

It's possible (perhaps unlikely, but possible) that AI learning systems could pick up real information that's not known to the crowd here, and present it along with links to credible sources for future readers.

But it's equally likely for them to misunderstand or invent something and present as established fact, and be completely indistinguishable from the former case.

So machine-generated answers should clearly indicate how they were created, allowing readers to use their own judgement and follow up the references. Anything detected as generated but without the disclaimer should be summarily deleted.

If nothing else, it's clearly fraudulent to pass off someone else's work (or some machine's) as one's own, and that's not something I would like to encourage.

Answer 6

No ML (so-called “AI”) content

Besides the very good reasons listed in the SO ban, the output of these models is derived by a program running on a deterministic computer from its inputs, and therefore almost always illegal because it’s a combination of suspiciously and unlawfully sourced material, including material under conflicting copyleft and other licences.

Answer 7

I think it's by now clear it's about this answer.

Let me sumarize my reaction when I saw it:

• I thought of it as really bad
• Only mentioning common 'knowledge'
• Yes, 'knowledge' is quoted as all information presented are platitudes of the worst kind without any reflection
• It doesn't touch the subject of the question asking about 'why' at all
• It's more like a rant going astray around a keyword - ELIZA on speed.
• Or better just a random collection of statements without any inherent value.

I guess it's pretty clear I do not give that Answer any credit. So, why didn't I downvote or flag?

1. No downvote, as it's right in that soft area of so-so answers not mentioning anything wrong but not providing any content.
2. No flaging, as it's on that neither good nor explicit bad just deep in the clueless area
3. Being friendly to newbies. For such common keywords we often get new users that simply dump their 'knowledge', no matter if useful or not, and this was a new, low score account.
4. While I was close to write a comment about it, I did abstain, as it seemed like a very fruitless endeavour to argue with people writing texts like that. Not just because of the OP, but also because of possible secondary arguments about me being nasty.

Long story short, I simply decided to ignore it.

---

It does, like many other answers in such 'common knowledge' areas, repeat what 'must be true'. In the end it's exactly what AIs can do: provide non-contend, non-offensive and not-wrong texts - like the schmooze those side table magazines use to sell their paparazzi pictures.

And also exactly the reason why AI generated texts have no place on RC.SE. AI can not provide any information. They do not and will never. RC.SE is a medium intended to increase knowledge, for each single question and the site at whole.

There is no filter of collected information against knowledge or even basic logic. They only re-play what they 'hear' most about around a given topic, resulting in not in a scientific summary of that topic, but collected and averaged gossip. It's well known from process control that output of systems working on feedback of averaged values will always degrade.

This is the exact counter-thesis to what RC.SE is intended for. We do not want to average knowledge but collect the peaks of knowledge.

Bottom line: There is no place in RC.SE for any kind of automated answers

---

On a side note, it's a good example why it's important to have your personal information filled out - something AIs not really can provide.

Answer 8

I didn’t find an answer that I felt was the best place to put the following information, so I’m adding and answer. This information is quoted from the ChatGPT web site [emphasis mine]:

Limitations

• ChatGPT sometimes writes plausible-sounding but incorrect or nonsensical answers. Fixing this issue is challenging, as: (1) during RL training, there’s currently no source of truth; (2) training the model to be more cautious causes it to decline questions that it can answer correctly; and (3) supervised training misleads the model because the ideal answer depends on what the model knows, rather than what the human demonstrator knows.

• ChatGPT is sensitive to tweaks to the input phrasing or attempting the same prompt multiple times. For example, given one phrasing of a question, the model can claim to not know the answer, but given a slight rephrase, can answer correctly.

• The model is often excessively verbose and overuses certain phrases, such as restating that it’s a language model trained by OpenAI. These issues arise from biases in the training data (trainers prefer longer answers that look more comprehensive) and well-known over-optimization issues.

• Ideally, the model would ask clarifying questions when the user provided an ambiguous query. Instead, our current models usually guess what the user intended.

• While we’ve made efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions or exhibit biased behavior. We’re using the Moderation API to warn or block certain types of unsafe content, but we expect it to have some false negatives and positives for now. We’re eager to collect user feedback to aid our ongoing work to improve this system.

Answer 9

The question has been up now for a week and has attracted some good, well argued answers. So let me put up an answer that summarises these.

Incidentally, as I write this, the generated answer that started this off is still on the site, with 3 upvotes and 1 downvote. I deliberately haven't linked to it.

The consensus seems to be we don't want them. However, the feeling is not to delete any answers flagged as being AI generated just because they appear to be AI generated - can we trust the flagging mechanism any more than we trust the AI. Treat them like any other answer in terms of Very Low Quality and Not an Answer flags. Either of those flags will put the answer into the review queue and those with the privilege may Vote To Delete. Suspicion of AI generation may be taken into account by the reviewers.

We are not under the barrage of answers that some other sites are receiving. I believe we are not really suitable for the AI - at present. So The issue is not a major problem here.

I'll keep an eye on answers posted and this question can stay open to receive your views.