Summary
Integrating Large Language Models (LLMs) into the software development life cycle (SDLC) is the most fundamental shift in day-to-day coding since the arrival of the Integrated Development Environment (IDE). Unlike earlier tools that amplified human intent through deterministic automation, generative AI (GenAI) adds an inherently probabilistic layer to engineering and with it new kinds of epistemic and operational risk. The result is a multi-dimensional crisis: hallucinated logic and dependencies, weaker maintainability and the erosion of technical expertise. Developers perceive a speed-up when they use GenAI, yet the cognitive load of fixing subtle AI errors increases task completion times, which confirms this contradictory dynamic. Moreover, longitudinal analyses of more than 200 million lines of code show a 48% increase in code duplication ("copy/paste") and a sharp drop in refactoring, a shift from long-term maintainability towards short-term speed. In parallel, "vibe coding", where AI output is accepted without critical evaluation, creates an "illusion of competence", especially among junior developers, weakening the feedback loops needed to build skill and causing a knowledge erosion described as epistemic degradation in engineering teams. Finally, as AI-generated code flows back into these models as training data, recursive training is expected to degrade models over time and trigger "knowledge collapse", threatening software reliability. Taken together, these signals show that GenAI-assisted software engineering is a systemic challenge not only to productivity metrics but to the integrity and security of the software ecosystem and the sustainability of professional knowledge.
1. Introduction: From Deterministic Engineering to Probabilistic Prediction, and Methodological Hallucination
Software engineering has historically been built on precision, determinism and formal verification. Compilers and interpreters follow defined syntactic rules: code either runs or fails; there is no room designed for ambiguity. With AI entering code generation, that ground has started to shift. Engineers no longer evaluate the output of a deterministic tool but the suggestions of a stochastic generator working with statistical probabilities. This means that not only the generated code, but even methodologies, languages and practices for "writing correct code" are exposed to a hallucinatory process.
This emerging "methodological hallucination" means software development is increasingly built on fast but unverified AI output rather than on verified results. Developers increasingly confuse speed with correctness, and syntactically valid-looking output with secure, semantically correct code. As human oversight decreases, languages and methods drift, weakening their internal consistency and their connection to reality.
The literature examines this transformation through three critical risk vectors:
- Hallucination of code: AI producing logical inconsistencies, non-existent libraries, wrong API calls, or code that is syntactically valid but semantically wrong.
- Hallucination of process (vibe coding): Developers accepting code simply because it "feels like it works", bypassing verification and understanding; this weakens learning loops and creates an illusion of competence.
- Hallucination of knowledge (knowledge collapse): Models retrained on their own output drifting further from reality and producing confident but systematically wrong information.
Taken together, these three vectors show that GenAI-assisted software engineering reshapes not only operational practice but also its epistemic foundations.
2. Code Hallucinations: How and Why They Happen
The reliability problem in AI-generated code is rooted in the architecture of the models behind it. LLMs do not process programming languages through the formal semantic and syntactic rules a compiler or interpreter has; instead they learn token probabilities from huge volumes of human-written code. In other words, when an LLM writes code it does not prove logical consistency or correctness; it predicts the sequence of characters and words that looks most fitting for the context. Because of this probabilistic generation, models often produce constructs that are syntactically fluent but semantically wrong, logically inconsistent or even factually non-existent. This creates a systematic risk in software production known as hallucination.
2.1.1 Functional and Logical Hallucinations
Functional hallucinations occur when a model produces code that compiles and looks correct on the surface but violates the developer's intent, business rules or contextual requirements. These errors are especially critical because they often pass standard "happy path" tests, so the flaw is found late. For example, a model can generate a classic sorting algorithm with correct syntax but fail to handle null values, extreme integers or edge cases. The main reason is that the probabilistic weight of "standard" patterns dominant in the training data overshadows the subtle requirements of a specific context.
Research also shows that models make more mistakes when they try to produce changes (diffs) to an existing codebase than when they write new files from scratch. In this kind of task the model has to preserve the old state and correctly predict the new one. This dual reasoning requirement leads to errors where the model references variables that no longer exist, fails to import new dependencies, or produces functions incompatible with existing modules. These outputs also fall out of step with test coverage, because the model writes new code without "understanding" what the existing tests verify. This leads to regressions, silent failures and unexpected side effects outside the standard tests, confirming the concern that "the code does not go through enough testing".
2.1.2 Library and Dependency Hallucinations (Package Hallucinations)
The most dangerous and insidious form of hallucination is when AI models invent software dependencies outright. Based on patterns in their training data, LLMs tend to invent libraries, modules or API endpoints that look real but have never existed. This is a natural consequence of statistical completion, which leads models to combine familiar morphemes into "plausible" names. For example, a model trained heavily on pandas and scikit-learn might suggest a package called pandas-sklearn-integration as if it were real, simply because those tokens often appear together in natural language. Such invented dependencies look reasonable on the surface and can be adopted quickly by developers, which makes them particularly risky.
A comprehensive empirical analysis of 16 popular coding models showed that this behaviour is far more common than assumed. Across 576,000 generated code samples, roughly 20% of outputs referenced packages that do not exist. The researchers identified 205,474 unique hallucinated package names in total, and these were not just typos (typo-squatting) or variants of known packages but entirely new, invented dependencies. This is a serious security threat, because attackers can register these imaginary packages and use them to distribute malware through slopsquatting attacks.
2.1.3 Contextual and "Ghost" Hallucinations
A subtler type of hallucination is when the model introduces "ghost" constraints, assumptions or variables that don't actually exist in the code. This happens when the model "assumes" that a certain variable or infrastructure component exists. For example, it may use a user_id variable as if it were globally available even though the function never receives it as an argument. Similarly, it may hallucinate operational context missing from the developer's infrastructure, such as a particular AWS IAM role, a defined permission policy or a specific runtime environment. These errors are especially insidious because they look consistent on the surface but can cause silent failures or security vulnerabilities at runtime.
The root of this phenomenon is that LLM training data largely consists of code snippets that depend on invisible external context. Unstated environmental assumptions (global configuration, role-based authorisation, hidden dependencies, CI/CD pipelines or runtime environment variables) are common in training data. When the model cannot generalise this missing or implicit context, it produces fake context that looks internally consistent but has no counterpart in the developer's real system. The resulting "ghost assumptions" are among the critical forms of hallucination that threaten both functional consistency and operational reliability.
2.2 The "Confident" Failure Mechanism
The danger of these hallucinations grows with the high confidence LLMs show in their output. Human engineers usually signal uncertainty with comments, warnings or explicit markers ("// TODO: verify this library actually exists"); LLMs present the same uncertainty under a guise of certainty. Because they behave as if they are sure their code is correct, they produce hallucinated output in the same authoritative tone as correct output. This creates a serious trust trap, especially for less experienced developers, who may assume the model has a broader, more current or more contextual knowledge base than they do.
This mechanism contributes directly to what researchers call the Valley of Dangerous Competence. In this phase, the model produces code that is structurally sound, syntactically valid and superficially convincing, but operationally wrong. This "confident wrongness" is especially damaging in the early stages of model degradation or user over-reliance, because instead of obvious nonsense it produces output that looks almost right but contains critical errors. As a result, verifying the code requires more, not less, human expertise. Errors become harder to detect and embedded risks grow.
3. The Security Crisis: Slopsquatting
The intersection of package hallucinations and the open-source supply chain has produced a new attack vector that is drawing growing attention: slopsquatting. The term combines "slop" (low-quality or unverified AI output) and "squatting" (claiming a desired resource in advance). The technique relies on attackers exploiting the statistical distribution of package names LLMs are likely to hallucinate.
The core logic of slopsquatting: The attacker systematically identifies package names that appear often in model output but don't actually exist, typically derivatives of popular library names, compound morphemes or terms likely to appear together in natural language. They then register these imaginary names as real packages on open-source registries such as PyPI or npm. When developers blindly accept the hallucinated dependency, or an automated process resolves it, the attacker's malicious package gets installed.
This differs from traditional typosquatting in an important way: typosquatting relies on human error (spelling mistakes), whereas slopsquatting relies on model error, the structural hallucination dynamics of LLMs. In other words, attackers target the model's language statistics, not user mistakes. This moves the attack surface away from people with weak security hygiene and into the LLM-assisted development process itself.
Slopsquatting is a paradigm shift in security, because the threat no longer comes only from developers typing wrong package names but from LLMs automatically producing imaginary names that exactly match a malicious resource. This dramatically increases the risk of unintentionally pulling malicious payloads into CI/CD pipelines and production.
4. Declining Code Quality: Code Churn and Technical Debt
Beyond immediate security risks, the spread of AI code generation is reshaping the macroscopic quality of the world's codebases. Data from 2024 and 2025 shows a troubling trend: code volume is rising while code quality and maintainability are falling fast.
4.1 GitClear Findings: A Shift to "Write-Only" Code
A comprehensive GitClear analysis of 211 million lines of code changed between 2020 and 2024 provides empirical evidence of this decline. It highlights a move away from the "Don't Repeat Yourself" (DRY) principle towards high-volume, repetitive code.
- The rise of copy/paste: The share of changes classified as "copy/paste" rose from 8.3% in 2020 to 12.3% in 2024, a relative increase in duplication of about 50%. For the first time, the volume of copy-pasted code exceeded the volume of refactored code.
- The collapse of refactoring: "Moved" lines, which represent reuse and refactoring, dropped sharply from 24.1% to 9.5%. Developers are using AI to add new blocks of code rather than restructuring existing code to fit new features. The ease of generating new code reduces the perceived value of reusing existing logic.
- Rising churn: Churn, defined as code changed or deleted within two weeks of being written, nearly doubled from 3.1% to 5.7%. High churn is a classic sign of low-quality initial commits: developers accept AI code, push it, realise it is flawed or incompatible and then quickly patch it.
These numbers statistically confirm the concern that "the code does not go through enough testing". High churn shows that code is not tested enough when first written and that errors are only noticed once it is already in the codebase.
4.2 The Productivity Paradox
This drop in quality challenges the main value proposition of AI coding assistants: productivity. Developers often report feeling 20–30% faster, but objective measurements tell a different story.
A randomised controlled trial with experienced developers found that those using AI tools took 19% longer to complete tasks than those who did not. The main cause was time spent debugging and reviewing AI-generated code. The study revealed a striking mismatch: despite taking longer, the AI-assisted developers believed they had finished faster.
This "productivity paradox" arises because AI speeds up the typing stage of programming (rarely the bottleneck) while complicating understanding and debugging. Generated code is often verbose and lacks the intuitive structure a human would create, making it harder to read and maintain. As the codebase grows with AI-induced bloat, the cognitive load needed to understand the system rises, potentially slowing future development: a classic manifestation of technical debt.
5. The Human Element: Epistemic Degradation and "Vibe Coding"
Operational problems such as bugs and vulnerabilities are symptoms of a deeper, cognitive shift in software engineering. We are moving from "epistemic agency", where the developer understands why and how code works, to "epistemic dependency", where the developer acts as the auditor of a black-box process.
5.1 "Vibe Coding" and the Abstraction of Understanding
The term "vibe coding", popularised by Andrej Karpathy, describes a workflow in which the developer focuses on high-level intent (the "vibe") and hands implementation details entirely to the AI. Instead of reading code line by line, the developer relies on "run and see".
This can speed up prototyping ("weekend projects"), but it is destructive for professional software engineering. It encourages uncritical acceptance of AI suggestions without verification. This behaviour is fuelled by "automation bias", the psychological tendency to trust machine output over one's own judgement, especially when the machine communicates with high confidence.
5.2 The Illusion of Competence and the Dunning-Kruger Effect
AI tools create an "illusion of competence" in which junior developers feel they are performing at a senior level because they can produce working code for complex tasks. But this performance is fragile. Because they were never challenged during implementation, they lack the "tacit knowledge" needed to debug the system when the AI fails or edge cases appear.
This is dangerous because it masks the build-up of risk. A junior developer can successfully deploy a microservice using AI-generated code without realising they introduced a race condition or a dependency on a hallucinated package. The feedback loop traditionally provided by the compiler or a failing test is short-circuited by the AI's ability to supply "fixes" that patch the symptom instead of the root cause.
5.3 The "Death of the Junior Developer" Debate
The industry is currently wrestling with the idea of the "death of the junior developer". This doesn't mean companies will stop hiring juniors, but that the way juniors learn, through repeated practice on simple tasks, is being automated away.
Historically, junior developers learned by writing boilerplate, debugging simple errors and reading documentation. These "struggles" built the mental models needed for senior-level architectural thinking. If AI handles all the boilerplate and simple logic, juniors lose that training ground. They are "parachuted" into complex architectural problems without the fundamental skills to handle them. This raises a critical long-term question: if we don't train junior developers today, where will the senior developers of 2030 come from?
6. Knowledge Collapse: The Recursive Training Trap
The crisis extends beyond individual developers to the AI models themselves. As the web fills with AI-generated code (driven by the "write-only" trends GitClear describes), future generations of LLMs will inevitably be trained on this synthetic data. Here the "hallucination of knowledge" takes on its most literal and most destructive meaning.
6.1 The Three Stages of Knowledge Collapse
Research describes "model collapse" or "knowledge collapse", a degenerative process that occurs when models are trained recursively on their own output. It unfolds in three stages:
- Stage A (knowledge preservation): The model performs well, drawing on the human signal still present in its training data.
- Stage B (the valley of dangerous competence): The model produces highly fluent, confident but increasingly factually wrong output. This is the most dangerous phase, because the "feel" (the vibe) of correctness remains while epistemic integrity is lost. The model loses the nuance and diversity of human solutions and converges on "average" or cliché code patterns.
- Stage C (incoherence): The output becomes nonsensical or completely unresponsive to instructions, similar to a loss of entropy in the system.
6.2 Impact on Software Ecosystems
In software, knowledge collapse means the homogenisation of coding solutions. If models are trained on "average" code produced by earlier models, they will become less capable of producing new, optimised or domain-specific solutions. Codebases will converge on a "beige" average: functional but inefficient, bloated and lacking the creative spark that drives software innovation. And any vulnerability present in the "synthetic" training set (for example, a common SQL injection flaw) will be reinforced and embedded in the foundations of future coding tools.
7. Testing and Verification: The C.L.E.A.R. Imperative
Given the risks of hallucination, slopsquatting and declining quality, traditional code review and testing are no longer enough. Eyeballing a diff is useless against a package name that looks perfectly legitimate but doesn't exist.
7.1 Why Standard Code Review Fails
Standard code review assumes the code was written by a human with intent. Reviewers look for logic errors or style violations. Reviewing AI code, however, requires checking for non-existence and hidden context. One might assume a human author verified a library; the AI author (the model) did no such thing. This "accountability gap" lets hallucinations slip through.
The concern that "the code does not go through enough testing" becomes an operational reality here. Traditional (unit) tests test the functions the code contains; they don't test the reliability or the existence of the libraries it calls. So even when the tests are "green", the code can still carry a "hallucination" (a security vulnerability).
7.2 The C.L.E.A.R. Framework
To reduce these risks, industry experts propose dedicated frameworks for reviewing AI-generated code. The C.L.E.A.R. framework offers a structured approach suited to this new paradigm:
- C — Context establishment: Review the prompt and requirements before the code. Does the prompt ask for something that could trigger a hallucination (for example, "use a library for X")?
- L — Layered examination: Don't start by reading line by line. Look at the structure and architecture first. Does the overall approach make sense, or is it a complicated "Rube Goldberg machine" trying to satisfy constraints the AI didn't understand?
- E — Explicit verification: The most critical step is to verify that every imported package and external API actually exists. This is the primary defence against slopsquatting. Reviewers should treat every import as a potential security breach until verified.
- A — Alternative consideration: Ask "why this way?". AI usually picks the statistically most likely path, which may not be the most performant or secure one for the context.
- R — Refactoring recommendations: Treat AI code as a "draft". To counter the GitClear findings, the code almost always needs refactoring to reduce verbosity and improve maintainability.
7.3 Automated Defences
Manual review must be backed by automated tools.
- SCA (software composition analysis): Tools should be updated to flag unknown or newly registered packages. If a package in a PR is less than 30 days old, a high-severity alert should fire.
- Hallucination benchmarks: New benchmarks such as HalluCode are being developed so organisations can choose models that prioritise accuracy over creativity.
- Provenance tracking: Strict software bill of materials (SBOM) requirements ensure every component in the codebase has a traceable source, making it harder for hallucinated packages to persist.
8. Outlook and Strategic Recommendations
The AI revolution in software engineering is inevitable, but its current trajectory is dangerous. The industry needs to stop seeing AI as a "speed" tool and start treating it as a "capability" tool that requires rigorous containment.
With AI's progress, concerns about the "hallucination of knowledge" and "insufficient testing" are fully confirmed by current data. AI hallucinates not only code but the process that establishes whether code is correct.
Strategic recommendations:
- Policy: Enforce strict "No Vibe Coding" policies for production software. All AI-generated code must be isolated, reviewed and explicitly verified.
- Tooling: Deploy "firewalls" in CI/CD pipelines that block packages with low reputation scores or that were registered recently (for example, less than 30 days ago). This neutralises slopsquatting.
- Training: Reinvent engineering hiring and training around "AI auditing" skills. Junior developers should be trained not just to write code but to aggressively critique and verify AI output. This could turn the "death" of the traditional junior role into the birth of the "systems auditor".
- Epistemic responsibility: Reinforce a culture in engineering teams where, even if the AI is the "author" of the code, a human is the one "responsible" for it. "The AI wrote it" must be an unacceptable excuse when something goes wrong.
In conclusion, the future of software reliability depends on treating AI not as an oracle but as an untrusted, high-volume assistant whose work requires the highest level of scrutiny.
Sources used in this research:
- https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
- https://www.jonas.rs/2025/02/09/report-summary-gitclear-ai-code-quality-research-2025.html
- https://www.gitclear.com/ai_assistant_code_quality_2025_research
- https://www.captechu.edu/blog/ai-driven-threats-in-software-supply-chains
- https://www.mend.io/blog/the-hallucinated-package-attack-slopsquatting/
- https://nmn.gl/blog/ai-and-learning
- https://chatpaper.com/paper/186428
- https://cloud.google.com/discover/what-is-vibe-coding
- https://arxiv.org/html/2404.00971v1
- https://simonwillison.net/2024/Apr/1/diving-deeper-into-ai-package-hallucinations/
- https://www.kusari.dev/learning-center/slopsquatting
- https://cset.georgetown.edu/wp-content/uploads/CSET-Cybersecurity-Risks-of-AI-Generated-Code.pdf
- https://softwarearchitectureinsights.com/articles/the-new-reality-of-software-development-ai-impact-on-code-quality
- https://c3.unu.edu/blog/the-invisible-threat-in-your-code-editor-ais-package-hallucination-problem
- https://arxiv.org/pdf/2404.03502
- https://docs.vibe-coding-framework.com/best-practices/code-review-guidelines