Alchemy and Artificial Intelligence: What a 1965 Critique Still Teaches AI Builders

"Alchemy and Artificial Intelligence" is the title of a 1965 paper written by philosopher Hubert Dreyfus and published by the RAND Corporation. In it, Dreyfus compared the early AI research programme to medieval alchemy: a discipline producing occasional impressive results through trial and error, while lacking a theory that explained why those results occurred. The comparison stung then, and it resurfaced in 2017 when researcher Ali Rahimi used his NeurIPS Test of Time award talk to argue that machine learning had become "alchemy" — a field where practitioners stack techniques that work without understanding the mechanism. In modern engineering usage, alchemy in artificial intelligence means development driven by empirical tinkering rather than by explanatory theory, reproducible method, and measurable cause and effect. Understanding that distinction is one of the most practical things an AI team can do, because alchemical projects are the ones that fail silently in production.

Quick Answer: "Alchemy and artificial intelligence" refers to a 1965 RAND paper by Hubert Dreyfus criticising AI research for producing results without theory. Today the phrase describes AI built by trial and error rather than reproducible engineering — systems that work in demos but cannot be explained, audited, or reliably reproduced in production.

How WebPeak Turns Experimental AI Ideas Into Production-Ready Systems

The gap between an AI prototype that impresses in a meeting and an AI feature that survives real traffic is mostly engineering discipline: versioned data, deterministic evaluation, logging, fallbacks, and a deployment path. That is the exact territory WebPeak works in as a full-service digital agency, combining AI implementation with the backend and infrastructure work that keeps a model useful after launch. Their teams handle the parts that alchemical projects skip — structured data pipelines, API layers, evaluation harnesses, and monitoring — through AI implementation services and back-end web development. Working worldwide across AI, content, design, and development, they are typically brought in when a promising model needs to become a dependable product rather than a recurring experiment.

What Dreyfus Actually Argued in "Alchemy and Artificial Intelligence"

Dreyfus's central claim was not that machines could never be intelligent. It was that the symbolic AI of the 1960s rested on an unexamined assumption: that human intelligence consists of manipulating discrete symbols according to formal rules, and that scaling up rule systems would therefore produce general intelligence. He argued this assumption was philosophically unsupported, drawing on phenomenology to point out that much human competence is embodied, contextual, and never explicitly represented as rules — a chess master's board sense, a driver's feel for traffic.

The alchemy analogy was precise rather than merely insulting. Alchemists genuinely advanced chemistry: they developed distillation, discovered acids, and refined laboratory technique. What they lacked was atomic theory, so they could not distinguish a promising line of work from a hopeless one. Dreyfus argued that AI researchers were in the same position — real progress on narrow problems, extrapolated into confident predictions about general intelligence, with no theory capable of telling them which extrapolations were valid.

The paper was received badly and Dreyfus was widely dismissed. History gave him a partial vindication: symbolic AI's grand promises did stall, contributing to the funding contractions later called the AI winters. It also gave him a partial refutation, because embodied, statistical, and connectionist approaches — the very methods he thought were needed — eventually delivered results he had said rule-based systems never would.

Seven Signs Your AI Project Is Running on Alchemy

The alchemy critique becomes useful when you translate it into observable symptoms. In practice, teams that ship unreliable AI features tend to show several of the following at once:

1. Nobody can reproduce the good result. A model performed well once, on a machine, with data nobody snapshotted. If the run cannot be repeated from a commit hash, it is not a result — it is an anecdote.
2. Hyperparameters are folklore. Values are copied from a blog post or a previous project, with no record of what was tried or why the current setting won.
3. Evaluation is vibes-based. The team judges quality by reading a handful of outputs rather than scoring against a fixed, versioned test set with agreed metrics.
4. The test set leaked. Examples used for tuning also appear in evaluation, inflating scores in a way that only shows up after launch.
5. Improvements are unattributable. Three changes shipped together, the numbers improved, and nobody knows which change mattered — so nobody knows what to protect.
6. There is no failure taxonomy. The team tracks an aggregate accuracy number but cannot say what kinds of inputs fail, which is the only information that guides the next fix.
7. No fallback path exists. When the model returns nonsense or the provider times out, the product simply breaks rather than degrading to a deterministic default.

Each symptom has a cheap remedy — data versioning, an experiment log, a held-out benchmark, ablation testing, error clustering, and a circuit breaker. None require research talent. They require the decision to treat AI as engineering.

Alchemy Versus Engineering: How the Two Approaches Differ in Practice

The table below contrasts the two working styles across the dimensions that decide whether an AI feature survives contact with real users.

Dimension	Alchemical Approach	Engineering Approach	Business Consequence
Experiment tracking	Notes in a chat thread or nowhere	Versioned runs with config, data hash, and metrics	Determines whether a win can be repeated after staff turnover
Evaluation	Manual spot checks on recent outputs	Frozen benchmark plus task-specific metrics run on every change	Decides whether regressions are caught before users find them
Failure handling	Assumed rare, handled ad hoc	Documented failure modes with automatic fallbacks	Controls blast radius during an outage or degraded response
Explanation of results	"It works better with this setting"	Ablations isolating each contributing factor	Governs how confidently the system can be extended
Cost visibility	Discovered on the monthly invoice	Per-request token and latency budgets logged	Prevents margin erosion as usage scales

What the Record Actually Shows About Reproducibility in AI

The reproducibility concern is not speculative folklore; it has been formally raised inside the machine learning community itself. Rahimi's 2017 NeurIPS talk prompted a public debate with Yann LeCun, who argued that engineering practice often outpaces theory and that this is normal and healthy in a young field. Both positions contain something true, and the resolution matters for practitioners: theory can lag, but method cannot. Steam engines were built before thermodynamics existed — but they were built with measurements, tolerances, and repeatable procedures.

The research community responded institutionally rather than rhetorically. Major machine learning conferences introduced reproducibility checklists and code-submission policies, requiring authors to document datasets, compute budgets, and hyperparameter search procedures. That shift is the clearest available evidence that the alchemy criticism was taken seriously by the field's own gatekeepers.

From hands-on delivery work, a consistent pattern emerges that no published statistic captures as well as direct observation: the AI projects that stall in production almost never fail because the model was insufficiently sophisticated. They fail because the surrounding system was never specified. Teams that invest a week in a frozen evaluation set before touching model selection consistently ship faster overall, because every subsequent decision becomes a measurement instead of an argument. The same discipline applies to the broader AI implementation stack, an area covered well in this overview of artificial intelligence service delivery for organisations moving from pilot to platform.

There is one more lesson worth taking from actual alchemy. Alchemy did not end because someone proved it wrong — it ended because chemistry offered better tools for the same curiosity. AI's alchemical phase will end the same way: not through criticism, but through measurement infrastructure good enough that guessing stops being competitive.

Key Takeaways

• "Alchemy and Artificial Intelligence" is a real 1965 RAND paper by Hubert Dreyfus arguing that AI produced results without an explanatory theory.
• The critique returned to prominence in 2017 when Ali Rahimi described modern machine learning as alchemy in his NeurIPS Test of Time talk.
• Alchemy in AI practice shows up as unreproducible results, folklore hyperparameters, leaked test sets, and vibes-based evaluation.
• Theory can legitimately lag behind engineering, but method cannot — measurement and reproducibility are non-negotiable even without theory.
• Machine learning conferences adopting reproducibility checklists is concrete evidence the field accepted the criticism and acted on it.

Frequently Asked Questions

Who wrote the paper "Alchemy and Artificial Intelligence"?

Philosopher Hubert Dreyfus wrote it in 1965 while consulting for the RAND Corporation. He argued that early symbolic AI resembled alchemy because it produced isolated successes without a theory explaining them, and that intelligence depends on embodied, contextual skill rather than formal rule manipulation alone.

Is modern machine learning still considered alchemy?

Partly. Deep learning delivers reliable results whose internal mechanisms remain poorly understood, which is why the criticism persists. However, the field has adopted reproducibility checklists, standard benchmarks, and code-release requirements, so the practice is considerably more disciplined than it was when the charge resurfaced in 2017.

What is the difference between AI alchemy and AI engineering?

Alchemy tunes settings until something works and cannot explain why. Engineering versions the data, logs every experiment, evaluates against a frozen benchmark, isolates each change through ablation, and defines failure behaviour in advance. The output may look similar; only one of them survives scaling and staff turnover.

Why does reproducibility matter so much for AI projects?

Because without it, a good result cannot be defended, rebuilt, or improved. If a model's performance depends on an unrecorded data snapshot or an undocumented setting, the team owns an artefact rather than a capability, and any future change becomes a gamble with no reliable baseline to compare against.

Was Hubert Dreyfus proved right or wrong about AI?

Both, in different respects. His prediction that rule-based symbolic AI would not reach general intelligence broadly held, and that approach did stall. But statistical and connectionist methods later achieved tasks he considered out of reach, so his conclusions about limits were narrower than his critics assumed at the time.

Conclusion

The single decision that separates a durable AI system from an expensive experiment is made before any model is chosen: whether the team commits to a frozen evaluation set and a versioned experiment log. Everything else — architecture, provider, prompt strategy — becomes tractable once you can measure the effect of a change and reproduce it tomorrow. Dreyfus's warning was never that AI is impossible; it was that confidence without method is indistinguishable from luck. Start your next AI project by writing down how you will know it is working, and the alchemy problem largely solves itself.