'AI Hallucinations' Have No Place in Court: Supreme Court Sets Aside NCLT Order Over Fake Case Laws

By Mahima Katal

New Delhi: In a landmark ruling that could shape the future use of artificial intelligence in India's legal system, the Supreme Court has held that courts and tribunals must adopt "zero tolerance" towards AI-generated fake or hallucinated judicial precedents. Setting aside an order of the National Company Law Tribunal (NCLT), the Court found that the tribunal had relied on non-existent case laws and fabricated passages that appeared to have been generated by artificial intelligence.

Calling the use of such material "catastrophic" for the judicial process, a Bench of Justices Pamidighantam Sri Narasimha and Alok Aradhe observed that judgments based on fake precedents undermine the rule of law and cannot be sustained.

Why did the Supreme Court intervene?

The case arose from insolvency proceedings involving Essel Infraprojects Ltd., where the NCLT admitted a plea under Section 7 of the Insolvency and Bankruptcy Code. During the appeal, senior advocate Madhavi Divan pointed out that several Supreme Court judgments relied upon by the NCLT either did not exist or contained fabricated extracts that could not be traced to the actual reported decisions.

After independently examining the citations, the Supreme Court found that some judgments cited by the tribunal were entirely fictitious, while others referred to genuine cases but quoted paragraphs that had never been part of those judgments. It also noted that these errors escaped scrutiny before the National Company Law Appellate Tribunal (NCLAT), which affirmed the NCLT's decision.

Holding that the adjudicatory process had been "tainted" by reliance on fake precedents, the Court set aside both the NCLT and NCLAT orders and remitted the matter back for fresh consideration.

What are AI 'hallucinations' and why are they a concern for courts?

Artificial intelligence tools based on large language models are capable of generating convincing text, including legal arguments and case citations. However, they are also known to produce "hallucinations"—responses that appear authentic but are factually incorrect or entirely fabricated.

The Supreme Court compared the entry of such fake legal material into judicial decision-making to the release of methyl isocyanate, describing it as invisible, insidious and catastrophic until the damage becomes apparent. According to the Bench, fake precedents contaminate the very foundation of judicial reasoning because courts depend on verified legal authorities while interpreting the law.

The Court clarified that its concern was not with artificial intelligence as a technology, but with the unverified use of AI-generated material in legal proceedings.

The Court draws a line between using AI and relying on it

Significantly, the judgment does not discourage the use of artificial intelligence in the justice system. Instead, it recognises AI's potential to improve efficiency and assist judges and lawyers in research and administration.

The Bench acknowledged that AI is transforming professional work and observed that courts have historically embraced technology to improve access to justice. However, it stressed that adjudication must always remain under "human control", with judges independently verifying every legal proposition before relying upon it. AI, the Court said, may assist judicial work but cannot replace judicial reasoning or decision-making.

A warning for both lawyers and judges

One of the most significant aspects of the judgment is that responsibility has been placed on both the Bar and the Bench.

The Court held that it amounts to professional misconduct for advocates to cite AI-generated precedents without verification. Equally, judges and tribunals cannot rely on such material without independently confirming its authenticity.

The Bench declared that if even "an iota" of fake or hallucinated material enters the decision-making process, the resulting judgment cannot be allowed to stand because it compromises the integrity of adjudication.

Bar Council directed to frame safeguards

Recognising that judicial warnings alone may not be sufficient, the Supreme Court directed the Bar Council of India to constitute a committee to examine the growing use of AI-generated legal material.

The committee has been asked to develop guiding principles for advocates and recommend disciplinary measures against lawyers who knowingly or negligently submit fabricated AI-generated precedents before courts. The direction signals that professional standards governing the use of AI in legal practice may soon become more formalised.

A precedent for the AI era

The judgment is among the clearest statements by the Supreme Court on the role of artificial intelligence in the administration of justice. Rather than rejecting AI altogether, the Court has endorsed its responsible use while making it clear that technology cannot replace judicial scrutiny.

As AI tools become increasingly common in legal research and drafting, the ruling establishes an important principle: efficiency cannot come at the cost of authenticity. In the Court's view, every legal authority cited before a court must be verified by humans because the credibility of the justice system ultimately rests on genuine precedents—not algorithmically generated fiction.