Did AI just predict Supreme Court decisions with 98% accuracy?

When I launched scotus-arguments.netlify.app, I suggested that AI could help readers follow the Supreme Court's oral arguments and get a sense of where the Justices were headed. Because my initial focus was on development of the web site and the current term, I didn't think through whether the same methodology applied to prior terms would correctly predict the outcomes. Shaken out of that complacency yesterday afternoon by the managing partner from a leading Houston law firm who found the site exciting, I decided to run an experiment last night. How did my analyzer do on the 2023, 2024 and 2025 terms? Answer: it correctly predicted the outcome in 167 out of 170 cases, a phenomenal 98% success rate. A full report of the predictions is in the file attached below. An appendix below discusses how well humans generally do in predicting the Supreme Court. Short version: usually less well, although the best humans have been getting close. A second appendix takes a closer look at some of the predictions in particular cases and how well they performed.

Numbers like these should make anyone suspicious, including the person who built the site. One explanation, however, is that the skill I wrote for the project (scotus-argument-analyzer), coupled with the advances in AI models has just made it an exceptionally good Court watcher. Or maybe Justices are now leaving more clues to connect. Since returning to the courtroom after the pandemic, the Court has kept a separate round in which, after the allotted argument time, each Justice in order of seniority may question counsel. A Justice with room to pursue a concern may reveal more than one competing as in years before to squeeze in a question. This explanation would make AI and my skill look good.

There is also a less glamorous possibility: the AI remembers what happened. Machine-learning researchers call this leakage, meaning that information about the answer has reached the system that is supposed to predict it. You might call it "cheating." Predicting last year's decisions is considerably easier when you have read last year's opinions! The AI learns how the cases came out and just neglects to tell us that little fact either because it is devious or because it lacks the self-awareness to do so. We lack the clean separation between prediction and answer that a forecasting experiment needs.

But this author finds it beyond his patience to wait until July 2027 for that determination. I want to look at the evidence now and see if there are any signs of leakage. And there is at least one intriguing piece of evidence suggesting that the AI is not cheating. Although my skill gets the ultimate result correct almost all the time, it does not do as well predicting the votes of the individual Justices. Would an AI that had secretly learned the outcome of the cases and incorporated it into its "predictions" make that kind of mistake? Of course, a particularly devious AI might do so, but that would require a kind of ego that we do not usually attribute to the non-human. There is another explanation. A model could remember that a particular side lost without remembering Alito's position. That information might be less prominent in its training data. It could then read the transcript, misjudge Alito, and still benefit from knowing the result. Leakage need not arrive as a complete copy of the opinion. Real analytical skill and leakage can coexist, and a capable forecaster with occasional access to the answers can look superhuman. The errors about individual Justices suggest that the AI did not wholesale cheat; they do not show that the outcome predictions were independent.

What this term will tell us

October Term 2026 gives us the test we actually want. The site is producing forecasts before the decisions come down, so the eventual opinions cannot contaminate them. Whatever accuracy the AI achieves this term will have to come from the prompt, the model, and the arguments themselves. Not to fear, I am preserving dated originals of every forecast and scoring the winner, the votes, and the reasoning separately. Genuine prospective accuracy anywhere near these historical scores would be a remarkable achievement. For now, the numbers justify excitement and investigation. I'll let you know in July how much of that excitement the AI has really earned.

Appendix: How human forecasters compare

We can compare the results of my AI with the results of humans in several ways. Professor Josh Blackman, who founded FantasySCOTUS, reports the crowd's accuracy each year: 83.05 percent for 2023, 76.36 percent for 2024, and 94.74 percent for 2025. FantasySCOTUS players can revise their predictions after argument, and the champions describe doing exactly that. The AI's advantage over the crowd is large in 2023 and 2024. In 2025, the humans nearly caught up.

The best individual players deserve more credit than the crowd average gives them. Kirill Muzyka, the 2024 FantasySCOTUS champion, earned 4,840 of 4,950 available points, about 97.8 percent. That is a contest score, not a case-outcome percentage, but it is enough to stop anyone from announcing that the AI has humiliated the best human predictor. Muzyka's account of his method will sound familiar: "I rely primarily on oral arguments." He listens especially for friendly questions that help an advocate, reasoning that a Justice is less likely to offer that kind of help to the side he ultimately opposes. Maybe my skill has just channeled Muzyka.

The research literature takes a longer view. Daniel Martin Katz and his colleagues tested ways of combining FantasySCOTUS forecasts from 2011 through 2016, and their best model reached 80.8 percent case accuracy. The recent crowd numbers, especially the 2025 season, show why a benchmark that old cannot stand in for today's competition. The AI's historical scores are higher still, but declaring a revolution requires knowing whether the AI faced the same uncertainty the humans did.

Kalshi, the prediction market, offers another comparison, though not a term-wide score. On February 17, 2026, three days before the tariff decision, Kalshi's own news service reported a 72 percent probability that the administration would lose and forecast three votes in Trump's favor. The market was right on both counts. That is one probability on one case, not an accuracy rate across a term, and on that case the market and the AI picked the same winner.

Appendix: What the AI got right and wrong

The AI's analysis of the tariff case picked the same winner, with useful detail. Learning Resources v. Trump asked whether the International Emergency Economic Powers Act, or IEEPA, lets the President impose tariffs. The AI predicted that "the tariffs are not authorized by IEEPA" and identified the six Justices who ultimately rejected the administration's interpretation, while treating Barrett as uncertain. Roberts's opinion concluded that "IEEPA does not authorize the President to impose tariffs." The forecast also expected the Court to rest on statutory interpretation rather than a standalone nondelegation holding. That is considerably more informative than a general prediction that Trump's lawyers were having a bad morning.

The less glamorous cases sometimes produced equally good matches. GEO Group v. Menocal asked whether a federal contractor could immediately appeal the rejection of its Yearsley defense, a defense that protects contractors who do what the federal government validly directed them to do. The forecast expected the Court to treat Yearsley as "a defense to liability rather than a right not to be tried." Kagan's opinion explained that "Yearsley operates as a defense to liability on the merits," so the contractor had to wait for final judgment before appealing. The AI found the distinction that decided the case, not merely the eventual loser.

But that same forecast shows why a correct outcome does not guarantee a correct coalition. It declared: "Justice Alito is the clearest vote for GEO." All nine Justices agreed with the judgment against GEO. Alito concurred in that judgment, although he disagreed with parts of Kagan's explanation. The principal result and the central doctrine were right, but the strongest apparent vote for the losing side never materialized. A scorer who awards a point for the winner and stops reading misses exactly the information that makes this experiment interesting.

Muzyka recognizes this problem too. Looking back on his 2025 season, he described Alito questioning criminal defendants aggressively and then joining unanimous decisions in Abouammo and Ellingburg. The expert listener and the AI face the same hazard: hostility toward an advocate can overstate a Justice's opposition to the advocate's result. That helps explain the AI's vote errors, but it tells us nothing about whether the AI already knew the outcomes. An AI that remembered how a case came out could still make this very human mistake about an individual Justice.

In Berk v. Choy, the AI did better at anticipating disagreement within the winning camp. The dispute concerned Delaware's requirement that a medical-malpractice complaint arrive with an expert's affidavit. The forecast expected the Federal Rules of Civil Procedure to displace that requirement in federal court, but noted that "Justice Jackson favored Rules 3 and 12 and doubted Rule 8," suggesting she might write separately. Jackson duly concurred in the judgment: "the relevant conflicts are with Federal Rules of Civil Procedure 3 and 12, not Rule 8." That is a satisfyingly specific match: the disagreement, the procedural rules, and the separate opinion.

Other correct winners concealed important mistakes. Landor v. Louisiana Department of Corrections and Public Safety concerned whether individual officers could be held liable under a federal religious-liberty statute. The forecast anticipated a ruling "which would leave the limits of Congress's power undecided." The Court instead addressed the constitutional limit. Gorsuch explained that "without independent regulatory authority, Congress must rely on consent," and the officers had not accepted the relevant funding conditions. The AI got the winner and the six-to-three coalition right while expecting the Court to reserve the very question its opinion answered. That is a useful prediction, but hardly a complete account of the law.

There were also outright misses. Postal Service v. Konan asked whether the Federal Tort Claims Act's postal exception barred a customer's suit over mail that postal workers intentionally failed to deliver. The AI wrote that "the evidence points weakly toward affirmance," expecting the customer's claims to fare better than they did. Thomas's opinion held that "the postal exception covers suits against the United States for the intentional nondelivery of mail." The Court vacated and remanded, five to four, leaving open which of the customer's claims and arguments remained available. Labeling the evidence weak made the forecast more cautious, but the forecast was still wrong.

Picking the winner and predicting each Justice's vote are different tasks. Picking the winner requires finding a sufficient majority. Predicting every Justice requires more. A six-to-three forecast can misplace one Justice on each side and still get the winner and the margin right. A forecast can also identify the right coalition while misjudging what its members will agree to say, as happened in Landor.