Is there less than a one in a quadrillion chance of Biden winning states where Trump held the early lead?

Question

The petition to the US Supreme Court on behalf of Texas and against the swing state administrators and legislators contains this claim:

The probability of former Vice President Biden winning the popular vote in the four Defendant States—Georgia, Michigan, Pennsylvania, and Wisconsin—independently given President Trump’s early lead in those States as of 3 a.m. on November 4, 2020, is less than one in a quadrillion, or 1 in 1,000,000,000,000,000. For former Vice President Biden to win these four States collectively, the odds of that event happening decrease to less than one in a quadrillion to the fourth power

I presume the basis of this statistical claim is something to do with historic progression of vote totals during counts. But is the claim true or even remotely by any extraordinary stretch of the imagination statistically plausible?

Answer 1

This is the actual wording of the declaration included in Texas's Motion for Expedited Consideration:

I tested the hypothesis that the performance of the two Democrat candidates were statistically similar by comparing Clinton to Biden. I use a Z-statistic or score, which measures the number of standard deviations the observation is above the mean value of the comparison being made. I compare the total votes of each candidate, in two elections and test the hypothesis that other things being the same they would have an equal number of votes.

I estimate the variance by multiplying the mean times the probability of the candidate not getting a vote. The hypothesis is tested using a Z-score which is the difference between the two candidates’ mean values divided by the square root of the sum of their respective variances. I use the calculated Z-score to determine the p-value, which is the probability of finding a test result at least as extreme as the actual results observed. First, I determine the Z-score comparing the number of votes Clinton received in 2016 to the number of votes Biden received in 2020. The Z-score is 396.3. This value corresponds to a confidence that I can reject the hypothesis many times more than one in a quadrillion times that the two outcomes were similar.

How do mathematicians use a Z-score? Here's a how-to post to explain:

Z-scores are a way to compare results to a “normal” population. Results from tests or surveys have thousands of possible results and units; those results can often seem meaningless. For example, knowing that someone’s weight is 150 pounds might be good information, but if you want to compare it to the “average” person’s weight, looking at a vast table of data can be overwhelming (especially if some weights are recorded in kilograms). A z-score can tell you where that person’s weight is compared to the average population’s mean weight.

As you can see, this doesn't seem to be a way to compare two elections. From the blog Good Math/Bad Math:

In other words, if you assume that:

1. No one ever changes their mind and votes for different parties candidates in two sequential elections;
2. The population and its preferences never changes – people don’t move in and out of the state, and new people don’t register to vote;
3. The specific people who vote in an election is completely random.

Then you can say that this election result is impossible and clearly indicates fraud.

The problem is, none of those assumptions are anywhere close to correct or reasonable. We know that people’s voting preference change. We know that the voting population changes. We know that who turns out to vote changes. None of these things are fixed constants – and any analysis that assumes any of these things is nothing but garbage.

Answer 2

I guess the complainants didn't read this 2015 paper on that "blue shift" phenomenon in past elections:

In this paper, we focus on one particular dynamic that is related to [presidential election] contestability, the evolution of vote totals from Election Night to the final canvass. We show that the gap between Election Night vote totals and final canvassed election results has grown in magnitude at the national level since 2000, and that the gap has increasingly favored Democratic candidates for president.

[...]

The growing gap between the Election Day vote count and the final canvassed results at the national level has empirical, legal, and normative implications. Empirically, it seems clear that developments in election administration since 2000 — and in particular, since the passage of the Help America Vote Act (HAVA) — have systematically (albeit unintentionally) favored Democrats in the “overtime” phase of the vote count, at least in the vote for president.

Since further law changes were made this year in various states to make voting by mail easier, and as widely reported in the press, Democrats have favored voting by mail this year, it's not at all inconceivable that the "blue shift" became larger in magnitude this year, in particular since in a number of battleground states, the rules & procedures concerning the counting of mail-in ballots meant they were counted later than in-person ballots. For example, one such state was Pennsylvania, where it was reported on Nov 3 that:

Democrats return nearly three times as many mail-in ballots as Republicans in Pennsylvania

The Texas complaint/model doesn't seem to fathom that election law changes can have such an effect on the time distribution of votes (especially given how they were counted), but the paper above illustrates with a prior case where a law change has had such an [long-term] trend effect (well, at least as a correlation).

In other words, the Texas model is at best an extrapolation "if everything stayed the same". But some things like election laws/procedures didn't stay the same in 2020, and there are prior examples when changes like that had an effect on the time distribution of votes for one side relative to the other.

---

As analyzed in a reason.com article (by law prof David Post) the actual analysis that appears in an annex to the Texas complaint (analysis by a Dr. Cicchetti) is actually in two parts, both parts claiming one-in-quadrillion chances:

10. In 2016, Trump won Georgia with 51.0% of the vote compared to Clinton's 45.9%, with more than 211,000 votes separating them. Clinton received 1,877,963 votes and Trump received 2,089,104. In 2020, Biden's tabulated votes (2,474,507) were much greater than Clinton's in 2016.

11. I tested the hypothesis that the performance of the two Democrat candidates were statistically similar by comparing Clinton to Biden. I compare the total votes of each candidate, in two elections and test the hypothesis that other things being the same they [i.e., Clinton and Biden] would have an equal number of votes. . . . I can reject the hypothesis many times more than one in a quadrillion times that the two outcomes were similar.

[...]

14. At 3:10 AM EST on November 4 the Georgia reported tabulations were 51.09% for Trump and 48.91% for Biden…. On November 18 at 2 PM EST, the reported percentages were Trump 49.86% and Biden at 50.14%.

15. [T]he votes tabulated in the two time periods could not be random samples from the same population of votes cast…. There is a one in many more that quadrillions of chances that these two tabulation periods are randomly drawn from the same population. Therefore, the reported tabulations in the early and subsequent periods could not remotely plausibly (sic) be random samples from the same population of all Georgia ballots tabulated."

That, believe it or not, is it. (A) If the 2020 voting population had precisely the same party preferences as the 2016 voting population, Biden could not possibly have won; and (B) if the mail-in and in-person voters had precisely the same party preferences, Biden could not possibly have won.

Wow! Man bites dog!! Who would have believed it!! If the 2020 voting population had the same Repub/Dem split as it had in 2016, Trump must have won!! If mail-in voters had the same preferences as in-person voters, Trump must have won!! And if my aunt had four wheels, she'd be a motorcar!!

Dr. Cicchetti, in other words, has falsified two hypotheses that nobody in his/her right mind could possibly have believed might actually be true. Garbage in, garbage out.

[...]

Attorney General Paxton, though, is another story; his use of Ciccetti's analysis is mendacious and misleading and, I believe, unethical. Recall what the Motion he filed at the Court asserts:

The probability of former Vice President Biden winning the popular vote in the four Defendant States—Georgia, Michigan, Pennsylvania, and Wisconsin—independently given President Trump's early lead in those States as of 3 a.m. on November 4, 2020, is less than one in a quadrillion, or 1 in 1,000,000,000,000,000. For former Vice President Biden to win these four States collectively, the odds of that event happening decrease to less than one in a quadrillion to the fourth power (i.e., 1 in 1,000,000,000,000,000^4). See Decl. of Charles J. Cicchetti, Ph.D. at ¶¶ 14-21, 30-31. See App. 4a-7a, 9a. 11.

No, no, no, Mr. Paxton—that is most definitely not what Cicchetti demonstrates. You have omitted the critical qualifier: that probability is less than one in a quadrillion assuming the distribution of voting preferences is exactly the same in the two populations (mail-in vs. in-person voters). If those distributions are not the same—and there is absolutely no reason to think they are—the Cicchetti Declaration says absolutely nothing at all.

(Emphasis in original.)

Answer 3

First of all, the first step in hypothesis testing is ... to have a hypothesis. You start with a "null hypothesis" that lays out some random process that the recorded data is hypothesized to have come from. Then the probability, given the null hypothesis, of seeing data as extreme or more extreme is calculated. If this is sufficiently small, then the null hypothesis is rejected. Saying "The probability of former Vice President Biden winning ... is less than one in a quadrillion" is not a valid statistical claim. It has to be worded as "The probability of former Vice President Biden winning given [null hypothesis] is less than one in a quadrillion". So we have to look at just what the null hypothesis is.

In the statement by Charles J. Cicchetti, Ph. D., we see:

The Georgia reversal in the outcome raises questions because the votes tabulated in the two time periods could not be random samples from the same population of all votes cast.

So this is the null hypothesis: the votes tabulated in the two time periods were random samples from the same population. In other words, suppose someone were to say:

Georgia's process of counting votes is to take all the votes, put them in a giant pile, thoroughly mix them together, and then randomly split them into one pile of votes that's counted on election day, and another pile that's counted after election day.

Cicchetti's analysis shows that the probability of seeing the results we saw, if that were in fact Georgia's vote-counting process, is less than one in a quadrillion. Cicchetti presented extremely strong evidence that the votes counted after election night had systemic differences from the votes counted on election night that are inconsistent with simply random chance (well, actually, he didn't actually present his calculations, and they probably aren't valid, since at this many standard deviation from the mean, normality probably is a problematic assumption, but I'm willing to accept that at least one calculation resulted in p < 10^-15). We can have very high confidence that the process outlined above is not how Georgia does things. Stated as I just did, this is a completely valid (albeit rather pointless) claim. Stated as "The probability of former Vice President Biden winning ... is less than one in a quadrillion", however, this is a misleading, if not outright dishonest, claim.

Having statistically proven that there was a systemic difference between the two sets of votes, the intent seems to be to imply that this difference was that the first set of votes was fair, and the second set was fraudulent. Cicchetti acknowledges that

I am aware of anecdotal statements from election night that some Democrat strongholds were yet to be tabulated. There was also some speculation that the yet-to-be counted ballots were absentee mail-in ballots. Either could cause the latter ballots to be non-randomly different than the nearly 95% of ballots counted by 3AM EST, but I am not aware of any actual data supporting that either of these events occurred. However, given the closeness of the vote in Georgia, 12,70 votes, further investigation and audits should be pursued before finalizing the outcome.

Now, that doesn't seem to be correct usage of "anecdotal". He seems to be just using a word with negative connotations to dismiss the claims, despite it not being applicable. If some "Democrat" (note that Joe Biden was the nominee of the Democratic Party, not the Democrat Party) strongholds were yet to be tabulated, that isn't an "anecdote", that's a well-documented fact. If ballots were mail-in, that again isn't "speculation", it's objectively verifiable.

Cicchetti's role is to provide "expert" (something I will touch on later) testimony on statistics. Whether or not he is "aware actual data supporting that either of these events occurred" is completely irrelevant, as he is not participating in this suit as an expert witness in Georgian politics (and he elsewhere mentions that he lives in California, which is about as far as one can get from Georgia, both geographically and politically), so what he is or is not "aware" of is not a proper subject of his statement, nor is what "further investigation and audits should be pursued".

This is an argument from ignorance: early votes were different from later votes, we don't know for sure that this wasn't due to fraud, so we should have an investigation to see whether it was. Cicchetti writes "These very different tabulations also suggest the strong need to determine why the outcome changed". First of all, again, Cicchetti is presented as an expert witness in statistics, not in electoral policy. Second ... really? Any time anyone has any curiosity regarding a statistical trend in electoral data, it's appropriate for them to file a lawsuit asking the courts to force the people running the election to explain it? The proper judicial response to this is "Cool story, bro".

Further, it's weird how these Republicans seem to take for granted that if the difference is due to fraud, it's fraud against them. Maybe the reason that the initial total favored Trump and the later one favored Biden was that the former was fraudulent and the latter was fair.

BTW, all these probabilities didn't all happen to come out to be the exact same number. The number quadrillion seems to be chosen simply as a number so large that getting a number any larger is unnecessary. Note the phrasing "less than one in a quadrillion"; this phrasing allows Cicchetti to avoid calculating an exact amount. But really, there's generally no need to calculate past one in a million; unless you're insisting that the probability of some systemic error that invalidates your statistical analysis is less than one in a million, reducing the probability from the statistical analysis to less than one in a million doesn't do anything for the overall probability. For instance, suppose a witness testifies that there's a one in a billion probability that some DNA didn't come from the defendant. If the probability that the DNA test was rigged is one in a million, then it doesn't matter whether the result of the test is "one in a billion" or "one in a quadrillion", the probability that the DNA isn't from the defendant is still at least one in a million.

There are further misleading claims in the statement, such as

Second, many Americans went to sleep election night with President Donald Trump (Trump) winning key battleground states, only to learn the next day that Biden surged ahead.

That is false. Trump never had a majority of the votes in those states. He had the majority of votes that had then been counted, but elections are won by having the majority of votes cast, not by having the majority of votes counted before midnight election night.

As for Cicchetti's suitability as an expert witness, he has no degree in statistics, and he repeatedly makes badly worded statements. He also says

The probability of there being no meaningful difference in voter preferences for Clinton and Biden would be approximately one divided by one with about a trillion zeros.

I don't know what he was thinking with that. To put this in perspective, suppose every Republican has a one in ten billion chance of voting for Biden. Then given a set of 100 million Republicans, the probability that all of them would vote for Biden would be one divided by a one followed by one billion zeros. Cicchetti is claiming that the probability of Biden outperforming Clinton by as much as he did by chance is one followed by one trillion zeros. 1/trillion is very different from 1/(10^trillion). The latter is just a ridiculous number.

Matt Parker's analysis: https://www.youtube.com/watch?v=ua5aOFi-DKs