Date: Mon, 24 Jan 2000 21:23:01 -0800
To: rdnelson@Princeton.EDU
From: Dean Radin <radin@boundaryinstitute.org>
Subject: 228 billion scrambled eggs 
Cc: may@lfr.org, james@jsasoc.com, shoup@interval.com

Attached are the latest results of my GCP explorations.   The peak here for Y2K
is -2 seconds before midnight.  Also attached is a similar analysis applied to
Y1999 midnight (this is slightly different than the Y2K analysis, and the
results aren't anywhere near as good).

The analytical steps are as follow:

1) Download 24 hrs of raw egg data starting from 12-31-99 10:00 AM.

2) Correct the two half-hour-off Indian eggs by shifting them forward a
half-hour.

3) Take the per-second variance across all raw egg values.  I.e., create one
variance score, per second, across all 28 eggs.  You now have a long vector of
variances, of length = 3600 secs x 24 hours.

4) Create an epoch matrix 24 columns wide (hours) by 3600 rows deep (seconds),
where second # 1800 = midnight in each time zone.  I start from -13 GMT, the
Fiji egg, and go to +10 GMT, to cover 24 hrs.

5) For each of the 3600 rows, you calculate the value EK = exp(-kurt(across all
24 time zones)), where exp is exponential and kurt is kurtosis. 

6) Apply a sliding median window of width 300 seconds against these 3600 EK
values.  This will result in a vector of 3330 median-ized EK elements.  Call
this MEK.

7) Find the overall mean of the MEK values.  Now find the standard error for
each MEK value as SE = stdev(EK)/sqrt(300), where EK is the same window of 300
seconds as used in step 6.  Now create a Z equivalent as (MEK - meanMEK)/SE. 
The graphs are based on these Zs and their equivalent odds scores.  I plot odds
as 1/p rather than (1-p)/p because I'm using 2 tailed p's, and for p's close to
1, the odds for (1-p)/p are close to zero and mess up a log graph.

7) Use randomized permutation to evaluate the likelihood of seeing (a) a
minimum permuted MEK value as low as the original MEK minimum AND (b) a minimum
permuted MEK as close to midnight as observed (in this case -2 seconds before
midnight).  In runs of 1,000 permutations of the original temporal order there
isn't a single case where (a) OR (b) are found (and thus obviously no cases of
where (a) AND (b) are found either).

8) Apply exactly this same sequence to Y2K+1 day, +2 days, and +15 days to
check the method for unexpected wierdness.  The graphs show that some
deviations can be quite large, but nothing is even close to the results
obtained with the original data.

What does this mean?

Kurtosis is a 4th order statistic.  Negative kurtosis means the distribution is
flatter than normal, and positive means the distribution is skinnier than
normal.  I.e., positive kurt shows a peculiar form of restricted variance.  I
am using -kurt for my graph rather than +kurt, thus going downwards in the
graph means going towards restricted variance.  I use -kurt rather than +kurt
because I take an exponential transform to smooth the data (and taking exp of
positive values quickly blows up).

So, the graph shows that as midnight approaches, the distribution of per-second
variances among eggs, across all time zones, gets skinnier and skinner,
becoming the skinniest at midnight.  To me this suggests that some sort of
coherence has been impressed into the emsemble of eggs?  I.e., something akin
to an effect that only appears in statistical ensembles, rather than something
happening to the individual eggs.  I see no evidence that any of the eggs
individually behaved strangely.  In fact, the egg network as a whole seems to
be behaving quite nicely as a random system the vast majority of the time.

I think it's clear then that (a) the direction of the results make sense in
terms of the overall egg network showing more constrained (2nd or 4th-order)
behavior near midnight, which I interpret as a sign of systemic coherence, and
(b) that the odds obtained using this peculiar set of transforms, even with 10
to 15 analytical interations required to find these maximum odds values, is so
far from chance that it appears to be meaningful.  You can throw Bonferroni at
this all day long and it will still be far from chance.  The odds at -2 seconds
is 228 billion to 1.

If asked to say what is really going on here I'd say my hunch is that something
akin to a thermodynamic argument is going on: Wierd emergent behaviors can show
up in ensembles that would not show up, or even make sense in, individual
RNGs.  I've thought for many years that something like a psychic switch might
actually be possible to build, but only among large arrays of RNGs, and only
with complex analyses applied to the ensemble.   The proof of the pudding, of
course, is whether these 4th order stats applied to other times/events, will
show anything, so I'm applying the same analysis to midnight 1999.

Note: The only difference to create the 1999 graph is use of kurt rather than
-kurt.  Why this makes a difference is yet another mystery to me!

