January 14, 2015:  

1.   New analysis of Riley's "On the probability of occurrence of extreme space weather events"
2.   Fate of "Comment" paper on Riley's submitted to  Space Weather
3.   Lessons from this experience

1.. New analysis of Riley's "On the probability of occurrence of extreme
space weather events" (see the August 22,  and Aug. 25, 2014 entries
for background).  A link to the new analysis, entitled "A second look at
`On the probability of occurrence of extreme space weather events'",
is given at the end of this section.  In the middle of the section is a link
to the old (i.e. original) analysis.

    Riley's paper has been widely cited in the popular press as a reliable scientific
analysis showing that the probability of a crippling magnetic storm as bad as
the worst known (the so-called "Carrington event" of 1859) in a decade is
around 12%.   Riley's conclusion states:

"Our results allowed us to answer a basic question, at least in an
approximate way: How likely are [Carrington] events? ... our results
overall suggest that the likelihood of another Carrington event
ocurring within the next decade is  ~12%."

This makes it sound as if one might quibble over whether the probability were
10% or 12%, but that its order of magnitude is around  a surprisingly high 10%.  

    The popular press generally accepts this estimate uncritically as
scientifically established.  I have heard it said that it has "gone viral"
on the Internet.  Even NASA [external link] has cited this work favorably,
despite glaring flaws which the popular press might easily overlook,
but which NASA should be sufficiently expert to detect.  This has been
cited [external link] as an endorsement by NASA of Riley's conclusions.

    I disagree that Riley's paper demonstrates, or even convincingly motivates, 
anything close to what people seem to think it does.  Earlier I posted  
a lengthy Analysis [15-page .pdf] of Riley's reasoning along with a shorter
Comment [3-page .pdf] summary.  The Comment was submitted to the journal
Space Weather  which published Riley.   The next entry describes what happened
 to that submission.  

    Space weather limits its "comment" papers to 4 pages, so "Comment"
could not address  the many issues which "Analysis" did.  
The main content of "Comment" was to report that my arithmetic
obtained probability estimates (for the probability of an event
worse than Carrington in the next decade)
which were generally an order of magnitude higher than Riley's,
too high to be credible.  (Riley obtains various estimates
from various data sets under various assumptions.)

    Riley's arithmetic is based on some ambiguously reported statistics
which he calls  "MLE fits".  Before posting "Analysis" and "Comment",
I wrote him asking for clarification of the ambiguity.  He didn't reply to any
of four courteously worded inquiries about various points in the paper,
not even to a message asking for acknowledgement of receipt.
So, I had to use my best guess to resolve the ambiguities.

    Later, evidence surfaced causing me to change my original best guess.  
After that change, I was able to reproduce the results of some, but not all,
of Riley's probability estimates.  (That is not  to say that I agree that they were
convincingly derived, only that I agree with the arithmetic based on
the new best guess.)  But serious anomalies remain.
Independently, evidence surfaced of other demonstrable and
serious errors in Riley's paper.  

    Given the changes in the situation, I thought that I should revise or replace
"Analysis". The new information was so extensive and surprising that
 replacement seemed the easier choice. The replacement is titled
 "A second look [30-page .pdf] at `On the probability of occurrence of
extreme space weather events', by P. Riley".  


2. Fate of "Comment" paper submitted to Space Weather

    A previous blog explained why I was submitting the "Comment" and
wondered if Space Weather would hold it for six  months or a year and
then reject it unreferreed with essentially no reason given.
(I wondered because the Journal of Physics A did that for another paper.)  
I was pleasantly surprised when Space Weather had the courtesy
not to delay for a year,  but rejected it immediately, unrefereed.  

    Here is the rejection letter .    It gives the following reasons:

• It claims that publishing the Comment would violate journal policy because
  I had previously "published" it on this website. It gives a link to the policy.
  However, the link specifically provides the exception:  
  

"Posting of a preprint of an article via electronic media does not
[emphasis mine] constitute prior publication  ... " 

[Note added Feb. 13, 2015:   Since the Comment was rejected,
the journal's policy has been revised. Following the link
will provide the revised policy rather than the above.
The above accurately quotes the published policy at the time of rejection.]

• The editor thinks that Riley's article contains adequate caveats
  
  
  "to cause any serious reader to understand that the work 
  was presenting estimates based on sparse data."
  
  I disagree, but this is a matter of opinion.  But it is not a matter of opinion
  that the question of caveats is scarcely mentioned in the"Comment".
  The thrust of the "Comment" was to report major discrepancies
  between my arithmetic leading to the probability estimates and the paper's.
  The editor's report entirely ignores the substance of the "Comment"!



• Next comes the most astounding aspect of the editor's report.
  I am going to quote it in full because if I paraphrase,
  I run the risk that it may sound unbelievable.
  
  
  "Shortly after the Riley paper was published,
  there was a Carrington-class event directed off axis from Earth.
  The magnitude of this event was recorded by space sensors
  and has been widely discussed in the scientific literature.
  Thus, even if the dual publication issue had not arisen,
  I see little reason to begin a comment-reply cycle with referees,
  who would most likely report that:
  1) the Riley paper provided adequate cautionary statements, and
  2) an extreme event has already been realized in a near-miss scenario."
  
  Can a scientist really be saying this?  It ignores the nature of probability.
  
  If a Carrington-class event had hit the Earth yesterday,
  it would be irrelevant to the probability of such an event in the next decade,
  under the usual "independence" assumptions of Riley's paper and of the
  "Comment". (Granted, we would probably think of it as more likely,
  but this is human psychology, not science.)
  The issue is whether Riley's conclusions follow from his assumptions,
  not whether his conclusions happen to be correct.
  Even if the editor happens to believe Riley's concluding estimate of
  probability 12% (for an event worse than Carrington in the next decade),
  this belief should be decoupled from the question of whether Riley
  has argued convincingly that it is correct.
  
  If I toss a penny ten times today and it comes up ten heads,
  am I justified to say that it is quite likely that
  the same thing will happen tomorrow? If last week, someone
  just missed getting a hole in one on my local golf course,
  is it reasonable to say that a hole in one is quite likely on that course?
  

    I sent back a 4-page reply.  It presents an expanded
version of  the above points, except that I thought it politic
to omit the last.  The editor's reply, received about 3 months ago was:



"Dr Parrott,

I have received you [sic] email and respond [sic] in due course, as time permits.

[signed by the editor] "


I am still waiting (but not very expectantly!).
  

3.  Lessons from this experience.   

    Space Weather  refused to look into allegations that the arithmetic of
Riley's paper might be wrong.  This is not a matter of opinion such as
whether he included sufficient caveats.  It is an objective matter
which should be easy to settle.  

    I would prefer to settle it privately with the author, but he has ignored
all my inquiries.  I submitted  the Comment in the belief that Space Weather
would send it to him (as is their stated policy), which would more or less
force him to deal with it.  But not only did Space Weather refuse to submit it
to a referee, there is no indication that they even asked the author about it.  
(My reply to the editor specifically asked about this, but has been ignored.)

    From this it is easy to draw conclusions about the integrity of
Space Weather.  But there are broader issues which are more disturbing.

    How many times does one read in the popular press that
"a study or studies have shown this or that"?  The reporter may be
 correct that some published paper claims to have shown this or that,
but who vets these claims?   And if a claim is false,either from honest error
(as I believe is probably the case for Riley's conclusions)
or from deliberate fabrication, what mechanism is there to correct it?  
If the experiences with Dressel/Jordan and with Space Weather
are any indication, correction is extremely unlikely.

    I am thinking in particular of claims about global warming.  I have not
looked into this in any serious  way.  All I know about it is what I read
in the popular press, articles with phrases like "scientists say that  ...  ".  
The cumulative impression that I have  obtained from dozens of
such articles is that anthropogenic (i.e.,  human-caused)
climate change is beyond scientific doubt  and that it will ruin the planet
if drastic measures are not taken immediately.  

    This may well be true.  I am not a climate change denier.  
But it makes me uneasy that I have no good way to
assess the reliability of the claims.

    Riley's conclusion that there is about a 12% chance of a Carrington-class
event in the next decade is said to have "gone viral" on the Internet.  
Unlike questions of global warming, this is a topic sufficiently restricted that
I  can assess the evidence.  I have done so and am convinced that
there is no reliable scientific basis for such a claim.  
(This is not to say that the claim could not turn out to be true,
only that there is at present no good evidence for it.)  
Is there better evidence for anthropogenic global warming?

    Scientists are people and subject to common frailties.  In my experience,
based on all the scientistsI have met (including mathematicians),
I would probably trust fewer than half to uphold scientific standards
when that conflicts with personal gain.  

    Grant-chasing is so embedded in the fabric of big science
that it is hard to imagine much of a career for anyone
who does not participate.   Even grant-givers with
 the highest standards and integrity have to rely on referees
to tell them who is deserving.  And the referees will be part of the system,
the most successful grant recipients.  It is a system with a built-in bias
toward maintenance of currently fashionable dogma.

    I am sure that many have recognized the problem.  
I am enormously impressed by a few who are doing something about it.  
There is a  website called pubpeer.com which  publishes peer reviews
of scientific papers.  It was built and is run by anonymous volunteers.  

    It appears to be run in a fair and professional way.
 Ideally, one would like all reviews to be signed,
but this would be unrealistic, so they do accept anonymous reviews.  
 When someone posts a  review, the site sends it to the author
 for his comments.

    I imagine that  thousands of hours have gone into its construction.  
I can't think of anyone more deserving of an award from
the MacArthur Foundation than those who built this site.  
I would nominate them if I knew how.  

    I will put my reservations about Riley's paper in a signed review
on that website.  Apart from "private"  websites such as
the university website on which you are reading this,
it is the only outlet that I know for such reservations
when journals refuse to even consider them.