“Phraud” that Dwarfs Climategate?

_{Barry Arrington

December 30, 2014

Intelligent Design}

Share: Facebook; Twitter; LinkedIn; Flipboard; Print; Email

Comments

The people involved admitted that they fudged the data.
They did?
They also refused to disclose their methods, taking offense that anyone should want to check up on them.
They did? Why don't we look at the original publication to make sure this is an accurate description of what went on? So, what is the exact publication in question? And where did they admit they fudged the data? And how did they refuse to disclose the methods?hrun0815_{January 15, 2015
January
01
Jan
15
15
2015
06:33 AM
6
06
33
AM
PDT}

Querius: The people involved admitted that they fudged the data. The historical data has discontinuities. While the original studies spliced the data manually, today, modern statistical techniques are used to extract the trends from the raw data. Querius: They also refused to disclose their methods, taking offense that anyone should want to check up on them. The methods were published in peer reviewed journals. More important, the raw data has always been available to researchers who were willing to take the trouble to aggregate it. Today, the raw data is easily available. Querius: The scientific method is definitely NOT force-fitting data into curves and then dreaming up semi-plausible explanations for why the data was altered. Multiple independent studies have largely confirmed the original findings.Zachriel_{January 15, 2015
January
01
Jan
15
15
2015
06:01 AM
6
06
01
AM
PDT}

No, Zachriel. That's not what I'm saying. The people involved admitted that they fudged the data. They also refused to disclose their methods, taking offense that anyone should want to check up on them. Thus, their method is by definition flawed, unscientific, and indefensible, and their "results" are as well. The scientific method is definitely NOT force-fitting data into curves and then dreaming up semi-plausible explanations for why the data was altered. -QQuerius_{January 14, 2015
January
01
Jan
14
14
2015
11:35 PM
11
11
35
PM
PDT}

Querius: Again, I’m not familiar with the methods used in meteorology to try to correct data collection errors, but they have to be clearly justified in each case and the algorithms that were used freely available... Sorry, I’m not buying into their results, whatever they might be. You seem to be saying that rather than consider the findings or methods, you'll reject the findings in any case.Zachriel_{January 8, 2015
January
01
Jan
8
08
2015
04:29 AM
4
04
29
AM
PDT}

Again, I’m not familiar with the methods used in meteorology to try to correct data collection errors, but they have to be clearly justified in each case and the algorithms that were used freely available.
Correct. And that's why we should look at the original paper and see if that's what they did.
However, I do have a problem with passing off interpolated values if they were raw weather station data, if that’s what they did.
Correct. And that's why we should look at the original paper and see if that's what they did.
For example, you can’t just draw a line between two weather stations and average the results in between (linear interpolation). You have to incorporate the rates of change (first and second derivatives). If multiple stations are involved, perhaps you would interpolate using RMS calculations. Again, I don’t know what the meteorologists used, but they shouldn’t be able to adjust raw data without scrupulous justification at every station.
Um, yeah. So let's take a look already.
I took only one upper division meteorology class in college and in the lab we did interpolate the data *between* weather stations to create isobaric and weather front surface maps. But we wouldn’t have dared to fill in any missing weather station data or change any data that were reported! And you can’t just discard data, unless you can justify it statistically.
Yes, yes, yes. Let's look already.
If it’s true that in the past some stations confused farenheit and centigrade data (which I find hard to believe), and allowed air conditioners to be installed next to weather data collection stations (which can be seen in some of the photographs), it gives me even less confidence in the competence of the people involved and the raw data that they recorded. Just imagine having a heat source that close, and your “homogenization” adjustments dependent on the direction and velocity of the breeze and wind eddies around the building.
So then we should look maybe how many of the weather stations we are talking about. One out of a thousand? Nine hundred out of a thousand?
Sorry, I’m not buying into their results, whatever they might be.
Aehhhhhh, so you didn't look at the paper, you have no idea what they actually did or did not do, you don't know how they justified the process, how many data points were involved, but... you don't buy their results. Why don't we just look at the paper and actually see what they did and if it is actually problematic.hrun0815_{January 7, 2015
January
01
Jan
7
07
2015
06:43 PM
6
06
43
PM
PDT}

Again, I'm not familiar with the methods used in meteorology to try to correct data collection errors, but they have to be clearly justified in each case and the algorithms that were used freely available. However, I do have a problem with passing off interpolated values if they were raw weather station data, if that's what they did. For example, you can't just draw a line between two weather stations and average the results in between (linear interpolation). You have to incorporate the rates of change (first and second derivatives). If multiple stations are involved, perhaps you would interpolate using RMS calculations. Again, I don't know what the meteorologists used, but they shouldn't be able to adjust raw data without scrupulous justification at every station. I took only one upper division meteorology class in college and in the lab we did interpolate the data *between* weather stations to create isobaric and weather front surface maps. But we wouldn't have dared to fill in any missing weather station data or change any data that were reported! And you can't just discard data, unless you can justify it statistically. If it's true that in the past some stations confused farenheit and centigrade data (which I find hard to believe), and allowed air conditioners to be installed next to weather data collection stations (which can be seen in some of the photographs), it gives me even less confidence in the competence of the people involved and the raw data that they recorded. Just imagine having a heat source that close, and your "homogenization" adjustments dependent on the direction and velocity of the breeze and wind eddies around the building. Sorry, I'm not buying into their results, whatever they might be. -QQuerius_{January 7, 2015
January
01
Jan
7
07
2015
06:24 PM
6
06
24
PM
PDT}

hrun0815: is that paper actually the original source for the discrepancy between raw and processed data in the link picked by Querius? The article about Australia cited by Querius seems more concerned with the homogenization algorithm than any misreporting of the raw data. http://joannenova.com.au/2014/08/the-heat-is-on-bureau-of-meteorology-altering-climate-figures-the-australian/Zachriel_{January 7, 2015
January
01
Jan
7
07
2015
03:34 PM
3
03
34
PM
PDT}

Zachariel, is that paper actually the original source for the discrepancy between raw and processed data in the link picked by Querius? Querius, do you know? I think it is pretty clear that ONLY if we compare the actual scientific publication of the processed data with the raw data can we actually make a decision in which of the listed categories this should fall (outright fraud, misleading, sloppy science, ...). Can you confirm that this is indeed the paper in question?hrun0815_{January 7, 2015
January
01
Jan
7
07
2015
08:38 AM
8
08
38
AM
PDT}

Querius: Why don’t we start with the last link, the one from Australia? Trewin, A daily homogenized temperature data set for Australia, International Journal of Climatology 2013:
A new homogenized daily maximum and minimum temperature data set, the Australian Climate Observations Reference Network—Surface Air Temperature data set, has been developed for Australia. This data set contains data from 112 locations across Australia, and extends from 1910 to the present, with 60 locations having data for the full post-1910 period. These data have been comprehensively analysed for inhomogeneities and data errors ensuring a set of station temperature data which are suitable for the analysis of climate variability and trends. For the purposes of merging station series and correcting inhomogeneities, the data set has been developed using a technique, the percentile-matching (PM) algorithm, which applies differing adjustments to daily data depending on their position in the frequency distribution. This method is intended to produce data sets that are homogeneous for higher-order statistical properties, such as variance and the frequency of extremes, as well as for mean values. The PM algorithm is evaluated and found to have clear advantages over adjustments based on monthly means, particularly in the homogenization of temperature extremes.
Zachriel_{January 7, 2015
January
01
Jan
7
07
2015
06:40 AM
6
06
40
AM
PDT}

Querius: The specific processing step(s) performed were apparently never revealed, so my contention is that because they cannot be independently validated, the results are not scientifically acceptable. That is simply incorrect. Homogenization is an important issue in climate science, and has been subject to intensive study. The problem is that older data was collected for local weather forecasting, and is of uneven quality. The protocols changed, personnel changed, the stations were moved, there was urban development around stations, the languages and cultures and government systems were different, the handwriting is often difficult to read, instrumentation changed without notice, some institutions won't release their data, some records are damaged or missing, the guy was sick that day, etc. The first modern studies were done by manually splicing data together where there were obvious discontinuities. Since then, statisticians have made the process rigorous. This has all been done in the open, in peer reviewed journals. The edges of science are often determined by very incomplete and tentative data. You will find scientists in many fields of study extracting usable information from very tenuous data.Zachriel_{January 7, 2015
January
01
Jan
7
07
2015
06:34 AM
6
06
34
AM
PDT}

Querius, could you help me out and directly cite the original publication where the data processing is not or only inadequately described?hrun0815_{January 7, 2015
January
01
Jan
7
07
2015
05:48 AM
5
05
48
AM
PDT}

Querius @ 50
Also, interpolated values are not measured data, since they depend on the algorithm used.Again, I’m not accusing them of fraud, just a very poor methodology.
If you have limited data and want finer measurement within any two points, or plot a curve, or integrate and find the area under curve, say, to measure the total temperature difference within certain period, you have to interpolate. Interpolation is standard practice not just in science, but engineering too. Interpolation is also a cool technique to 'zoom' into a particular curve to see finer trends.Me_Think_{January 7, 2015
January
01
Jan
7
07
2015
12:25 AM
12
12
25
AM
PDT}

There's a profound difference between transferring a small amount of heat from the outdoors into a thermometer and observing the resulting change, and performing a mathematical operation on temperature data. The specific processing step(s) performed were apparently never revealed, so my contention is that because they cannot be independently validated, the results are not scientifically acceptable. Also, interpolated values are not measured data, since they depend on the algorithm used. Again, I'm not accusing them of fraud, just a very poor methodology. -QQuerius_{January 6, 2015
January
01
Jan
6
06
2015
10:18 PM
10
10
18
PM
PDT}

Thinking back, if I “homogenized” my results in a quantitative analysis or a physics lab, I’d have probably failed the class, and possibly faced disciplinary action.
I'm pretty sure if what you did was defined and reported it would have been fine-- no matter whether the procedure was called reconstruction, homogenization, processing, normalizing, correcting, or some other term.
Depending on the experiment, I sometimes had to account for heat capacity, buoyancy, tare, or other significant source of error. But I don’t agree with you that all of science is as slipshod as indicated, nor do I feel obligated to defend this travesty.
Yep, depending on the instrument or experimental setup you have to somehow process the raw data to arrive at some form of useful output. And in virtually all cases you are not measuring the actual piece of information you are interested in, but rather some form of proxy that is somehow converted into what you actually want to know. Which means that as far as I can tell virtually all data is both derived and processed. The key question is: is this processing step described in the original publication of the data or not. If not, it's sloppy science, if it is, then you can criticize the process, but rather than shoring general outrage you should probably explain why this particular processing step should not have been performed.hrun0815_{January 6, 2015
January
01
Jan
6
06
2015
08:02 PM
8
08
02
PM
PDT}

Zachriel @ 39,
That is incorrect. Data and methods are published in peer reviewed journals. Automated statistical methods have replaced manual splicing, and include regression models, percentile-matching and kriging.
And fudging. -QQuerius_{January 6, 2015
January
01
Jan
6
06
2015
07:50 PM
7
07
50
PM
PDT}

Me_Think,
What else did you think homogenization is? There are various methods too, as indicated in section 3.4
It sounded like blending a mixture, so I didn't know that this is what used to be called "fudging the results." If you had read the article, you would know that the people involved declined to reveal what corrections were applied. I'm not accusing them of fraud, but as far as scientific ethics goes, this is a fail. -QQuerius_{January 6, 2015
January
01
Jan
6
06
2015
07:42 PM
7
07
42
PM
PDT}

hrun0815, Thinking back, if I "homogenized" my results in a quantitative analysis or a physics lab, I'd have probably failed the class, and possibly faced disciplinary action.
It turns out that virtually all data are derived estimates.
Depending on the experiment, I sometimes had to account for heat capacity, buoyancy, tare, or other significant source of error. But I don't agree with you that all of science is as slipshod as indicated, nor do I feel obligated to defend this travesty. Me_Think answered my question regarding "homogenization" of historical weather data. I believe you'll find most of the answers to your questions in the referenced article. -QQuerius_{January 6, 2015
January
01
Jan
6
06
2015
07:31 PM
7
07
31
PM
PDT}

I read the complete paper “Homogenization” apparently is a set of methods of trying to estimate and correct for the error in historical data.
What else did you think homogenization is? There are various methods too, as indicated in section 3.4Me_Think_{January 6, 2015
January
01
Jan
6
06
2015
07:07 PM
7
07
07
PM
PDT}

“Homogenization” apparently is a set of methods of trying to estimate and correct for the error in historical data. I won’t fault meteorologists for trying to reconstruct the data, but the result is a derived estimate. One wouldn’t even call it corrected, since not all the factors are known. I’m speechless.
Yes, sure. Data are derived estimates. It turns out that virtually all data are derived estimates. I just don't understand why you are speechless. The question still remains: In your example were the processing steps explained at the time the data was published or was it claimed that the data was unprocessed? Did anything nefarious go on? Was something hidden? We're mistakes made? Should they have used a different processing procedure? None at all?hrun0815_{January 6, 2015
January
01
Jan
6
06
2015
06:44 PM
6
06
44
PM
PDT}

Thank you, Me_Think @ 41. I read the complete paper, which started out extremely basic:
If we measure rainfall, in order for the data to be useful for future users, we also need to document where and how the measurements were made. Station documentation is information about the data or data about the data: metadata. The word metadata is made up by the addition of the Greek “meta” (beyond) and the Latin “datum” (a given fact). Metadata should reflect how, where, when and by whom information was collected. Ideally, a complete metadata should register all the changes a station has undergone during its lifetime, composing what is called the station history.
Ya think? It ended pretty basic, too. Examples they cited of the problems that resulted from not keeping station histories included a period of time when one station apparently switched between farenheit and centigrade without using a conversion formula! Are you kidding me??? "Homogenization" apparently is a set of methods of trying to estimate and correct for the error in historical data. I won't fault meteorologists for trying to reconstruct the data, but the result is a derived estimate. One wouldn't even call it corrected, since not all the factors are known. I'm speechless. -QQuerius_{January 6, 2015
January
01
Jan
6
06
2015
06:31 PM
6
06
31
PM
PDT}

Hmmm, yeah, so this whole thing predictably fizzled out. All investigations found no fraud occurring at the so-called climategate. There was no 'phraud' or any impropriety in the paper on pH measurements. No data was hidden. There were a whole bunch of suggestions of massaging data, but so far we couldn't drill down into a single example to actually show that anything improper was done. And even though there is this great pause going on, the past decade saw the four hottest years of the nstrumental temperature record. Maybe people would take this and rethink their position on human-caused climate change... :) Nah. Of course not. There's just another fraud or scandal or evil conspiracy plot waiting just around the corner.hrun0815_{January 6, 2015
January
01
Jan
6
06
2015
11:47 AM
11
11
47
AM
PDT}

Querius @ 38, If you want to learn more - There are guidelines for homogenization given by World Meteorological Organization : Guidelines Me_Think_{January 5, 2015
January
01
Jan
5
05
2015
06:48 AM
6
06
48
AM
PDT}

Querius, so we can dispense with slanted looks at this lets identify the sources of the raw data and the published (or altered) data. From your description it sounds to me that 'homogenized' refers to a form of data processing. The question is if this process is described in the paper where this 'homogenized' data was published? Depending on how this question is answered I guess this example could fall into categories 1, 2, or 4.hrun0815_{January 5, 2015
January
01
Jan
5
05
2015
05:38 AM
5
05
38
AM
PDT}

Querius: The raw historical data was altered by a process called “homogenized” without any information as to what this process entailed. That is incorrect. Data and methods are published in peer reviewed journals. Automated statistical methods have replaced manual splicing, and include regression models, percentile-matching and kriging.Zachriel_{January 5, 2015
January
01
Jan
5
05
2015
05:20 AM
5
05
20
AM
PDT}

Why don't we start with the last link, the one from Australia? I don't think this one fits into one of your categories. The raw historical data was altered by a process called "homogenized" without any information as to what this process entailed. I think the post processing needs to be clearly documented, and the resulting data distinguished from raw data. What do you think about it? -QQuerius_{January 4, 2015
January
01
Jan
4
04
2015
11:21 PM
11
11
21
PM
PDT}

So into which of the categories I listed do these example fall, Q? Where is the officially published data and where did the raw data come from? Maybe you can start with one example and we could start chasing the issue down.hrun0815_{January 4, 2015
January
01
Jan
4
04
2015
02:04 PM
2
02
04
PM
PDT}

Sorry for spamming, but the issue of massaging historical data continues. Foolishly, I thought it had been done only once. http://www.principia-scientific.org/breaking-new-climate-data-rigging-scandal-rocks-us-government.html And now also in Australia . . . http://joannenova.com.au/2014/08/the-heat-is-on-bureau-of-meteorology-altering-climate-figures-the-australian/ Gosh, the past is changing faster than the present! Orwell was right! -QQuerius_{January 4, 2015
January
01
Jan
4
04
2015
01:36 PM
1
01
36
PM
PDT}

And this one . . . http://wattsupwiththat.com/2012/09/26/nasa-giss-caught-changing-past-data-again-violates-data-quality-act/ This isn't right. -QQuerius_{January 4, 2015
January
01
Jan
4
04
2015
01:23 PM
1
01
23
PM
PDT}

I'm still looking for the original article, but I found these two links that are interesting: http://stevengoddard.wordpress.com/2014/06/23/noaanasa-dramatically-altered-us-temperatures-after-the-year-2000/ http://wattsupwiththat.com/2014/06/28/the-scientific-method-is-at-work-on-the-ushcn-temperature-data-set/ I loved the photo of the MMTS sensor in the parking lot! LOL! ;-) -QQuerius_{January 4, 2015
January
01
Jan
4
04
2015
01:19 PM
1
01
19
PM
PDT}

hrun0815, IIRC, the author plotted the deviations between the original and the current historical data. From the single-cycle sinusoidal shape of these deviations, I think it would be reasonable to conclude that a methodological correction was applied to the historical raw data to create new "raw" historical data. The author's issue was that the correction was not disclosed along with the revised data. If true, would suggest that the revised historical data is not raw but derived. I'll look for the article. -QQuerius_{January 4, 2015
January
01
Jan
4
04
2015
12:48 PM
12
12
48
PM
PDT}

1 2 Next

You must be logged in to post a comment.

Leave a Reply