1. S2VSR 2023-06¶
The PyrNet setup for Small Scale Variability of Solar Radiation (S2VSR) was setup for calibration on the ARM-SGP guest instrument facility (GIF) for calibration from 2023-06-02 to 2023-06-08. Cross-calibration is done versus reference observations from the ARM-SGP broadband radiometer station (BRS) (doi:10.5439/1550918)
1.1. Imports¶
1.2. Prepare PyrNet data¶
For calibration preparation the PyrNet data is processed to level l1b using a calibration factor of 7 (uV W-1 m2) for all pyranometers with the pyrnet process l1b tool. This is done to unify the conversion to sensor voltage during calibration and not run into valid_range limits for netcdf encoding. Here we generate the calibration.json file for the processing to l1b:
box_numbers = np.arange(1,101)
calibrations = {f"{bn:03d}":[7,7] for bn in box_numbers}
calibjson = {"2000-01-01": calibrations}
# with open("pyrnet_calib_prep.json","w") as txt:
# json.dump(calibjson, txt)
Within pyrnet_config_calibration_prep.json:
{"file_calibration" : "pyrnet_calib_prep.json"}
Workflow for preparation
Prepare pyrnet_config_calibration_prep.json with contributors metadata and the dummy calibration config file.
$ pyrnet process l1a -c pyrnet_config_calibration_prep.json raw_data/*.bin l1a/$ pyrnet process l1b -c pyrnet_config_calibration_prep.json l1a/*.nc l1b/$ d=2023-06-02; until [[ $d > 2023-06-08 ]]; do pyrnet merge l1b/pyrnet_${d}_*.nc l1b_network/pyrnet_${d}_s2vsr_calibration.c01.nc; d=$(date -I -d "$d + 1 day"); doneUpload the network files to the Thredds server
1.3. Configuration¶
Data is available via Thredds server. The URLs might have changed when trying to rerun this notebook.
For the ARM data, create a ARM user account first and then order the doi:10.5439/1550918, and choose the Thredds-Server option. You will receive the Thredds-URL via e-mail.
The PyrNet data should be available via tds.tropops.de, if not, contact witthuhn@tropos.de.
ARM_URL = "https://archive.arm.gov/orders/dodsC/orders/witthuhnj1/240295/sgpbrsC1.b1.{date:%Y%m%d}.000000.custom.cdf"
PYR_URL = "https://tds.tropos.de/thredds/dodsC/scccJher/2023_s2vsr/calibration/pyrnet_{date:%Y-%m-%d}_s2vsr_calibration.c01.nc"
dates = pd.date_range("2023-06-02","2023-06-08")
stations = np.arange(1,101)
# lookup which box contains actually a pyranometer/ extra pyranometer
mainmask, extramask = [] , []
for box in stations:
_, serials, _ = pyrnet.meta_lookup(dates[0],box=box)
mainmask.append( True if len(serials[0])>0 else False )
extramask.append( True if len(serials[1])>0 else False )
1.3.1. Load ARM-SGP Data¶
1.3.2. Load PyrNet Data¶
1.4. Calibration¶
The calibration follows the ISO 9847:1992 - Solar energy — Calibration of field pyranometers by comparison to a reference pyranometer.
TODO: Revise versus 2023 EU version.
Cloudy sky treatment is applied.
1.4.1. Step 1¶
Drop Night measures and low signal measures from pyranometer data. Since calibration without incoming radiation doesnt work.
This data is kept for calibration:
solar zenith angle < 80° ( as recommended in ISO 9847)
Measured Voltage > 0.033 V, e.g. ADC count is 0 or 1 of 1023 (drop the lowest ~1%)
Voltage measured ($V_m$) at the logger is the amplified Senor voltage ($V_S$) by a gain of 300.
$ V_m = 300 V_S$
As the uncalibrated flux measurements ($F_U$) are calibrated with a fixed factor of 7 uV W-1 m2:
$ V_s = 71e-6 F_U $
# Set flux values to nan if no pyranometer is installed.
pyr.ghi.values = pyr.ghi.where(mainmask).values
pyr.gti.values = pyr.gti.where(extramask).values
# convert to measured voltage
pyr.ghi.values = pyr.ghi.values * 7 * 1e-6
pyr.gti.values = pyr.gti.values * 7 * 1e-6
# Step 1, select data
pyr = pyr.where(pyr.szen<80, drop=True)
pyr.ghi.values = pyr.ghi.where(pyr.ghi>0.033/300.).values
pyr.gti.values = pyr.gti.where(pyr.gti>0.033/300.).values
1.4.2. Step 2¶
Interpolate reference to PyrNet samples and combine to a single Dataset
# interpolate reference to PyrNet
arm = arm.interp(time=pyr.time)
# Calibration datasets for main and extra pyranometer
Cds_main = xr.Dataset(
data_vars={
'reference_Wm2': ('time', arm['down_short_hemisp'].data),
'pyrnet_V': (('time','station'), pyr['ghi'].data)
},
coords= {
"time": pyr.time,
"station": pyr.station
}
)
Cds_extra = xr.Dataset(
data_vars={
'reference_Wm2': ('time', arm['down_short_hemisp'].data),
'pyrnet_V': (('time','station'), pyr['gti'].data)
},
coords= {
"time": pyr.time,
"station": pyr.station
}
)
1.4.3. Step 3¶
Remove outliers from series using xarray grouping and apply function. The following functions removes outliers (deviation more than 2% according to ISO 9847) from a selected group. This step involves calculating calibration series and the integration of one hour intervals to smooth out high variable situation, which would break the calibration even when time synchronization is slightly off. Also this gets rid of some random shading events like birds / chimney / rods in line of sigth, which would affect calibration otherwise. We following ISO 9847 5.4.1.1 equation (2) here.
def remove_outliers(x):
"""
x is an xarray dataset containing these variables:
coords: 'time' - datetime64
'pyrnet_V' - array - voltage measures of pyranometer
'reference_Wm2' - array - measured irradiance of reference
"""
# calculate calibration series for single samples
C = x['pyrnet_V'] / x['reference_Wm2']
# integrated series
ix = x.integrate('time')
M = ix['pyrnet_V'] / ix['reference_Wm2']
while np.any(np.abs(C-M) > 0.02*M):
#calculate as long there are outliers deviating more than 2 percent
x = x.where(np.abs(C-M) < 0.02*M)
C = x['pyrnet_V'] / x['reference_Wm2']
#integrated series
ix = x.integrate('time')
M = ix['pyrnet_V'] / ix['reference_Wm2']
#return the reduced dataset x
return x
# remove outliers
Cds_main = Cds_main.groupby('time.hour').apply(remove_outliers)
Cds_extra = Cds_extra.groupby('time.hour').apply(remove_outliers)
# hourly mean
Cds_main = Cds_main.coarsen(time=60*60,boundary='trim').mean(skipna=True)
Cds_extra = Cds_extra.coarsen(time=60*60,boundary='trim').mean(skipna=True)
1.4.4. Step 4¶
The series of measured voltage and irradiance is now without outliers. So we use equation 1 again to calculate from this reduced series the calibration factor for the instant samples.
C_main = 1e6*Cds_main['pyrnet_V'] / Cds_main['reference_Wm2']
C_extra = 1e6*Cds_extra['pyrnet_V'] / Cds_extra['reference_Wm2']
1.4.5. Step 5¶
We just found the Calibration factor to be the mean of the reduced calibration factor series and the uncertainty to be the standard deviation of this reduced series. Steo 3, 4 and 5 are done for every pyranometer seperate.
C_main_mean = C_main.mean(dim='time',skipna=True)
C_main_std = C_main.std(dim='time',skipna=True)
C_extra_mean = C_extra.mean(dim='time',skipna=True)
C_extra_std = C_extra.std(dim='time',skipna=True)
1.5. Results¶
<Figure size 640x480 with 0 Axes>
<Figure size 640x480 with 0 Axes>
calibration_new = {}
print(f"Box: Main , Extra ")
for box in C_main_mean.station:
Cm = C_main_mean.sel(station=box).values
Ce = C_extra_mean.sel(station=box).values
Um = C_main_std.sel(station=box).values
Ue = C_extra_std.sel(station=box).values
print(f"{box:3d}: {Cm:.2f} +- {Um:.3f} , {Ce:.2f} +- {Ue:.3f}")
Cm = float(np.round(Cm,2)) if not np.isnan(Cm) else None
Ce = float(np.round(Ce,2)) if not np.isnan(Ce) else None
Um = float(np.round(Um,2)) if not np.isnan(Um) else None
Ue = float(np.round(Ue,2)) if not np.isnan(Ue) else None
calibration_new.update({
f"{box:03d}": [Cm, Ce]
})
calibjson = {"2023-06-02": calibration_new}
with open("pyrnet_calib_2023-06_s2vsr.json","w") as txt:
json.dump(calibjson, txt)
Box: Main , Extra
1: 7.64 +- 0.360 , 6.88 +- 0.481
2: 7.56 +- 0.322 , nan +- nan
3: 7.69 +- 0.282 , nan +- nan
4: 7.43 +- 0.728 , 8.25 +- 0.553
5: 7.57 +- 0.321 , 6.27 +- 0.469
6: 6.62 +- 0.374 , nan +- nan
7: 7.63 +- 0.341 , 6.73 +- 0.865
8: 7.87 +- 0.355 , nan +- nan
9: 7.62 +- 0.327 , 6.73 +- 0.476
10: 7.73 +- 0.357 , 2.14 +- 1.451
12: 7.68 +- 0.322 , 6.71 +- 0.523
13: 7.08 +- 0.409 , nan +- nan
15: 7.63 +- 0.316 , 6.77 +- 0.651
16: 7.42 +- 0.038 , nan +- nan
17: 7.43 +- 0.456 , nan +- nan
18: 7.63 +- 0.403 , 6.74 +- 0.579
19: 7.55 +- 0.041 , nan +- nan
20: 7.37 +- 0.505 , nan +- nan
21: 7.66 +- 0.360 , nan +- nan
22: 7.17 +- 0.570 , nan +- nan
23: 7.41 +- 0.026 , nan +- nan
24: 7.71 +- 0.429 , 6.30 +- 0.556
25: 7.36 +- 0.445 , 7.03 +- 0.492
26: 7.64 +- 0.366 , 6.88 +- 0.523
28: 7.56 +- 0.031 , nan +- nan
29: 7.41 +- 0.036 , nan +- nan
32: 7.79 +- 0.328 , 6.83 +- 0.459
33: 7.84 +- 0.356 , 6.64 +- 0.504
35: 7.71 +- 0.384 , 6.98 +- 0.568
37: 7.80 +- 0.324 , 6.61 +- 0.529
38: 7.30 +- 0.501 , 6.83 +- 0.495
43: 7.42 +- 0.469 , 7.20 +- 0.610
44: 7.17 +- 0.433 , 7.05 +- 0.559
46: 7.70 +- 0.337 , 6.95 +- 0.562
47: 7.38 +- 0.542 , 7.45 +- 0.519
49: 7.85 +- 0.472 , 7.19 +- 0.600
53: 7.62 +- 0.346 , 6.06 +- 0.610
54: 7.47 +- 0.038 , nan +- nan
55: 7.39 +- 0.260 , nan +- nan
58: 7.60 +- 0.300 , 6.80 +- 0.509
60: 7.70 +- 0.261 , 6.61 +- 0.434
61: 7.25 +- 0.435 , nan +- nan
63: 7.58 +- 0.303 , nan +- nan
64: 7.17 +- 0.411 , nan +- nan
65: 7.58 +- 0.438 , nan +- nan
66: 7.63 +- 0.311 , nan +- nan
67: 7.13 +- 0.339 , nan +- nan
68: 6.78 +- 0.336 , nan +- nan
70: 6.88 +- 0.404 , nan +- nan
71: 7.48 +- 0.231 , nan +- nan
72: 7.49 +- 0.312 , nan +- nan
73: 7.16 +- 0.326 , nan +- nan
75: 6.60 +- 0.334 , nan +- nan
77: 7.50 +- 0.252 , nan +- nan
78: 6.76 +- 0.368 , nan +- nan
79: 7.54 +- 0.258 , nan +- nan
80: 6.88 +- 0.047 , nan +- nan
81: 7.52 +- 0.280 , nan +- nan
83: 7.31 +- 0.392 , nan +- nan
84: 7.21 +- 0.039 , nan +- nan
85: 7.46 +- 0.256 , nan +- nan
86: 7.29 +- 0.435 , 6.62 +- 0.517
87: 7.28 +- 0.432 , nan +- nan