3. 2015 MelCol¶

processed with pyrnet-0.2.16

The PyrNet was setup for calibration in a dense array on the Melpitz measurement field from 2015-05-06 to 2015-05-11. Cross-calibration is done versus reference observations from the TROPOS MObile RaDiation ObseRvatory (MORDOR) station.

3.1. Imports¶

#|dropcode
from IPython.display import display, Latex
import os
import xarray as xr
import pandas as pd
import numpy as np
import datetime as dt
import matplotlib.pyplot as plt
import jstyleson as json

from scipy.optimize import differential_evolution

from pvlib import clearsky
from pvlib.location import Location

import pyrnet.pyrnet
import pyrnet.utils

3.2. Prepare PyrNet data¶

For calibration preparation the PyrNet data is processed to level l1b using a calibration factor of 7 (uV W-1 m2) for all pyranometers with the pyrnet process l1b tool. This is done to unify the conversion to sensor voltage during calibration and not run into valid_range limits for netcdf encoding.

Before running this notebook, new absolute calibration factors have to be determined with calibration_melcol.ipynb

3.3. Configuration¶

Set local data paths and lookup metadata.

pf_mordor = "mordor/{date:%Y-%m-%d}_Radiation.dat"
pf_pyrnet = "l1b_network/{date:%Y-%m-%d}_P1D_pyrnet_melcol_n000l1bf1s.c01.nc"
dates = pd.date_range("2015-05-06","2015-05-11")

loc = Location(51.525175, 12.91648, altitude=90) # Melpitz

stations = np.arange(1,101)
# lookup which box contains actually a pyranometer/ extra pyranometer
mainmask = [] 
for box in stations:
    _, serials, _, _ = pyrnet.pyrnet.meta_lookup(dates[0],box=box)
    mainmask.append( True if len(serials[0])>0 else False )

3.4. Calibration¶

The calibration follows the ISO 9847:1992 - Solar energy — Calibration of field pyranometers by comparison to a reference pyranometer. For selecting and masking the data.

TODO: Revise versus 2023 EU version.

Cloudy sky treatment is applied.

3.4.1. Step 1¶

Drop Night measures and low signal measures from pyranometer data. Since calibration without incoming radiation doesnt work.

This data is kept for calibration:

solar zenith angle < 80° ( as recommended in ISO 9847)
Measured Voltage > 0.033 V, e.g. ADC count is 0 or 1 of 1023 (drop the lowest ~1%)

Voltage measured ($V_m$) at the logger is the amplified Senor voltage ($V_S$) by a gain of 300.

$ V_m = 300 V_S$

As the uncalibrated flux measurements ($F_U$) are calibrated with a fixed factor of 7 uV W-1 m2:

$ V_s = 71e-6 F_U $

# Set flux values to nan if no pyranometer is installed.
pyr.ghi.values = pyr.ghi.where(mainmask).values

# convert to measured voltage
pyr.ghi.values = pyr.ghi.values * 7 * 1e-6

# Step 1, select data
pyr = pyr.where(pyr.szen<80, drop=True)
pyr.ghi.values = pyr.ghi.where(pyr.ghi>0.033/300.).values

3.4.2. Step 2¶

Apply new absolute calibration from calibration_melcol.ipynb

calib_new = pyrnet.utils.read_json("pyrnet_calib_new.json")
calib_new = calib_new[list(calib_new.keys())[0]]
for station in pyr.station:
    pyr.ghi.sel(station=station).values /= calib_new[f"{station:03d}"][0] * 1e-6 

3.4.3. Step 3¶

Interpolate reference to PyrNet samples and combine to a single Dataset

# interpolate reference to PyrNet
mordor = mordor.interp(time=pyr.time).interpolate_na()


# Calibration datasets for main and extra pyranometer
Cds_main = xr.Dataset(
    data_vars={
        'reference_Wm2': ('time', mordor['TP2_Wm2'].data[mordor.cs_mask]),
        'pyrnet_Wm2': (('time','station'), pyr['ghi'].data[mordor.cs_mask,:]),
        'szen': ('time',pyr.szen.mean("station",skipna=True).data[mordor.cs_mask])
    },
    coords= {
        "time": pyr.time[mordor.cs_mask],
        "station": pyr.station
    }
)

3.4.4. Step 4¶

Remove outliers from series using xarray grouping and apply function.

def remove_outliers(x):
    """
    x is an xarray dataset containing these variables:
    coords: 'time' - datetime64
    'pyrnet_V' - array - voltage measures of pyranometer
    'reference_Wm2' - array - measured irradiance of reference
    """

    # calculate calibration series for single samples
    C = x['pyrnet_Wm2'] / x['reference_Wm2']
    # integrated series 
    ix = x.integrate('time')
    M = ix['pyrnet_Wm2'] / ix['reference_Wm2']
    
    while np.any(np.abs(C-M) > 0.01*M):
        #calculate as long there are outliers deviating more than 2 percent
        x = x.where(np.abs(C-M) < 0.01*M)
        C = x['pyrnet_Wm2'] / x['reference_Wm2']
        #integrated series 
        ix = x.integrate('time')
        M = ix['pyrnet_Wm2'] / ix['reference_Wm2']
        
    #return the reduced dataset x
    return x

# remove outliers
Cds_main = Cds_main.groupby('time.hour').apply(remove_outliers)

# hourly mean
Cds_main = Cds_main.coarsen(time=60*60,boundary='trim').mean(skipna=True)

3.4.5. Step 5¶

The series of measured reference and pyrnet irradiance is now without outliers. Now we can fit a cubic function - (reference/pyrnet) vs. szen - to determine the cosine correction function for pyrnet.

fig,ax = plt.subplots(1,1)
p = ax.plot(Cds_main.szen,Cds_main.reference_Wm2/Cds_main.pyrnet_Wm2,ls='',marker='.')

png

def cubic_function(x, a, b, c, d):
    return a * x ** 3 + b * x ** 2 + c * x + d

def objective_function(coefficients, x, y):
    a, b, c, d = coefficients
    y_pred = cubic_function(x, a, b, c, d)
    error = np.sum((y - y_pred) ** 2)
    return error
bounds = [(-5, 5), (-5, 5), (-5, 5), (-5, 5)]


ratio = Cds_main.reference_Wm2/Cds_main.pyrnet_Wm2

result = differential_evolution(
    objective_function,
    args=(np.cos(np.deg2rad(Cds_main.szen)), ratio),
    bounds=bounds,
    seed=1
)
result

 message: Optimization terminated successfully.
 success: True
     fun: 0.39807904993609633
       x: [-2.335e+00  4.402e+00 -2.437e+00  1.379e+00]
     nit: 56
    nfev: 3545
     jac: [ 2.380e-04  4.970e-04  7.820e-04  1.921e-04]

3.5. Results¶

print("Best cubic fit:")
a3, a2, a1, a0 = result.x
display(Latex(
    rf"""
    {a3:+.3f}x^3{a2:+.3f}x^2{a1:+.3f}x{a0:+.3f}
    """
))

Best cubic fit:

-2.335x^3+4.402x^2-2.437x+1.379

# Coefficients from other calibrations:

display(Latex(
    rf"""
    MelCol: {a3:+.3f}x^3{a2:+.3f}x^2{a1:+.3f}x{a0:+.3f}
    """
))

b3, b2, b1, b0 = [ -3.01, 5.59, -3.04, 1.45 ]
display(Latex(
    rf"""
    MetPVNet: {b3:+.3f}x^3{b2:+.3f}x^2{b1:+.3f}x{b0:+.3f}
    """
))
c3, c2, c1, c0 = [ -2.227, 4.366, -2.524, 1.385 ]
display(Latex(
    rf"""
    S2VSR: {c3:+.3f}x^3{c2:+.3f}x^2{c1:+.3f}x{c0:+.3f}
    """
))

MelCol: -2.335x^3+4.402x^2-2.437x+1.379

MetPVNet: -3.010x^3+5.590x^2-3.040x+1.450

S2VSR: -2.227x^3+4.366x^2-2.524x+1.385

szen = np.arange(1,80)
mu0 = np.cos(np.deg2rad(szen))

fig,ax = plt.subplots(1,1)
p = ax.plot(Cds_main.szen,Cds_main.reference_Wm2/Cds_main.pyrnet_Wm2,ls='',marker='.')
ax.plot(szen, a3*mu0**3 + a2*mu0**2 + a1*mu0 + a0,
         color='C1', label=f'best-cubic-fit: {a3:+.3f}x^3{a2:+.3f}x^2{a1:+.3f}x{a0:+.3f}')
ax.plot(szen, b3*mu0**3 + b2*mu0**2 + b1*mu0 + b0,
         color='C2', label=f'MetPvNet: {b3:+.3f}x^3{b2:+.3f}x^2{b1:+.3f}x{b0:+.3f}')
ax.plot(szen, c3*mu0**3 + c2*mu0**2 + c1*mu0 + c0,
         color='C3', label=f'S2VSR: {c3:+.3f}x^3{c2:+.3f}x^2{c1:+.3f}x{c0:+.3f}')

ax.legend()
ax.set_xlabel('Zenith angle [deg] ')
ax.set_ylabel('reference_ghi/pyr_ghi // correction_factor [-]')
ax.grid(True)

png

# dump to json
calibjson = {"2015-05-06": {"CC":list(result.x[::-1])}}
with open("pyrnet_calib_cc_new.json","w") as txt:
    json.dump(calibjson, txt)

3. 2015 MelCol¶

3.1. Imports¶

3.2. Prepare PyrNet data¶

3.3. Configuration¶

3.3.1. Load reference MORDOR data¶

3.3.2. Load PyrNet Data¶