{ "cells": [ { "cell_type": "markdown", "id": "b35a2847-3fc4-4135-b512-013eec5bfeb6", "metadata": {}, "source": [ "(sec-calibration-s2vsr)=\n", "# S2VSR 2023-06\n", "\n", "The PyrNet setup for [Small Scale Variability of Solar Radiation (S2VSR)](https://www.arm.gov/research/campaigns/sgp2023s2vsr) was setup for calibration on the ARM-SGP guest instrument facility (**GIF**) for calibration from 2023-06-02 to 2023-06-08. Cross-calibration is done versus reference observations from the ARM-SGP broadband radiometer station (**BRS**) ([doi:10.5439/1550918](http://dx.doi.org/10.5439/1550918))\n", "\n", "\n" ] }, { "cell_type": "markdown", "id": "d1e73058-68a6-4794-aa53-77b0cde30ac5", "metadata": {}, "source": [ "## Imports" ] }, { "cell_type": "code", "execution_count": 1, "id": "e187e5d7-bdd6-42f3-8e99-a2eee2a55cc9", "metadata": { "ExecuteTime": { "end_time": "2023-06-20T06:02:29.797824800Z", "start_time": "2023-06-20T06:02:26.719252900Z" }, "tags": [ "hide-input" ] }, "outputs": [], "source": [ "#|dropcode\n", "import os\n", "import xarray as xr\n", "import pandas as pd\n", "import numpy as np\n", "import datetime as dt\n", "import matplotlib.pyplot as plt\n", "import jstyleson as json\n", "\n", "from pyrnet import pyrnet" ] }, { "cell_type": "markdown", "id": "5a9f1c48-a626-4358-9e17-43d88b77b7ac", "metadata": {}, "source": [ "## Prepare PyrNet data\n", "For calibration preparation the PyrNet data is processed to level l1b using a calibration factor of **7 (uV W-1 m2)** for all pyranometers with the ```pyrnet process l1b``` tool. This is done to unify the conversion to sensor voltage during calibration and not run into valid_range limits for netcdf encoding. Here we generate the *calibration.json* file for the processing to l1b: " ] }, { "cell_type": "code", "execution_count": 2, "id": "592e4125-4dde-435b-9d71-acaf96a1d6b6", "metadata": { "ExecuteTime": { "end_time": "2023-06-20T06:02:29.802630200Z", "start_time": "2023-06-20T06:02:29.802126300Z" } }, "outputs": [], "source": [ "box_numbers = np.arange(1,101)\n", "calibrations = {f\"{bn:03d}\":[7,7] for bn in box_numbers}\n", "calibjson = {\"2000-01-01\": calibrations}\n", "# with open(\"pyrnet_calib_prep.json\",\"w\") as txt:\n", "# json.dump(calibjson, txt)" ] }, { "cell_type": "markdown", "id": "883d00f0-6946-4460-a7bb-d45d2f0e93ca", "metadata": {}, "source": [ "Within *pyrnet_config_calibration_prep.json*:\n", "```\n", "{\"file_calibration\" : \"pyrnet_calib_prep.json\"}\n", "```\n", "\n", "**Workflow for preparation**\n", "1. Prepare *pyrnet_config_calibration_prep.json* with contributors metadata and the dummy calibration config file.\n", "1. ```$ pyrnet process l1a -c pyrnet_config_calibration_prep.json raw_data/*.bin l1a/```\n", "1. ```$ pyrnet process l1b -c pyrnet_config_calibration_prep.json l1a/*.nc l1b/```\n", "1. ```$ d=2023-06-02; until [[ $d > 2023-06-08 ]]; do pyrnet merge l1b/pyrnet_${d}_*.nc l1b_network/pyrnet_${d}_s2vsr_calibration.c01.nc; d=$(date -I -d \"$d + 1 day\"); done```\n", "1. Upload the network files to the Thredds server" ] }, { "cell_type": "markdown", "id": "6a3703b0-d8c5-4859-80f0-02211758a8c5", "metadata": {}, "source": [ "## Configuration\n", "Data is available via Thredds server. The URLs might have changed when trying to rerun this notebook. \n", "\n", "For the ARM data, create a ARM user account first and then order the [doi:10.5439/1550918](http://dx.doi.org/10.5439/1550918), and choose the Thredds-Server option. You will receive the Thredds-URL via e-mail.\n", "\n", "The PyrNet data should be available via [tds.tropops.de](tds.tropos.de), if not, contact witthuhn@tropos.de.\n" ] }, { "cell_type": "code", "execution_count": 3, "id": "def5e995-bfa7-4de9-a142-04effdafed8d", "metadata": { "ExecuteTime": { "end_time": "2023-06-20T06:02:30.302208800Z", "start_time": "2023-06-20T06:02:29.815678800Z" } }, "outputs": [], "source": [ "ARM_URL = \"https://archive.arm.gov/orders/dodsC/orders/witthuhnj1/240295/sgpbrsC1.b1.{date:%Y%m%d}.000000.custom.cdf\"\n", "PYR_URL = \"https://tds.tropos.de/thredds/dodsC/scccJher/2023_s2vsr/calibration/pyrnet_{date:%Y-%m-%d}_s2vsr_calibration.c01.nc\"\n", "dates = pd.date_range(\"2023-06-02\",\"2023-06-08\")\n", "stations = np.arange(1,101)\n", "\n", "# lookup which box contains actually a pyranometer/ extra pyranometer\n", "mainmask, extramask = [] , []\n", "for box in stations:\n", " _, serials, _ = pyrnet.meta_lookup(dates[0],box=box)\n", " mainmask.append( True if len(serials[0])>0 else False )\n", " extramask.append( True if len(serials[1])>0 else False )" ] }, { "cell_type": "markdown", "id": "e09f6b19-50e9-49aa-a892-59296be96c43", "metadata": {}, "source": [ "### Load ARM-SGP Data" ] }, { "cell_type": "code", "execution_count": 4, "id": "023830e8-cad2-4239-b27a-c4c31de0dade", "metadata": { "ExecuteTime": { "end_time": "2023-06-20T06:02:47.424335Z", "start_time": "2023-06-20T06:02:30.311960Z" }, "tags": [ "hide-input", "hide-output" ] }, "outputs": [ { "data": { "text/html": [ "
<xarray.Dataset>\n",
"Dimensions: (time: 10080)\n",
"Coordinates:\n",
" * time (time) datetime64[ns] 2023-06-02 ... 2023-06-08T23:59:00\n",
"Data variables:\n",
" down_short_hemisp (time) float32 7.642 6.241 5.941 ... 165.9 165.4 170.4\n",
"Attributes: (12/26)\n",
" command_line: sirs_ingest -n brs -s sgp -f C1\n",
" Conventions: ARM-1.3\n",
" process_version: ingest-sirs-12.9-0.el7\n",
" dod_version: brs-b1-4.0\n",
" input_source: /data/collection/sgp/sgpbrsC1.00/BRS_Tab...\n",
" site_id: sgp\n",
" ... ...\n",
" lithium_battery: 3.37\n",
" low_12v_count: 4872.0\n",
" skipped_scan: 0.0\n",
" doi: 10.5439/1550918\n",
" history: created by user dsmgr on machine prod-pr...\n",
" DODS_EXTRA.Unlimited_Dimension: time<xarray.Dataset>\n",
"Dimensions: (station: 100, time: 604800)\n",
"Coordinates:\n",
" * station (station) int64 1 2 3 4 5 6 7 8 9 ... 92 93 94 95 96 97 98 99 100\n",
" * time (time) datetime64[ns] 2023-06-02 ... 2023-06-08T23:59:59\n",
"Data variables:\n",
" ghi (time, station) float64 nan nan nan nan nan ... nan nan nan nan nan\n",
" gti (time, station) float64 nan nan nan nan nan ... nan nan nan nan nan\n",
" szen (time, station) float64 nan nan nan nan nan ... nan nan nan nan nan\n",
"Attributes: (12/31)\n",
" title: TROPOS pyranometer network (PyrNet) observations\n",
" institution: Leibniz Institute for Tropospheric Research (T...\n",
" department: Remote Sensing of Atmospheric Processes\n",
" department_team: Clouds, Aerosol and Radiation\n",
" address: Permoser Str. 15, 04318 Leipzig, Germany\n",
" source: TROPOS pyranometer network (PyrNet)\n",
" ... ...\n",
" geospatial_lon_max: -97.48579382293494\n",
" geospatial_lon_units: degE\n",
" time_coverage_start: 2023-06-02T17:38:35\n",
" time_coverage_end: 2023-06-02T23:59:59\n",
" time_coverage_duration: P0DT6H21M24S\n",
" time_coverage_resolution: P0DT0H0M1S