OSVAS

Parameter Estimation Workflows - Implementation Summary

Overview

Complete workflows have been integrated into OSVAS to automatically estimate key surface parameters from observational data:

• Albedo Estimation (Step 2b): Estimates monthly surface albedo values (NIR and VIS) from validation radiation data
• LAI Estimation (Step 2c): Estimates monthly Leaf Area Index (LAI) from CGLS satellite data via openEO

Changes Made

1. Configuration Updates

Modified Files: Station YAML configuration files

• config_files/Stations/Cabauw/Cabauw.yml
• config_files/Stations/Loobos/Loobos.yml
• config_files/Stations/Majadas_del_tietar/Majadas_del_tietar.yml
• config_files/Stations/Meteopole/Meteopole.yml

Change: Added estimate_albedo: false flag to Station_metadata section

Station_metadata:
  ...
  vegtype: 10
  estimate_albedo: false  # Set to true to enable albedo estimation

2. Python Scripts

New File: scripts/python_scripts/estimate_albedo.py

Purpose: Core albedo estimation engine

Functionality:

• Reads SW_OUT/SW_IN from SQLite OBSTABLEs (validation data)
• Selects observations between 11:00-13:00 UTC daily (high solar elevation)
• Calculates daily albedos: α = SW_OUT / SW_IN
• Computes monthly averages from daily values
• Generates Fortran namelist format output
• Saves results to namelists/{station}/albedo_estimates.nam

Usage:

python3 scripts/python_scripts/estimate_albedo.py <station> <osvas_root> \
    --validation-period START_DATE END_DATE

Key processing:

• Physical QC: Removes unrealistic albedos (outside [0, 1])
• Data filtering: Only uses SW_IN > 0 (daylight) and non-null values
• Monthly aggregation: Simple average of valid daily values
• Handles missing data: Uses annual mean if month has no valid data

New File: scripts/python_scripts/update_namelist_albedos.py

Purpose: Apply estimated albedos to SURFEX namelists

Functionality:

• Reads estimated albedo namelist block
• Locates existing albedo parameter ranges in experiment namelists
• Extracts new blocks from estimates
• Replaces old blocks with new values (preserves indentation)
• Creates backup copies of original namelists

Usage:

python3 scripts/python_scripts/update_namelist_albedos.py <station> <osvas_root> \
    --expnames EXP1 EXP2 ... [--no-backup]

Parameters Updated:

• XUNIF_ALBNIR_VEG(10,1-12) - Vegetation NIR albedo (12 monthly values)
• XUNIF_ALBVIS_VEG(10,1-12) - Vegetation VIS albedo (12 monthly values)
• XUNIF_ALBNIR_SOIL(10,1-12) - Soil NIR albedo (12 monthly values)
• XUNIF_ALBVIS_SOIL(10,1-12) - Soil VIS albedo (12 monthly values)
• (Vegtype index is read from Station_metadata)

3. Workflow Integration

Modified File: scripts/python_scripts/surfex_OSVAS_run_linux.py

Change: Added Step 2b: Albedo Estimation

• Executes after Step 2 (Get validation data)
• Checks estimate_albedo flag in Station_metadata
• Calls estimate_albedo.py with validation period from config
• Calls update_namelist_albedos.py to update experiment namelists
• Graceful error handling: continues with original namelists on failure

Code Location: Lines ~137-170 (after “Step 2b: Estimate albedos from validation data (if enabled)”)

# Step 2b: Estimate albedos from validation data (if enabled)
estimate_albedo = config.get('Station_metadata', {}).get('estimate_albedo', False)
if estimate_albedo and get_validation:
    print("▶ Running Step 2b: Estimate surface albedos from validation data")
    # ... (validation period extraction and script execution)

Modified File: scripts/python_scripts/surfex_OSVAS_run_atos.py

Change: Added identical Step 2b for HPC execution

• Same functionality as Linux version
• Enables albedo estimation on ATOS HPC systems
• Consistent behavior across platforms

Code Location: Lines ~138-171 (same structure as Linux version)

4. Documentation

New File: docs/step2b_albedo_estimation.md

Content:

• Complete technical documentation (1000+ lines)
• Physical basis and methodology
• Configuration instructions
• Output files description
• Manual execution guide
• Validation and QC procedures
• Troubleshooting guide
• Advanced usage patterns
• References

Sections:

1. Overview & motivation
2. Physical basis (albedo calculation)
3. Configuration (Station YAML)
4. Workflow integration
5. Output files
6. Manual execution (with examples)
7. Validation and quality control
8. Troubleshooting
9. Advanced usage (customization options)
10. References

New File: docs/albedo_estimation_quickref.md

Content: Quick reference guide for users

• TL;DR setup (3 steps)
• What gets computed
• Output files
• Key parameters
• Troubleshooting table
• Link to full documentation

5. LAI Estimation Scripts and Documentation

New File: scripts/python_scripts/estimate_lai.py

Purpose: Core LAI estimation engine using CGLS satellite data

Functionality:

• Connects to openEO backend (Copernicus Data Space Ecosystem)
• Fetches CGLS LAI using BIOPAR process (VITO Algorithm Plaza)
• Retrieves 10-day composite LAI observations at 300 m resolution
• Selects valid observations within bounding box around station (±0.005°)
• Computes monthly climatology from all 10-day values per month
• Handles gap-filling using nearest-neighbor interpolation for missing months
• Generates Fortran namelist format output
• Saves results to namelists/{station}/lai_estimates.nam

Usage:

python3 scripts/python_scripts/estimate_lai.py <station> <osvas_root> \
    --run-period START_DATE END_DATE [OPTIONS]

Key features:

• OIDC authentication (device-code flow, cached credentials)
• Multi-year climatology support: --start-year 2017 --end-year 2022
• Physical LAI units directly from CGLS (no scale factor)
• Gap-filling for months with no observations
• Quality control: filters unrealistic values
• Batch processing by year to manage openEO job sizes

Parameters Updated:

• XUNIF_LAI(vegtype,1-12) - Monthly LAI climatology [m²/m²]

New File: scripts/python_scripts/update_namelist_lais.py

Purpose: Apply estimated LAI values to SURFEX namelists

Functionality:

• Reads estimated LAI namelist block
• Locates existing or seeks insertion point for LAI parameters in experiment namelists
• Replaces old LAI blocks with new values or inserts into &NAM_DATA_ISBA group
• Creates backup copies of original namelists

Usage:

python3 scripts/python_scripts/update_namelist_lais.py <station> <osvas_root> \
    --expnames EXP1 EXP2 ... [OPTIONS]

Features:

• Handles namelists with or without existing LAI blocks
• Preserves formatting and indentation
• Creates backup files with .backup_lai suffix
• Inserts into &NAM_DATA_ISBA if block doesn’t exist

Modified Files: Station YAML Configuration

Change: Added estimate_lai: false flag to Station_metadata section

Station_metadata:
  ...
  vegtype: 10
  estimate_lai: false   # Set to true to enable LAI estimation

Files updated:

• config_files/Stations/Cabauw/Cabauw.yml
• config_files/Stations/Loobos/Loobos.yml
• config_files/Stations/Majadas_del_tietar/Majadas_del_tietar.yml
• config_files/Stations/Meteopole/Meteopole.yml

Modified Workflow Scripts

surfex_OSVAS_run_linux.py: Added Step 2c: LAI Estimation

• Executes after Step 2b (Albedo Estimation)
• Checks estimate_lai flag in Station_metadata
• Calls estimate_lai.py with run period from Forcing_data config
• Calls update_namelist_lais.py to update experiment namelists
• Graceful error handling: continues with original namelists on failure
• Optional OIDC authentication for first run

surfex_OSVAS_run_atos.py: Added identical Step 2c for HPC execution

• Same functionality as Linux version
• Consistent behavior across platforms
• Supports batch processing multiple stations

New File: docs/step2c_lai_estimation.md

Content:

• Complete technical documentation (800+ lines)
• Physical basis and methodology
• Configuration instructions with authentication guide
• Output files description
• Manual execution guide with multiple examples
• Validation and QC procedures
• Troubleshooting guide
• Advanced usage patterns (multi-year climatology, pre-computed values)
• References to CGLS and openEO documentation

Sections:

1. Overview & motivation
2. Physical basis (LAI definition and sources)
3. Configuration (Station YAML, OIDC authentication)
4. Workflow integration
5. Output files
6. Manual execution (with examples)
7. Validation and quality control
8. Troubleshooting
9. Advanced usage
10. References

New File: docs/lai_estimation_quickref.md

Content: Quick reference guide for LAI estimation

• TL;DR setup (3 steps including authentication)
• What gets computed
• Output files
• Key parameters and typical LAI ranges
• Troubleshooting table
• Manual command examples
• Link to full documentation

Workflow Sequence

┌─────────────────────────────────────────────────────────────┐
│ OSVAS Workflow (Updated)                                    │
└─────────────────────────────────────────────────────────────┘

Step 1: Create forcing data
        | (Load atmospheric ICOS data)
        ▼
Step 2: Get validation data
        | (Download & process flux data → OBSTABLEs)
        ▼
Step 2b: ✨ Estimate albedos (NEW!)
        │
        ├─ Read SW_OUT/SW_IN from OBSTABLEs
        ├─ Select midday obs (11:00-13:00 UTC)
        ├─ Calculate daily albedos (SW_OUT/SW_IN)
        ├─ Compute monthly averages
        ├─ Generate namelist blocks → albedo_estimates.nam
        └─ Update experiment namelists with new albedos
        │   └─ Create backups (OPTIONS.nam.backup)
        ▼
Step 2c: ✨ Estimate LAI (NEW!)
        │
        ├─ Connect to openEO (Copernicus Data Space)
        ├─ Fetch CGLS LAI via BIOPAR process
        ├─ Compute monthly climatology from 10-day data
        ├─ Generate namelist blocks → lai_estimates.nam
        └─ Update experiment namelists with new LAI
        │   └─ Create backups (OPTIONS.nam.backup_lai)
        ▼
Step 3: Run SURFEX simulations
        | (With updated albedos and LAI!)
        ├─ PGD: Physiography
        ├─ PREP: Initialization
        └─ OFFLINE: Main simulation
        ▼
Step 4: Extract model outputs
        | (Convert NetCDF → SQLite FCTABLES)
        ▼
Step 5: HARP verification
        | (Compare model vs observations)
        ▼
Step 6: Visualization
        └─ Display results

Enabling/Disabling

Enable both Albedo and LAI estimation for a station:

Station_metadata:
  ...
  estimate_albedo: true
  estimate_lai: true

Then run normal OSVAS workflow:

python3 scripts/python_scripts/surfex_OSVAS_run_linux.py

Enable only Albedo (skip LAI):

Station_metadata:
  ...
  estimate_albedo: true
  estimate_lai: false

Enable only LAI (skip Albedo):

Station_metadata:
  ...
  estimate_albedo: false
  estimate_lai: true

Disable both (uses original namelists):

Station_metadata:
  ...
  estimate_albedo: false
  estimate_lai: false

Execution Flow

Automatic (via workflow launcher):

surfex_OSVAS_run_linux.py
    │
    ├─ Check estimate_albedo flag
    │
    ├─ if true:
    │   ├─ estimate_albedo.py
    │   │   └─ Output: namelists/{station}/albedo_estimates.nam
    │   │
    │   └─ update_namelist_albedos.py
    │       ├─ Read albedo_estimates.nam
    │       ├─ Update OPTIONS.nam_{expname}
    │       └─ Create .backup files
    │
    ├─ Check estimate_lai flag
    │
    └─ if true:
        ├─ estimate_lai.py
        │   ├─ Connect to openEO (OIDC auth if needed)
        │   └─ Output: namelists/{station}/lai_estimates.nam
        │
        └─ update_namelist_lais.py
            ├─ Read lai_estimates.nam
            ├─ Update OPTIONS.nam_{expname}
            └─ Create .backup_lai files

Manual (for testing):

Albedo estimation:

# Step 1: Estimate albedos
python3 scripts/python_scripts/estimate_albedo.py Cabauw $OSVAS \
    --validation-period 2017-11-01 2018-01-31

# Step 2: Update namelists
python3 scripts/python_scripts/update_namelist_albedos.py Cabauw $OSVAS \
    --expnames DIFMEB_v9 DIFMEB_v9DSL

LAI estimation:

# Step 1: Estimate LAI (requires authentication on first run)
python3 scripts/python_scripts/estimate_lai.py Cabauw $OSVAS \
    --run-period 2017-11-01 2018-01-31

# Step 2: Update namelists
python3 scripts/python_scripts/update_namelist_lais.py Cabauw $OSVAS \
    --expnames DIFMEB_v9 DIFMEB_v9DSL

Output Examples

Albedo Estimates File

namelists/Cabauw/albedo_estimates.nam

! Vegetation NIR albedo (monthly)
XUNIF_ALBNIR_VEG(10,1)  = 0.19024633,
XUNIF_ALBNIR_VEG(10,2)  = 0.18576372,
...
XUNIF_ALBNIR_VEG(10,12) = 0.1538921,

! Vegetation VIS albedo (monthly)
XUNIF_ALBVIS_VEG(10,1)  = 0.19024633,
...

! Soil albedos (same format for NIR and VIS)
...

LAI Estimates File

namelists/Cabauw/lai_estimates.nam

! XUNIF_LAI estimates from CGLS LAI 300m (openEO/Copernicus Data Space)
! Location  : lat=51.9703, lon=4.9264
! Period    : 2017-11-01 – 2018-01-31
! Generated by estimate_lai.py
XUNIF_LAI(10, 1)  = 0.521,
XUNIF_LAI(10, 2)  = 0.412,
XUNIF_LAI(10, 3)  = 0.634,
...
XUNIF_LAI(10,12)  = 0.387,

Script Output

Albedo Estimation: Found 2 OBSTABLE files Loaded 8760 records with valid SW_OUT/SW_IN

Step 2: Selecting midday observations (11:00-13:00 UTC) Selected 1460 records between 11:00-13:00 UTC

Step 3: Calculating daily albedos (SW_OUT/SW_IN) Calculated 365 daily average albedos

Step 4: Computing monthly averages Monthly albedo averages: Month 1: 0.28905634 Month 2: 0.29123456 …

Step 5: Generating namelist format Using vegtype: 10

Step 6: Saving results ✅ Albedo estimates saved to: namelists/Cabauw/albedo_estimates.nam NIR albedo: min=0.15635234, max=0.45123456, mean=0.28901234 VIS albedo: min=0.15635234, max=0.45123456, mean=0.28901234

====================================================================== ✅ Albedo estimation completed successfully ======================================================================

## Key Features

✅ **Automatic integration** - Executes as Step 2b in workflow (no manual intervention)
✅ **Data-driven** - Uses actual measurements from validation sites
✅ **Physical basis** - Albedo = SW_OUT/SW_IN (standard radiative transfer)
✅ **Robust QC** - Removes unrealistic values, handles missing data
✅ **Configurable** - Enable/disable per station via YAML flag
✅ **Flexible** - Manual scripts for batch processing or customization
✅ **Safe** - Creates backups of original namelists before updating
✅ **Detailed** - Comprehensive logging and error handling
✅ **Well-documented** - Full docs + quick reference guide

## Requirements

- Python 3.6+
- pandas
- numpy
- sqlite3 (built-in)
- yaml (included in conda/pip environments)

## Backward Compatibility

- ✅ **Existing workflows unaffected** - Default is `estimate_albedo: false`
- ✅ **Optional feature** - Users opt-in per station
- ✅ **No changes to other steps** - Only adds Step 2b
- ✅ **Graceful failures** - Continues with original namelists if estimation fails

## Testing Recommendations

1. **Test with validation data available:**
   ```bash
   export STATION_NAME=Cabauw
   # Run with Get_validation=true and estimate_albedo=true
   python3 scripts/python_scripts/surfex_OSVAS_run_linux.py

1. Verify output files:

   ls -lh namelists/Cabauw/albedo_estimates.nam
   ls -lh namelists/Cabauw/OPTIONS.nam_*.backup
   

2. Compare albedos:

   # Before & after
   diff namelists/Cabauw/OPTIONS.nam_DIFMEB_v9.backup \
        namelists/Cabauw/OPTIONS.nam_DIFMEB_v9
   

3. Run SURFEX simulation:

   # Set Run_surfex: true in Station YAML
   python3 scripts/python_scripts/surfex_OSVAS_run_linux.py
   # Verify simulation completes successfully
   

Future Enhancements

Possible improvements for future releases:

1. Differentiate NIR vs VIS - Use spectral decomposition from multi-band radiometers
2. Separate soil & vegetation - Use vegetation index or LST to distinguish surfaces
3. Temporal smoothing - Apply moving averages to reduce noise
4. Uncertainty quantification - Estimate albedo uncertainty from measurement variability
5. Seasonal trends - Fit harmonic functions to capture smooth seasonal cycles
6. Surface reflectance model - Account for bidirectional reflectance distribution function (BRDF)

Support

For Albedo Estimation issues:

1. Check docs/step2b_albedo_estimation.md troubleshooting section
2. Verify validation data quality: sqlite3 sqlites/OBSTABLES/validation_data/{station}/OBSTABLE_YYYY.sqlite "SELECT COUNT(*), COUNT(SW_OUT), COUNT(SW_IN) FROM SYNOP;"
3. Review Python script output for diagnostic messages
4. Check backups are created before namelist updates
5. Examine albedo_estimates.nam file format

For LAI Estimation issues:

1. Check docs/step2c_lai_estimation.md troubleshooting section
2. Verify Copernicus Data Space account is active (free registration at https://dataspace.copernicus.eu/)
3. Check OIDC authentication: device code should be printed on first run
4. Verify cached credentials in system (typically ~/.config/eodc/ or similar)
5. Review LAI values for expected seasonal patterns
6. Check openEO backend connectivity: curl -s https://openeofed.dataspace.copernicus.eu