Configuration

Configuration#

While dart-pipeline can process individual data sources, for convenience, we provide a config.sh that can be used by a set of Bash scripts to run the pipeline. This simplifies calling the pipeline by allowing all the options to be defined in one place. The configuration file has the following settings. that are used for data selection, resolution, and optional bias correction during pipeline execution.

Configuration Parameters#

ISO3

Three-letter ISO country code. Example: VNM (Vietnam)

ADMIN

Administrative level for spatial aggregation. Options: 1, 2, or 3

TEMPORAL_RESOLUTION

Time resolution for processing core climate parameters such as temperature (t2m), relative humidity (r), and maximum temperature (mx2t24). Options: daily, weekly Default: weekly Note: SPI and SPEI are always computed at a weekly resolution. At daily resolution, outputs are not concatenated into single files due to resolution mismatches.

START_YEAR

First year of the study period. Example: 2001

END_YEAR

Last year of the study period. Example: 2019 Note: Data is also fetched one year before and after this period to align with ISO week requirements.

INDEX_START_YEAR

Start year for SPI and SPEI gamma parameter estimation. Defaults to START_YEAR, can be overridden if necessary for statistical robustness.

INDEX_END_YEAR

End year for SPI and SPEI gamma parameter estimation. Defaults to END_YEAR.

BC_ENABLE

Enables or disables bias correction. Values: 1 (enabled), anything else disables it.

BC_PRECIP_REF

Reference dataset for total precipitation, used in bias correction. Example: Dataset such as REMOCLIC. 2016 that provides a reference precipitation dataset (for Vietnam)

BC_CLIP_PRECIP_PERCENTILE

Percentile at which reference data is clipped for precipitation bias correction. This is done to reduce the effect of outliers in bias correction using quantile mapping. Default: 0.99

BC_HISTORICAL_FORECAST

Historical forecast dataset from ECMWF MARS service. Notes: This is used for correcting forecast data. This data is not open-access.

BC_HISTORICAL_OBS

Historical observation dataset used for bias correction. Must include:

  • t2m: 2-meter air temperature

  • r: Relative humidity

  • tp: Total precipitation Notes: This can be generated using uv run dart-pipeline process era5.prep_bias_correct

Note that data is always fetched one year prior to and one year after the included period (_fetch_start_year and _fetch_end_year in the script). This is so that we can perform ISO week alignment for calculation of standardised indices such as SPI and SPEI.

Once the configuration is setup, individual scripts can be called as follows

bash scripts/scriptname.sh config.sh

This will read variables from the configuration file