Skip to content
Snippets Groups Projects
Commit 66ac9d8d authored by Lene Wasskog's avatar Lene Wasskog
Browse files

feat: First version of ADASMELIAE, with pipeline config

parent 4cd15b4e
No related branches found
No related tags found
No related merge requests found
...@@ -3,15 +3,11 @@ ...@@ -3,15 +3,11 @@
languages=en,nb languages=en,nb
[i18n.en] [i18n.en]
no_risk = No infection risk
low_risk = Low infection risk low_risk = Low infection risk
medium_risk = Medium infection risk
high_risk = High infection risk high_risk = High infection risk
whs = Wet Hour Sum (0-72) temperature = Temperature
[i18n.nb] [i18n.nb]
no_risk = Ingen infeksjonsrisiko
low_risk = Lav infeksjonsrisiko low_risk = Lav infeksjonsrisiko
medium_risk = Middels infeksjonsrisiko
high_risk = Høy infeksjonsrisiko high_risk = Høy infeksjonsrisiko
whs = Sum fuktige timer (0-72) temperature = Temperatur
\ No newline at end of file \ No newline at end of file
This diff is collapsed.
...@@ -6,18 +6,21 @@ This model is based on the work of Ferguson et al., (2015) to predict migration ...@@ -6,18 +6,21 @@ This model is based on the work of Ferguson et al., (2015) to predict migration
Input data for opprinnelig modell: Input data for opprinnelig modell:
DSSInput_Parameters DSSInput_Parameters
- weatherData weatherData (required) - weatherData weatherData (required)
- int growthStage (required) - int growthStage (required)
- optionalData optionalData - optionalData optionalData
weatherData weatherData
- string timeStart
- string timeEnd - string timeStart
- int Interval - string timeEnd
- List<int> weatherParameters - int Interval
- List<int> weatherParameters
- List<locationWeatherData> LocationWeatherData z - List<locationWeatherData> LocationWeatherData z
locationWeatherData locationWeatherData
- double longitude - double longitude
- double latitude - double latitude
- double altitude - double altitude
...@@ -28,36 +31,39 @@ locationWeatherData ...@@ -28,36 +31,39 @@ locationWeatherData
- int length - int length
optionalData optionalData
- double temp_threshold - double temp_threshold
- DateTime? startDate - DateTime? startDate
- DateTime? endDate - DateTime? endDate
Se MELIAEController.cs for oppstart av applikasjonen Se MELIAEController.cs for oppstart av applikasjonen
Se MELIAE_DSS.cs for kjøring av modellen Se MELIAE_DSS.cs for kjøring av modellen
## Technical description ## Technical description
The model has been implemented by [Tor-Einar Skog](https://nibio.no/en/employees/tor-einar-skog), [NIBIO](https://nibio.no/en). It is designed to fit into the gridded pest prediction models of [VIPS](https://nibio.no/en/services/vips). The model has been implemented by [Tor-Einar Skog](https://nibio.no/en/employees/tor-einar-skog), [NIBIO](https://nibio.no/en). It is designed to fit into the gridded pest prediction models of [VIPS](https://nibio.no/en/services/vips).
### Software requirements ### Software requirements
The model can only be run on Linux, as some of the tools mentioned below are only available on Linux. The development and testing of the model has been done using [Ubuntu Linux 22.04LTS](https://ubuntu.com/). The model can only be run on Linux, as some of the tools mentioned below are only available on Linux. The development and testing of the model has been done using [Ubuntu Linux 22.04LTS](https://ubuntu.com/).
#### CDO and GDAL #### CDO and GDAL
The heavy lifting in this model is done by the tools [CDO](https://code.mpimet.mpg.de/projects/cdo) and [GDAL](https://gdal.org/). These tools need to be installed and available. CDO is only available on Linux. The heavy lifting in this model is done by the tools [CDO](https://code.mpimet.mpg.de/projects/cdo) and [GDAL](https://gdal.org/). These tools need to be installed and available. CDO is only available on Linux.
#### Python requirements #### Python requirements
The Python requirements are specified in `requirements.txt` file, and are included in the virtualenv created by the `run_ADASMELIAE.sh` (see below). The Python requirements are specified in `requirements.txt` file, and are included in the virtualenv created by the `run_ADASMELIAE.sh` (see below).
### Input data requirements ### Input data requirements
The model (as per 2023-10-25) assumes that weather data files named `met_1_0km_nordic-[YYYY-MM-DD].nc` are available in the `in/` folder. The files must contain hourly timesteps with the following weather parameters: The model (as per 2023-10-25) assumes that weather data files named `met_1_0km_nordic-[YYYY-MM-DD].nc` are available in the `in/` folder. The files must contain hourly timesteps with the following weather parameters:
* RR (Rainfall in mm) - RR (Rainfall in mm)
* UM (Relative humidity in %) - UM (Relative humidity in %)
### Running the model ### Running the model
It is required that you have set the following environment variables: It is required that you have set the following environment variables:
```bash ```bash
...@@ -68,9 +74,9 @@ HOME_DIR=/home/foo/2023_vips_in_space/ ...@@ -68,9 +74,9 @@ HOME_DIR=/home/foo/2023_vips_in_space/
# Path to the weather data # Path to the weather data
WEATHER_DATA_DIR=in/ WEATHER_DATA_DIR=in/
# Used for iterating the weather data files # Used for iterating the weather data files
FILENAME_PATTERN="met_1_0km_nordic-*.nc" WEATHER_DATA_FILENAME_PATTERN="met_1_0km_nordic-*.nc"
# Used to extract date info from the filename # Used to extract date info from the filename
FILENAME_DATEFORMAT="met_1_0km_nordic-%Y-%m-%d.nc" WEATHER_DATA_FILENAME_DATEFORMAT="met_1_0km_nordic-%Y-%m-%d.nc"
# Names of weather parameters in NetCDF files # Names of weather parameters in NetCDF files
# Hourly precipitation # Hourly precipitation
RR="RR" RR="RR"
...@@ -78,7 +84,7 @@ RR="RR" ...@@ -78,7 +84,7 @@ RR="RR"
UM="UM" UM="UM"
# Timezone for weather data/daily aggregations # Timezone for weather data/daily aggregations
LOCAL_TIMEZONE="Europe/Oslo" LOCAL_TIMEZONE="Europe/Oslo"
# Path to optional CSV file with polygons for masking result. # Path to optional CSV file with polygons for masking result.
MASK_FILE=Norge_landomrader.csv MASK_FILE=Norge_landomrader.csv
# Path to the output (GeoTIFF) files as seen from the running model code # Path to the output (GeoTIFF) files as seen from the running model code
DATA_DIR=out/ DATA_DIR=out/
...@@ -101,9 +107,11 @@ MAPSERVER_EXTENT="-1.5831861262936526 52.4465003983706595 39.2608060398730458 71 ...@@ -101,9 +107,11 @@ MAPSERVER_EXTENT="-1.5831861262936526 52.4465003983706595 39.2608060398730458 71
```bash ```bash
$ ./run_ADASMELIAE.sh $ ./run_ADASMELIAE.sh
``` ```
This creates a Python virtualenv, installs all the Python dependencies, runs the model and stores output in a log file.
This creates a Python virtualenv, installs all the Python dependencies, runs the model and stores output in a log file.
Alternatively, primarily for development purposes, you can run the Python script ADASMELIAE directly: Alternatively, primarily for development purposes, you can run the Python script ADASMELIAE directly:
```bash ```bash
$ ./ADASMELIAE.py $ ./ADASMELIAE.py
``` ```
...@@ -114,28 +122,23 @@ All intermediary files are stored in the `tmp/` folder, and they are all deleted ...@@ -114,28 +122,23 @@ All intermediary files are stored in the `tmp/` folder, and they are all deleted
The model outputs GeoTIFF files, two per day in the season/period of calculation: The model outputs GeoTIFF files, two per day in the season/period of calculation:
* `result_WARNING_STATUS_[YYYY-MM-DD].tif`, wich indicates infection risk of Septoria. - `result_WARNING_STATUS_[YYYY-MM-DD].tif`, wich indicates infection risk of Septoria.
* 0 = No infection risk (grey) - 0 = No infection risk (grey)
* 2 = Low infection risk (yellow) - 2 = Low infection risk (yellow)
* 3 = Medium infection risk (orange) - 3 = Medium infection risk (orange)
* 4 = High risk (red) - 4 = High risk (red)
* `result_[YYYY-MM-DD].tif`, which contains the wet hour sum (WHS) - which is the sum of wet hours for "yesterday", "today" and "tomorrow", relative to the current file date. - `result_[YYYY-MM-DD].tif`, which contains the wet hour sum (WHS) - which is the sum of wet hours for "yesterday", "today" and "tomorrow", relative to the current file date.
A [Jinja2](https://pypi.org/project/Jinja2/) template mapfile (for [Mapserver](https://mapserver.org/)) with separate layers (WARNING_STATUS and WHS) for each date is found in the `mapfile/` folder. A [Jinja2](https://pypi.org/project/Jinja2/) template mapfile (for [Mapserver](https://mapserver.org/)) with separate layers (WARNING_STATUS and WHS) for each date is found in the `mapfile/` folder.
Examples of the two layers are shown below Examples of the two layers are shown below
![WARNING_STATUS example. Showing Norway and surrounding area](./WARNING_STATUS_layer_example.png) ![WARNING_STATUS example. Showing Norway and surrounding area](./WARNING_STATUS_layer_example.png)
*WARNING_STATUS example. Showing Norway* _WARNING_STATUS example. Showing Norway_
![WHS (Wet Hour Sum) example. Showing Norway and surrounding area](./WHS_layer_example.png) ![WHS (Wet Hour Sum) example. Showing Norway and surrounding area](./WHS_layer_example.png)
*WHS (Wet Hour Sum) example. Showing Norway* _WHS (Wet Hour Sum) example. Showing Norway_
## Notes to self ## Notes to self
File added
File added
File added
...@@ -7,9 +7,9 @@ HOME_DIR=/home/foo/2023_vips_in_space/ADASMELIAE/ ...@@ -7,9 +7,9 @@ HOME_DIR=/home/foo/2023_vips_in_space/ADASMELIAE/
# Path to the weather data # Path to the weather data
WEATHER_DATA_DIR=in/ WEATHER_DATA_DIR=in/
# Used for iterating the weather data files # Used for iterating the weather data files
FILENAME_PATTERN="met_1_0km_nordic-*.nc" WEATHER_DATA_FILENAME_PATTERN="met_1_0km_nordic-*.nc"
# Used to extract date info from the filename # Used to extract date info from the filename
FILENAME_DATEFORMAT="met_1_0km_nordic-%Y-%m-%d.nc" WEATHER_DATA_FILENAME_DATEFORMAT="met_1_0km_nordic-%Y-%m-%d.nc"
# Names of weather parameters in NetCDF files # Names of weather parameters in NetCDF files
# Hourly precipitation # Hourly precipitation
RR="RR" RR="RR"
......
<!--mapserver template--> <!--mapserver template-->
<?xml version="1.0" encoding="UTF-8"?> <?xml version="1.0" encoding="UTF-8"?>
<vipsResult> <vipsResult>
<modelName value="Referansefuktmodell"/> <modelName value="Pollen beetle migration model"/>
<modelId value="ADASMELIAE"/> <modelId value="ADASMELIAE"/>
<warningStatus value="[value_0]"/> <warningStatus value="[value_0]"/>
</vipsResult> </vipsResult>
\ No newline at end of file
...@@ -3,5 +3,5 @@ ...@@ -3,5 +3,5 @@
<vipsResult> <vipsResult>
<modelName value="Pollen beetle migration model"/> <modelName value="Pollen beetle migration model"/>
<modelId value="ADASMELIAE"/> <modelId value="ADASMELIAE"/>
<parameter name="WHS" value="[value_0]"/> <parameter name="air_temperature_2m" value="[value_0]"/>
</vipsResult> </vipsResult>
\ No newline at end of file
...@@ -35,54 +35,44 @@ WEB ...@@ -35,54 +35,44 @@ WEB
# List of standard metadata: https://mapserver.org/ogc/wms_server.html#web-object-metadata # List of standard metadata: https://mapserver.org/ogc/wms_server.html#web-object-metadata
# i18n support: https://mapserver.org/ogc/inspire.html#inspire-multi-language-support # i18n support: https://mapserver.org/ogc/inspire.html#inspire-multi-language-support
METADATA METADATA
"wms_keywordlist" "VIPS model Septoria Reference Humidity Model (ADASMELIAE)" "wms_keywordlist" "Pollen Beetle Migration Model (ADASMELIAE)"
{% if languages %} {% if languages %}
"wms_inspire_capabilities" "embed" "wms_inspire_capabilities" "embed"
"wms_languages" "{{ language_codes|join(",")}}" # The first is the default "wms_languages" "{{ language_codes|join(",")}}" # The first is the default
{% endif %} {% endif %}
"wms_abstract" "<div id='preamble'> "wms_abstract" "<div id='preamble'>
<p>The reference humidity model was developed as a supplement to <p>Pollen beetle (Meligethes spp.) adults are approximately 2.5 mm,
the Humidity model. In this model 20 consecutive hours are metallic greenish-black. Females bite oilseed rape buds and lay their eggs
required to fulfil a risk period. One constraint in this inside. Adults and larvae attack buds and flowers, resulting in withered
method is that you can have 19 consecutive risk hours or, buds and reduced pod set. In oilseed rape, adult and larval feeding can
fx 14 hours with risk then one hour below the Rh threshold lead to bud abortion and reduced pod set. However, damage rarely results
and then maybe 14 hours again with risk hours. In one of in reduced yields for winter crops. Spring crops are more vulnerable, as
these situations, the model will indicate a risk. In the the susceptible green/yellow bud stage often coincides with beetle
reference model the definition of Humid hours was introduced. migration. </p>
The Rh threshold was avoided as humid hours do not need to be
consecutive. The running sum of humid hours across three days
indicate that the Septoria risk is higher than if you have
three days with humid conditions than two or one. The operation
of the model should include weather forecast data and should
run 6 days ahead from current day if you include a 7-day weather
forecast (60 hours from national met office and until 7 days from ECMWF)</p>
</div> </div>
<div id='body'> <div id='body'>
<p> <p>
The model was tested against the Humidity model in a Danish Oilseed rape is only vulnerable if large numbers of pollen
Septoria project funded by the GUDP. The threshold of 40 was beetle migrate into the crop during green bud stage. This DSS predicts
defined as high risk as this coincided with periods when the migration into crops based on air temperature, and so can be used to
humidity model recommended to apply fungicide (if not already protected). evaluate risk to crop. Daily maximum air temperature is used to predict Migration
The humidity model includes a decision model about when to spray, Risk. The default value of 15 degrees celsius is used, as that is the
protection periods ect. The reference model was used to quality temperature advised in the UK at which pollen beetles will fly.
control the recommendations in a way that, if the reference humidity </p>
hours were higher than 40 (no thresholds) then the user should <p>This DSS was adapted from work carried out in the UK, and is
check the raw data for calculation of the Humidity model (threshold, considered applicable, but not yet validated in, Belgium, Luxembourg,
20 consecutive hours). If 2-3 periods of 17, 18, or 19 consecutive Netherlands, France, Germany, Rep. Ireland, and Denmark. Only to be used during Oilseed rape growth stages 51-59. This
hours appear, then one can consider to protect the crop based on the model is a simplification of a more detailed model described in Ferguson et al. (2015) Pest Management Science 72, 609-317.
reference model alone.</p> <a href="https://doi.org/10.1002/ps.4069">https://doi.org/10.1002/ps.4069</a></p>
<p>The Humidity model is considered as a DSS with several components.
The reference humidity model is considered as a weather based submodel
for the risk of Septoria, Easy to map and calculate based on weather data alone.</p>
<h3>Explanation of parameters</h3> <h3>Explanation of parameters</h3>
<ul> <ul>
<li>WHS = <span itemprop=\"WHS\">Wet hour sum</span></li> <li>TM_MAX = <span itemprop=\"TM_MAX\">Maximum Air Temperature at 2m</span></li>
</ul> </ul>
</div> </div>
" "
"wms_enable_request" "*" "wms_enable_request" "*"
"wms_title" "Septoria Reference Humidity Model" "wms_title" "Pollen Beetle Migration Model"
"wms_getfeatureinfo_formatlist" "text/plain,text/html,text/xml" "wms_getfeatureinfo_formatlist" "text/plain,text/html,text/xml"
"wms_accessconstraints" "none" "wms_accessconstraints" "none"
"wms_addresstype" "" "wms_addresstype" ""
...@@ -124,27 +114,17 @@ LAYER ...@@ -124,27 +114,17 @@ LAYER
STATUS ON STATUS ON
METADATA METADATA
"wms_title" "Reference humidity model {{ timestep_date }}" "wms_title" "Pollen Beetle Migration Model {{ timestep_date }}"
{% for language in languages %} {% for language in languages %}
"wms_abstract.{{language.language_code}}" " "wms_abstract.{{language.language_code}}" "
{ {
\"isWarningStatus\": true, \"isWarningStatus\": true,
\"legendItems\": [ \"legendItems\": [
{
\"classification\": 0,
\"legendLabel\": \"{{ language.no_risk }}\",
\"legendIconCSS\": \"width: 25px; background-color: #707070;\"
},
{ {
\"classification\": 2, \"classification\": 2,
\"legendLabel\": \"{{ language.low_risk }}\", \"legendLabel\": \"{{ language.low_risk }}\",
\"legendIconCSS\": \"width: 25px; background-color: #FFCC00;\" \"legendIconCSS\": \"width: 25px; background-color: #FFCC00;\"
}, },
{
\"classification\": 3,
\"legendLabel\": \"{{ language.medium_risk }}\",
\"legendIconCSS\": \"width: 25px; background-color: #FFCC99;\"
},
{ {
\"classification\": 4, \"classification\": 4,
\"legendLabel\": \"{{ language.high_risk }}\", \"legendLabel\": \"{{ language.high_risk }}\",
...@@ -159,30 +139,16 @@ LAYER ...@@ -159,30 +139,16 @@ LAYER
# class using simple string comparison, equivalent to ([pixel] = 0) # class using simple string comparison, equivalent to ([pixel] = 0)
CLASS
NAME "No infection risk"
EXPRESSION ([pixel] >= 0 AND [pixel] < 2)
STYLE
COLOR 112 112 112
END
END
CLASS CLASS
NAME "Low infection risk" NAME "Low infection risk"
EXPRESSION ([pixel] >= 2 AND [pixel] < 3) EXPRESSION ([pixel] < 3)
STYLE STYLE
COLOR 255 204 0 COLOR 112 112 112
END
END
CLASS
NAME "Medium infection risk"
EXPRESSION ([pixel] >= 3 AND [pixel] < 4)
STYLE
COLOR 255 153 0
END END
END END
CLASS CLASS
NAME "High infection risk" NAME "High infection risk"
EXPRESSION ([pixel] >= 4) EXPRESSION ([pixel] >= 3)
STYLE STYLE
COLOR 255 0 0 COLOR 255 0 0
END END
...@@ -190,25 +156,25 @@ LAYER ...@@ -190,25 +156,25 @@ LAYER
END # Layer END # Layer
LAYER LAYER
NAME "{{model_id}}.WHS.{{ timestep_date }}" NAME "{{model_id}}.temperature.{{ timestep_date }}"
DATA "{{mapserver_data_dir}}result_{{ timestep_date }}.tif" DATA "{{mapserver_data_dir}}result_{{ timestep_date }}.tif"
TEMPLATE "{{mapserver_mapfile_dir}}query_template_WHS.xml" TOLERANCE 1 TOLERANCEUNITS PIXELS TEMPLATE "{{mapserver_mapfile_dir}}query_template_temperature.xml" TOLERANCE 1 TOLERANCEUNITS PIXELS
TYPE RASTER TYPE RASTER
#PROCESSING "BANDS=1" # WHS band on top (others invisible, but band values are available in the query template) #PROCESSING "BANDS=1" # Temperature band on top (others invisible, but band values are available in the query template)
#PROCESSING "SCALE=AUTO" #PROCESSING "SCALE=AUTO"
#PROCESSING "NODATA=-1" #PROCESSING "NODATA=-1"
STATUS ON STATUS ON
METADATA METADATA
"wms_title" "Reference humidity model WHS {{ timestep_date }}" "wms_title" "Pollen Beetle Migration Model temperature {{ timestep_date }}"
{% for language in languages %} {% for language in languages %}
"wms_abstract.{{language.language_code}}" " "wms_abstract.{{language.language_code}}" "
{ {
\"isWarningStatus\": false, \"isWarningStatus\": false,
\"legendItems\": [ \"legendItems\": [
{ {
\"legendLabel\": \"{{ language.whs }}\", \"legendLabel\": \"{{ language.temperature }}\",
\"legendIconCSS\": \"width: 25px; background: linear-gradient(to right, #FFFF00, #0000FF);\" \"legendIconCSS\": \"width: 25px; background: linear-gradient(to right, #FFFF00, #0000FF);\"
} }
] ]
...@@ -218,10 +184,10 @@ LAYER ...@@ -218,10 +184,10 @@ LAYER
END END
CLASSITEM "[pixel]" CLASSITEM "[pixel]"
CLASS CLASS
NAME "Wet hour sum (yesterday + today + tomorrow) [0-72]" NAME "Temperature range"
EXPRESSION ([pixel] >= 0 AND [pixel] <= 72) EXPRESSION ([pixel] >= -40 AND [pixel] <= 40)
STYLE STYLE
DATARANGE 0 72 DATARANGE -40 40
COLORRANGE 255 255 0 0 0 255 COLORRANGE 255 255 0 0 0 255
END END
END END
......
Jinja2 Jinja2
netCDF4 netCDF4
pytz pytz
python-dotenv python-dotenv
\ No newline at end of file pytest
\ No newline at end of file
import pytest
from datetime import datetime
from ADASMELIAE import find_start_date
@pytest.fixture
def example_result_files():
return [
"result_2023-04-15.nc",
"result_2023-04-16.nc",
"result_2023-04-17.nc",
"result_WARNING_STATUS_2023-04-15.nc",
"result_WARNING_STATUS_2023-04-16.nc",
"result_WARNING_STATUS_2023-04-17.nc",
"result_WARNING_STATUS_2023-04-18.nc",
]
def test_find_start_date_with_previous_results(example_result_files, monkeypatch):
MODEL_START_DATE = datetime(2023, 3, 1)
monkeypatch.setenv("DATA_DIR", "out")
# Mock os.listdir to return the example result files
with monkeypatch.context() as m:
m.setattr("os.listdir", lambda _: example_result_files)
start_date = find_start_date(MODEL_START_DATE)
# Assert the expected start date
expected_start_date = datetime(2023, 4, 18)
assert start_date == expected_start_date
def test_find_start_date_without_previous_results(monkeypatch):
MODEL_START_DATE = datetime(2023, 3, 1)
monkeypatch.setenv("DATA_DIR", "out")
# Mock os.listdir to return the example result files
with monkeypatch.context() as m:
m.setattr("os.listdir", lambda _: [])
start_date = find_start_date(MODEL_START_DATE)
# Assert the expected start date
assert start_date == MODEL_START_DATE
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment