<img src="illustrations/vipslogo_512.png" alt="VIPS Logo" height="250"/>
# VIPS Grid models
Tor-Einar Skog, Senior developer, NIBIO
Updated: 2023-12-20
## What you will learn
This document describes how to set up a VIPS compatible gridded DSS model, such as in the screenshot below. It also describes briefly [how to set up a client to a VIPS compatible DSS gridded model](#vips-grid-model-client).
The gridded models are using the same "traffic light" color pattern for indicating risk as the location based models. In addition, any other numeric parameter can be displayed as a separate layer, with its own legend and colour scheme.
<img src="illustrations/grid_models_01.png" alt="Grid models example" width="1024"/>
*The screenshot shows how results from the Carrot Rust Fly temperature model are presented in VIPS. You select date using the slider in the lower left corner, and you select parameter using the radio buttons in the lower right corner. By clicking in the map, a popup shows the warning status and values of any other parameters.*
## Prerequisites
* You should be familiar with web maps and [Web Map Services (WMS)](https://en.wikipedia.org/wiki/Web_Map_Service)
* You should be familiar with using and manipulating gridded weatherdata files, e.g. NetCDF files or GRIB files.
## Architecture of the VIPS gridded model system
Referring to the architecture illustration below:
* How you collect input data and calculate your results is entirely up to you
* The results must be presented as a WMS following the VIPS conventions. Scroll down for details.
<img src="illustrations/VIPS-grid-architecture.drawio.png" alt="VIPS Logo"/>
## Get started
### Get gridded weather data
Gridded weather data typically comes in [NetCDF](https://www.unidata.ucar.edu/software/netcdf/) or [GRIB](https://en.wikipedia.org/wiki/GRIB) format. In VIPS, we have been using NetCDF files, which can be inspected with command-line tools like [`ncdump`](https://www.unidata.ucar.edu/software/netcdf/workshops/2011/utilities/Ncdump.html) and [`ncview`](https://cirrus.ucsd.edu/ncview/).
<img src="illustrations/grid_models_02.png" alt="VIPS Logo" />
*`ncview` is a great tool to familiarize yourself with a NetCDF file*
### Manipulate data, perform calculations
Performing calculations on gridded data are typically done using tools like [R](https://www.r-project.org/), [numPy](https://numpy.org/)/[netCDF4](https://pypi.org/project/netCDF4/) or [CDO](https://code.mpimet.mpg.de/projects/cdo). The latter is by far the quickest, but - depending on the problem - could be cumbersome to work with from a programmer's point of view.
```bash
$ cdo -selname,TM -seldate,2023-04-01T00:00:00,2023-12-31T00:00:00 met_1_0km_nordic-2023.nc TM_from_april.nc
```
*The above example extracts one variable (`TM` - mean temperature) for the period starting with April 1st 2023 and outputs this to a new NetCDF file. To see all CDO operators, run `cdo --operators`*
### Create output files
The output files must be data files for a WMS. A common format for this purpose is [GeoTIFF](https://en.wikipedia.org/wiki/GeoTIFF). To convert data from NetCDF to GeoTIFF, we use [GDAL](https://gdal.org/), for example:
```bash
$ gdal_translate -ot Int16 -of GTiff -b 1 NETCDF:"tmp/result.nc":WARNING_STATUS out/result_WARNING_STATUS_2023-04-01.tif
```
*This selects the first timestep (`-b 1`) of result.nc, specifically the WARNING_STATUS variable, and creates a GeoTIFF file from this.*
### Set up a WMS (Web Map Service)
[Introduction to WMS (Wikipedia)](https://en.wikipedia.org/wiki/Web_Map_Service)
A WMS can be created using server platforms such as:
* [MapServer](https://mapserver.org/)
* [GeoServer](https://geoserver.org/)
* [ArcGIS server](https://enterprise.arcgis.com/en/server/) (proprietary)
The code/configuration examples below are for MapServer.
## The structure of the WMS
VIPS expects the WMS to follow certain patterns in order to display it in a map.
### Model metadata
You need at least these metadata:
* `wms_title` - Used to display the model's name (see illustration above)
* `wms_abstract` - This is used to display the model's description (see illustration above). HTML format
#### Optional: abstract sections
For clients that wish to display the abstract in more than just one block, it is possible to use the `<div>` tag with ids `preamble` and `body`, like e.g. this:
```html
<div id='preamble'>
<p>
The reference humidity model was developed as a supplement to
the Humidity model. In this model 20 consecutive hours are
required to fulfil a risk period. The Humidity model is considered
as a DSS with several components. The reference humidity model is
considered as a weather based submodel for the risk of Septoria,
easy to map and calculate based on weather data alone.
</p>
</div>
<div id='body'>
<p>
One constraint in this method is that you can have 19 consecutive
risk hours or, fx 14 hours with risk then one hour below the Rh threshold
and then maybe 14 hours again with risk hours. In one of
these situations, the model will indicate a risk. In the
reference model the definition of Humid hours was introduced.
The Rh threshold was avoided as humid hours do not need to be
consecutive. The running sum of humid hours across three days
indicate that the Septoria risk is higher than if you have
three days with humid conditions than two or one. The operation
of the model should include weather forecast data and should
run 6 days ahead from current day if you include a 7-day weather
forecast (60 hours from national met office and until 7 days from ECMWF)
</p>
</div>
```
### WMS Layers
In order for VIPS to be able to utilize your WMS, you need to structure the layers in a specific way:
* The layer names must follow this namespacing pattern: `[MODEL_ID].[PARAMETER ID].[YYYY-MM-DD]`. For instance, the PSILARTEMP (Carrot Rust Fly) model has these layers (for one day):
* `PSILARTEMP.WARNING_STATUS.2023-04-01`
* `PSILARTEMP.DD.2023-04-01`
**There is one mandatory layer, and that is of course the `WARNING_STATUS` layer. It must have an integer data type - at least if you're using [Mapserver](https://mapserver.org/) for the WMS**
For each layer, you need to specify metadata for classification and legend building, e.g. like in this mapfile excerpt:
```mapfile
LAYER
NAME "PSILARTEMP.WARNING_STATUS.2023-04-01"
DATA "/disks/data01/mapserver/data/PSILARTEMP/result_WARNING_STATUS_2023-04-01.tif"
TEMPLATE "/disks/data01/mapserver/wms/PSILARTEMP/query_template.xml" TOLERANCE 1 TOLERANCEUNITS PIXELS
TYPE RASTER
PROCESSING "BANDS=1" # WARNING_STATUS band on top (others invisible, but band values are available in the query template)
PROCESSING "NODATA=-1"
STATUS ON
METADATA
"wms_title" "Carrot rust fly (Psila rosae) temperature model 2023-04-01"
END
CLASSITEM "[pixel]"
# class using simple string comparison, equivalent to ([pixel] = 0)
CLASS
NAME "Model not running"
EXPRESSION ([pixel] >= 0 AND [pixel] < 2)
STYLE
COLOR 112 112 112
END
END
CLASS
NAME "No infection risk"
EXPRESSION ([pixel] >= 2 AND [pixel] < 3)
STYLE
COLOR 0 180 87
END
END
CLASS
NAME "Possible infection risk"
EXPRESSION ([pixel] >= 3 AND [pixel] < 4)
STYLE
COLOR 255 204 0
END
END
CLASS
NAME "High infection risk"
EXPRESSION ([pixel] >= 4)
STYLE
COLOR 255 0 0
END
END
END # Layer
```
This will result in correct color presentation in the map, and the availability of a legend for the map (see the upper right corner in the screenshot above)
### WMS queries
A feature request to a WMS layer should get an XML response following this structure
```xml
<?xml version="1.0" encoding="UTF-8"?>
<vipsResult>
<modelName value="Carrot rust fly temperature model"/>
<modelId value="PSILARTEMP"/>
<!--[ACTUAL DATA]-->
</vipsResult>
```
The `[ACTUAL_DATA]` are elements with parameter names and values. For instance:
```xml
<!--mapserver template-->
<?xml version="1.0" encoding="UTF-8"?>
<vipsResult>
<modelName value="Carrot rust fly temperature model"/>
<modelId value="PSILARTEMP"/>
<parameter name="DD" value="46.2"/> <!-- Day degrees for the selected location at a given time -->
</vipsResult>
```
If you click on the WARNING_STATUS layer, you get a special response, the `warningStatus` element:
```xml
<?xml version="1.0" encoding="UTF-8"?>
<vipsResult>
<modelName value="Carrot rust fly temperature model"/>
<modelId value="PSILARTEMP"/>
<warningStatus value="3"/> <!-- Moderate infection risk for the selected location at a given time -->
</vipsResult>
```
### Multi language support
Mapserver supports the &language=[language code] query parameter. Read more about the specifics [here](https://mapserver.org/ogc/inspire.html). Through this, we have been able to add language specific titles and abstracts. See an example below
```mapserver
MAP
[...]
WEB
METADATA
[...]
"wms_inspire_capabilities" "embed"
"wms_title.en" "Carrot rust fly (Psila rosae) temperature model"
"wms_title.nb" "Gulrotflue svermetidspunktmodell"
END
END
END
```
#### Hacking i18n of mapserver generated legends
The auto generated legends from these mapserver sections do not support internationalization:
```mapserver
CLASS
NAME "Model not running"
EXPRESSION ([pixel] >= 0 AND [pixel] < 2)
STYLE
COLOR 112 112 112
END
END
CLASS
NAME "No infection risk"
EXPRESSION ([pixel] >= 2 AND [pixel] < 3)
STYLE
COLOR 0 180 87
END
END
CLASS
NAME "Possible infection risk"
EXPRESSION ([pixel] >= 3 AND [pixel] < 4)
STYLE
COLOR 255 204 0
END
END
CLASS
NAME "High infection risk"
EXPRESSION ([pixel] >= 4)
STYLE
COLOR 255 0 0
END
END
```
We have hacked this by using the `"wms_abstract"` metadata option for each layer. For example:
```mapserver
LAYER
NAME "PSILARTEMP.WARNING_STATUS.2023-04-01"
[...]
METADATA
"wms_title" "Reference humidity model 2023-10-28"
"wms_abstract.en" "
{
\"isWarningStatus\": true,
\"legendItems\": [
{
\"classification\": 0,
\"legendLabel\": \"No infection risk\",
\"legendIconCSS\": \"width: 25px; background-color: #707070;\"
},
{
\"classification\": 2,
\"legendLabel\": \"Low infection risk\",
\"legendIconCSS\": \"width: 25px; background-color: #FFCC00;\"
},
{
\"classification\": 3,
\"legendLabel\": \"Medium infection risk\",
\"legendIconCSS\": \"width: 25px; background-color: #FFCC99;\"
},
{
\"classification\": 4,
\"legendLabel\": \"High infection risk\",
\"legendIconCSS\": \"width: 25px; background-color: #FF0000;\"
}
]
}
"
"wms_abstract.nb" "
{
\"isWarningStatus\": true,
\"legendItems\": [
{
\"classification\": 0,
\"legendLabel\": \"Ingen infeksjonsrisiko\",
\"legendIconCSS\": \"width: 25px; background-color: #707070;\"
},
{
\"classification\": 2,
\"legendLabel\": \"Lav infeksjonsrisiko\",
\"legendIconCSS\": \"width: 25px; background-color: #FFCC00;\"
},
{
\"classification\": 3,
\"legendLabel\": \"Middels infeksjonsrisiko\",
\"legendIconCSS\": \"width: 25px; background-color: #FFCC99;\"
},
{
\"classification\": 4,
\"legendLabel\": \"Høy infeksjonsrisiko\",
\"legendIconCSS\": \"width: 25px; background-color: #FF0000;\"
}
]
}
"
END
END
```
The Json schema for the legend is as follows: (TODO: place on a URI)
```json
{
"title": "Layer legend",
"type": "object",
"required": [
"legendItems"
],
"properties": {
"isWarningStatus": {
"type": "boolean",
"title": "Legend type is warning status?",
"default": false
},
"legendItems": {
"type": "array",
"title": "Legend items",
"description": "The lengend items in their preferred order",
"items": {
"required": [
"legendLabel"
],
"type": "object",
"title": "Legend item",
"properties": {
"classification": {
"type": "number",
"title": "Classification"
},
"legendLabel": {
"type": "string",
"title": "Legend label"
},
"legendIconCSS": {
"type": "string",
"title": "Legend icon CSS"
}
}
}
}
}
}
```
Use this schema e.g. in [Json-editor](https://json-editor.github.io/json-editor/) and create the legend without coding. Remember to escape all double quotes before adding it to a mapfile.
The client must then check if the layer abstract exists in the requested language. If not, default to the auto generated legend. For example like this (using OpenLayers):
```javascript
// See if there is a language specific legend available
if(currentLayer.Abstract != undefined)
{
// You need to generate and return HTML from the Json in the getLegendHTML function
document.getElementById("layerLegend").innerHTML=getLegendHTML(currentLayer.Abstract);
}
else // Fallback to auto generated legend
{
document.getElementById("layerLegend").innerHTML='<img id="layerLegendImg" src="' + currentLayer.Style[0].LegendURL[0].OnlineResource + '"/>';
}
```
### Example Mapserver file
For your reference, please see the file `examples/PSILARTEMP.map` in this repository.
## Existing VIPS grid models for reference:
* The Carrot Rust Fly Temperature Model
* [Source code](https://gitlab.nibio.no/VIPS/models/grid/grid-psilartemp)
* [In action](https://testvips.nibio.no/spatial/gridmap/PSILARTEMP/)
* Septoria Reference Humidity Model
* [Source code](https://gitlab.nibio.no/VIPS/models/grid/SEPTREFHUM)
* [In action](https://testvips.nibio.no/spatial/gridmap/SEPTREFHUM/)
## VIPS grid model client
### Reference implementation
The reference implementation for a VIPS grid model client is found in the [VIPSWeb repository](https://gitlab.nibio.no/VIPS/VIPSWeb/-/tree/develop/spatial). These two files are the most important:
* [gridmap.html](https://gitlab.nibio.no/VIPS/VIPSWeb/-/blob/develop/spatial/templates/spatial/gridmap.html)
* [gridmap.js](https://gitlab.nibio.no/VIPS/VIPSWeb/-/blob/develop/spatial/static/spatial/js/gridmap.js)
The map API is an [old version of OpenLayers](https://openlayers.org/en/v4.6.5/apidoc/index.html). We strongly encourage to use the latest stable version!