grid_models.md

VIPS Grid models

Tor-Einar Skog, Senior developer, NIBIO

Updated: 2023-12-20

What you will learn

This document describes how to set up a VIPS compatible gridded DSS model, such as in the screenshot below. It also describes briefly how to set up a client to a VIPS compatible DSS gridded model.

The gridded models are using the same "traffic light" color pattern for indicating risk as the location based models. In addition, any other numeric parameter can be displayed as a separate layer, with its own legend and colour scheme.

The screenshot shows how results from the Carrot Rust Fly temperature model are presented in VIPS. You select date using the slider in the lower left corner, and you select parameter using the radio buttons in the lower right corner. By clicking in the map, a popup shows the warning status and values of any other parameters.

Prerequisites

• You should be familiar with web maps and Web Map Services (WMS)
• You should be familiar with using and manipulating gridded weatherdata files, e.g. NetCDF files or GRIB files.

Architecture of the VIPS gridded model system

Referring to the architecture illustration below:

• How you collect input data and calculate your results is entirely up to you
• The results must be presented as a WMS following the VIPS conventions. Scroll down for details.

Get started

Get gridded weather data

Gridded weather data typically comes in NetCDF or GRIB format. In VIPS, we have been using NetCDF files, which can be inspected with command-line tools like ncdump and ncview.

ncview is a great tool to familiarize yourself with a NetCDF file

Manipulate data, perform calculations

Performing calculations on gridded data are typically done using tools like R, numPy/netCDF4 or CDO. The latter is by far the quickest, but - depending on the problem - could be cumbersome to work with from a programmer's point of view.

$ cdo -selname,TM -seldate,2023-04-01T00:00:00,2023-12-31T00:00:00 met_1_0km_nordic-2023.nc TM_from_april.nc

The above example extracts one variable (TM - mean temperature) for the period starting with April 1st 2023 and outputs this to a new NetCDF file. To see all CDO operators, run cdo --operators

Create output files

The output files must be data files for a WMS. A common format for this purpose is GeoTIFF. To convert data from NetCDF to GeoTIFF, we use GDAL, for example:

$ gdal_translate -ot Int16 -of GTiff -b 1 NETCDF:"tmp/result.nc":WARNING_STATUS out/result_WARNING_STATUS_2023-04-01.tif

This selects the first timestep (-b 1) of result.nc, specifically the WARNING_STATUS variable, and creates a GeoTIFF file from this.

Set up a WMS (Web Map Service)

Introduction to WMS (Wikipedia)

A WMS can be created using server platforms such as:

• MapServer
• GeoServer
• ArcGIS server (proprietary)

The code/configuration examples below are for MapServer.

The structure of the WMS

VIPS expects the WMS to follow certain patterns in order to display it in a map.

Model metadata

You need at least these metadata:

• wms_title - Used to display the model's name (see illustration above)
• wms_abstract - This is used to display the model's description (see illustration above). HTML format

Optional: abstract sections

For clients that wish to display the abstract in more than just one block, it is possible to use the <div> tag with ids preamble and body, like e.g. this:

<div id='preamble'>
  <p>
    The reference humidity model was developed as a supplement to
    the Humidity model. In this model 20 consecutive hours are
    required to fulfil a risk period. The Humidity model is considered 
    as a DSS with several components. The reference humidity model is 
    considered as a weather based submodel for the risk of Septoria, 
    easy to map and calculate based on weather data alone.
  </p>
</div>
<div id='body'>
  <p>
    One constraint in this method is that you can have 19 consecutive 
    risk hours  or, fx 14 hours with risk then one hour below the Rh threshold
    and then maybe 14 hours again with risk hours. In one of
    these situations, the model will indicate a risk. In the
    reference model the definition of Humid hours was introduced.
    The Rh threshold was avoided as humid hours do not need to be
    consecutive. The running sum of humid hours across three days
    indicate that the Septoria risk is higher than if you have
    three days with humid conditions than two or one. The operation
    of the model should include weather forecast data and should
    run 6 days ahead from current day if you include a 7-day weather
    forecast (60 hours from national met office and until 7 days from ECMWF)
  </p>
</div>

WMS Layers

In order for VIPS to be able to utilize your WMS, you need to structure the layers in a specific way:

• The layer names must follow this namespacing pattern: [MODEL_ID].[PARAMETER ID].[YYYY-MM-DD]. For instance, the PSILARTEMP (Carrot Rust Fly) model has these layers (for one day):
  • PSILARTEMP.WARNING_STATUS.2023-04-01
  • PSILARTEMP.DD.2023-04-01

There is one mandatory layer, and that is of course the WARNING_STATUS layer. It must have an integer data type - at least if you're using Mapserver for the WMS

For each layer, you need to specify metadata for classification and legend building, e.g. like in this mapfile excerpt:

LAYER
        NAME "PSILARTEMP.WARNING_STATUS.2023-04-01"
        DATA "/disks/data01/mapserver/data/PSILARTEMP/result_WARNING_STATUS_2023-04-01.tif"
        TEMPLATE "/disks/data01/mapserver/wms/PSILARTEMP/query_template.xml" TOLERANCE 1 TOLERANCEUNITS PIXELS
        TYPE RASTER
        PROCESSING "BANDS=1" # WARNING_STATUS band on top (others invisible, but band values are available in the query template)
        PROCESSING "NODATA=-1"
       
        
        STATUS ON
        METADATA
          "wms_title"     "Carrot rust fly (Psila rosae) temperature model 2023-04-01"  
        END  
          CLASSITEM "[pixel]"
      
        # class using simple string comparison, equivalent to ([pixel] = 0)
      
        CLASS
          NAME "Model not running"
          EXPRESSION ([pixel] >= 0 AND [pixel] < 2) 
          STYLE
              COLOR 112 112 112
          END
        END
        CLASS
          NAME "No infection risk"
          EXPRESSION ([pixel] >= 2 AND [pixel] < 3) 
          STYLE
              COLOR 0 180 87
          END
        END
        CLASS
          NAME "Possible infection risk"
          EXPRESSION ([pixel] >= 3 AND [pixel] < 4) 
          STYLE
              COLOR 255 204 0
          END
        END
        CLASS
          NAME "High infection risk"
          EXPRESSION ([pixel] >= 4) 
          STYLE
              COLOR  255 0 0
          END
        END
    END # Layer

This will result in correct color presentation in the map, and the availability of a legend for the map (see the upper right corner in the screenshot above)

WMS queries

A feature request to a WMS layer should get an XML response following this structure

<?xml version="1.0" encoding="UTF-8"?>
<vipsResult>
    <modelName value="Carrot rust fly temperature model"/>
    <modelId value="PSILARTEMP"/>
    <!--[ACTUAL DATA]-->
</vipsResult>

The [ACTUAL_DATA] are elements with parameter names and values. For instance:

<!--mapserver template-->
<?xml version="1.0" encoding="UTF-8"?>
<vipsResult>
    <modelName value="Carrot rust fly temperature model"/>
    <modelId value="PSILARTEMP"/>
    <parameter name="DD" value="46.2"/> <!-- Day degrees for the selected location at a given time -->
</vipsResult>

If you click on the WARNING_STATUS layer, you get a special response, the warningStatus element:

<?xml version="1.0" encoding="UTF-8"?>
<vipsResult>
    <modelName value="Carrot rust fly temperature model"/>
    <modelId value="PSILARTEMP"/>
    <warningStatus value="3"/> <!-- Moderate infection risk for the selected location at a given time -->
</vipsResult>

Multi language support

Mapserver supports the &language=[language code] query parameter. Read more about the specifics here. Through this, we have been able to add language specific titles and abstracts. See an example below

MAP
[...]
  WEB
    METADATA
    [...]
    "wms_inspire_capabilities" "embed"
    "wms_title.en"     "Carrot rust fly (Psila rosae) temperature model"
    "wms_title.nb"     "Gulrotflue svermetidspunktmodell"
    END
  END
END

Hacking i18n of mapserver generated legends

The auto generated legends from these mapserver sections do not support internationalization:

CLASS
      NAME "Model not running"
      EXPRESSION ([pixel] >= 0 AND [pixel] < 2) 
      STYLE
          COLOR 112 112 112
      END
    END
    CLASS
      NAME "No infection risk"
      EXPRESSION ([pixel] >= 2 AND [pixel] < 3) 
      STYLE
          COLOR 0 180 87
      END
    END
    CLASS
      NAME "Possible infection risk"
      EXPRESSION ([pixel] >= 3 AND [pixel] < 4) 
      STYLE
          COLOR 255 204 0
      END
    END
    CLASS
      NAME "High infection risk"
      EXPRESSION ([pixel] >= 4) 
      STYLE
          COLOR  255 0 0
      END
    END

We have hacked this by using the "wms_abstract" metadata option for each layer. For example:

  LAYER
    NAME "PSILARTEMP.WARNING_STATUS.2023-04-01"
    [...]
    METADATA
      "wms_title"     "Reference humidity model 2023-10-28"  
      "wms_abstract.en" "
      {
          \"isWarningStatus\": true,
          \"legendItems\": [
          {
            \"classification\": 0,
            \"legendLabel\": \"No infection risk\",
            \"legendIconCSS\": \"width: 25px; background-color: #707070;\"
          },
          {
            \"classification\": 2,
            \"legendLabel\": \"Low infection risk\",
            \"legendIconCSS\": \"width: 25px; background-color: #FFCC00;\"
          },
          {
            \"classification\": 3,
            \"legendLabel\": \"Medium infection risk\",
            \"legendIconCSS\": \"width: 25px; background-color: #FFCC99;\"
          },
          {
            \"classification\": 4,
            \"legendLabel\": \"High infection risk\",
            \"legendIconCSS\": \"width: 25px; background-color: #FF0000;\"
          }
        ]
      }
      "
      
      "wms_abstract.nb" "
      {
          \"isWarningStatus\": true,
          \"legendItems\": [
          {
            \"classification\": 0,
            \"legendLabel\": \"Ingen infeksjonsrisiko\",
            \"legendIconCSS\": \"width: 25px; background-color: #707070;\"
          },
          {
            \"classification\": 2,
            \"legendLabel\": \"Lav infeksjonsrisiko\",
            \"legendIconCSS\": \"width: 25px; background-color: #FFCC00;\"
          },
          {
            \"classification\": 3,
            \"legendLabel\": \"Middels infeksjonsrisiko\",
            \"legendIconCSS\": \"width: 25px; background-color: #FFCC99;\"
          },
          {
            \"classification\": 4,
            \"legendLabel\": \"Høy infeksjonsrisiko\",
            \"legendIconCSS\": \"width: 25px; background-color: #FF0000;\"
          }
        ]
      }
      "
      
    END  
  END

The Json schema for the legend is as follows: (TODO: place on a URI)

{
  "title": "Layer legend",
  "type": "object",
  "required": [
    "legendItems"
  ],
  "properties": {
    "isWarningStatus": {
      "type": "boolean",
      "title": "Legend type is warning status?",
      "default": false
    },
    "legendItems": {
      "type": "array",
      "title": "Legend items",
      "description": "The lengend items in their preferred order",
      "items": {
        "required": [
          "legendLabel"
        ],
        "type": "object",
        "title": "Legend item",
        "properties": {
          "classification": {
            "type": "number",
            "title": "Classification"
          },
          "legendLabel": {
            "type": "string",
            "title": "Legend label"
          },
          "legendIconCSS": {
            "type": "string",
            "title": "Legend icon CSS"
          }
        }
      }
    }
  }
}

Use this schema e.g. in Json-editor and create the legend without coding. Remember to escape all double quotes before adding it to a mapfile.

The client must then check if the layer abstract exists in the requested language. If not, default to the auto generated legend. For example like this (using OpenLayers):

// See if there is a language specific legend available
if(currentLayer.Abstract != undefined)
{
  // You need to generate and return HTML from the Json in the getLegendHTML function
    document.getElementById("layerLegend").innerHTML=getLegendHTML(currentLayer.Abstract);
}
else // Fallback to auto generated legend
{
    document.getElementById("layerLegend").innerHTML='<img id="layerLegendImg" src="' + currentLayer.Style[0].LegendURL[0].OnlineResource + '"/>';
}

Example Mapserver file

For your reference, please see the file examples/PSILARTEMP.map in this repository.

Existing VIPS grid models for reference:

• The Carrot Rust Fly Temperature Model
  • Source code
  • In action
• Septoria Reference Humidity Model
  • Source code
  • In action

VIPS grid model client

Reference implementation

The reference implementation for a VIPS grid model client is found in the VIPSWeb repository. These two files are the most important:

• gridmap.html
• gridmap.js

The map API is an old version of OpenLayers. We strongly encourage to use the latest stable version!