Py-QGIS-WPS

New in 1.8: OGC api processes support

Py-QGIS-WPS is an implementation of the Web Processing Service standard from the Open Geospatial Consortium based on the QGIS Processing API.

Since 1.8 Py-QGIS-WPS supports OGC API REST processes api

This implementation allows you to expose and run on a server:

• QGIS Processing algorithms available on Desktop (side note, exposing a QGIS model or Processing script are recommended)
• QGIS Processing models and scripts
• QGIS plugins having a Processing provider according to their metadata.txtfile

It is written in Python and was originally a fork of PyWPS.

Requirements and limitations :

• Python 3.10 minimum
• Windows is not officially supported
• Redis server

Documentation

Latest documentation is available on docs.3liz.org.

Why Py-QGIS-WPS ?

Py-QGIS-WPS differs from PyWPS in the following:

• QGIS centric
• Handle all requests in an asynchronous way: all jobs run in a non-blocking way, even when synchronous operation is requested.
• Use multiprocessing Pool to handle task queue instead of instantiating a new process each time.
• Uniform Logging with the 'logging' module
• Implements OGC processes api.
• Use Redis for asynchronous status storage.
• Support streamed/chunked requests for stored data
• Add extensions to WPS: TIMEOUT and EXPIRE
• No Windows support

All these changes were not easy to implement without some drastic changes of the original code, and we think that it deviates too much from the PyWPS original intentions.

That is, we have decided to fork the original project and go along with it.

So, we are really grateful to the original authors of PyWPS for the nice piece of software that helped us very much to start quickly this project.

Why moving to Tornado instead WSGI

• We need to support asyncio: asyncio requires a blocking running loop. This cannot be achieved simply in a WSGI architecture.
• Tornado is fully integrated with native python asyncio library and provide a great framework for developing an HTTP server.

Extensions to WPS

TIMEOUT extension

Specify the timeout for a process: if the process takes more than TIMEOUT seconds to run, the worker is then killed, and an error status is returned.

Set the TIMEOUT=<seconds> in GET requests.

In POST requests, set the timeout=<seconds> attribut in the <ResponseDocument> tag.

The server may configure the maximum timeout value.

EXPIRE extension

Specify the expiration time for stored results: after EXPIRE seconds after the end of the wps process, all results will be flushed from disks and local cache. Trying to request the results again will return a 404 HTTP error.

Set the EXPIRE=<seconds> in GET requests.

In POST requests, set the expire=<seconds> attribut in the <ResponseDocument> tag.

The server may configure maximum expiration value.

Status API

Now implemented with the processes api: The status REST api will return the list of the stored status for all running and terminated wps processes.

Example for returning all stored statuses:

http://localhost:8080/jobs

Example for returning status for one given job from its id:

http://localhost:8080/jobs/<job_id>

Extensions to processes api:

Files

http://localhost:8080/jobs/<job_id>/files

Running QGIS processing

WPS input/output layer mapping

With QGIS desktop, QGIS processing algorithms usually apply on a QGIS source project and computed layers are displayed in the same context as the source project.

Py-qgis-wps works the same way: a qgis project will be used as a source of input layers. The difference is that, when an algorithm runs, it creates a QGIS project file associated with the current task and register computed layers to it.

The created project may be used as an OWS source with QGIS Server. Output layers are returned as complex objects holding a reference to a WMS/WFS uri that can be used directly with QGIS server. The URI template is configurable using the server/wms_response_uri configuration setting.

Contextualized input parameters

Tasks parameters are contextualized using the MAP query param. If a MAP parameters is given when doing a DescripProcess requests, allowed values for input layers will be taken from the qgis source project according the type of the input layers.

QGIS project (.qgs) files and project stored in Postgres databases are both supported.

The best practice is to always provide a MAP parameters and include the possible input layer in a qgs project. This way you may connect whatever data source supported by qgis and use them as input data in a safe way.

If you need to pass data to your algorithm from client-side, prefer inputs file parameter and small payloads.

Dependencies

See requirements.txt file.

Installation from python package

ADVICE: You should always install in a python virtualenv. If you want to use system packages, set up your environment with the --system-site-packages option.

See the official documentation for how to set up a python virtualenv: https://virtualenv.pypa.io/en/stable/.

From source

Install in development mode

make build
pip install -e .

From python package server

pip install py-qgis-wps

Running the server

The server from a command line interface:

The server does not run as a daemon by itself, there are several ways to run a command as a daemon.

For example:

• Use Supervisor http://supervisord.org/ will give you full control over logs and server status notifications.
• Use the daemon command.
• Use Docker

Running the server

Usage

usage: wpsserver [-h] [-d] [-c [PATH]]
                 [--version] [-p PORT] [-b IP] [-u SETUID]

WPS server

optional arguments:
  -h, --help            show this help message and exit
  -d, --debug           Set debug mode  
  -c [PATH], --config [PATH]
                        Configuration file
  --version             Return version number and exit
  -p PORT, --port PORT  http port
  -b IP, --bind IP      Interface to bind to
  -u SETUID, --setuid SETUID
                        uid to switch to

Requests to OWS services

The OWS requests use the following format: /ows/?<ows_query_params>

Example:

http://myserver:8080/ows/?SERVICE=WPS&VERSION=1.0.0&REQUEST=GetCapabilities

Configuration

From config ini file

By default, the wps server is not using any config file, but one can be used with the --config option. A config file is a simple ini file, a sample config file is given with the sources.

From environment variables

The server can be configured with environnement variables:

Configuration is done with environment variables:

• QGSWPS_SERVER_WORKDIR: set the current dir processes, all processes will be running in that directory.
• QGSWPS_SERVER_HOST_PROXY: When the service is behind a reverse proxy, set this to the proxy entrypoint.
• QGSWPS_SERVER_PARALLELPROCESSES: Number of parallel process workers
• QGSWPS_SERVER_RESPONSE_TIMEOUT: The max response time before killing a process.
• QGSWPS_SERVER_RESPONSE_EXPIRATION: The max time (in seconds) the response from a WPS process will be available.
• QGSWPS_SERVER_WMS_SERVICE_URL: The base url for WMS service. Default to /wms. Responses from processing will be retourned as WMS urls. This configuration variable sets the base url for accessing results.
• QGSWPS_SERVER_RESULTS_MAP_URI

Logging

• QGSWPS_LOGLEVEL: the log level, should be INFO in production mode, DEBUG for debug output.

REDIS storage configuration

• QGSWPS_REDIS_HOST: The redis host
• QGSWPS_REDIS_PORT: The redis port. Default to 6379
• QGSWPS_REDIS_DBNUM: The redis database number used. Default to 0

QGIS project Cache configuration

• QGSWPS_CACHE_ROOTDIR: Absolute path to the qgis projects root directory, projects referenced with the MAP parameter will be searched at this location

Processing configuration

• QGSWPS_PROCESSSING_PROVIDERS_MODULE_PATH: Path to look for processing algorithms provider to publish, algorithms from providers specified here will be runnable as WPS processes.

Exposing algorithms as WPS services

Processing providers follows the same rules as for any QGIS regular processing plugin: implementing processing providers .

Deprecation notices:

• Since 1.1, the __algorithms__.py method for declaring providers is no longer supported.
• since 1.9.3 the the special factory entrypoint: WPSClassFactory(iface) in the __init__.py file. and the wps flag in the metadata.txt are deprecated in favor of hasProcessingProvider=yes anthe call of initProcessing().

The metadata.txt file

As regular QGIS plugin, a metadata.txt file must be present with a special entry hasProcessingProvider=yes indicating that the plugin is available as processing service provider.

Registering providers

At plugin initialization, Py-Qgis-WPS will call the initProcessingMethod() of the object returned by the classFactory function.

[!NOTE] The initProcessing method will be the one and only one method called by

[!IMPORTANT] The iface: QgsInterface parameter is used for initializing Gui component of the plugin in Qgis desktop. This parameter will be set to None when loaded from py-qgis-wps. Implementors should take care to check the value of the iface parameter and drop all gui initialization if not set. The only thing to do is to register the providers the same way as for using in Qgis Desktop.

Example:

from qgis.core import QgsApplication

from .provider import TestAlgorithmProvider


class Test:
    def __init__(self):
        pass

    def initProcessing(self):
        reg = QgsApplication.processingRegistry()

        # XXX we *MUST* keep instance of provider
        self._provider = TestAlgorithmProvider()
        reg.addProvider(self._provider)


def classFactory(iface: QgsInterface|None) -> Test:
    if iface is not None:
        # Initialize GUI
        ... 

    return Test()

Controlling what is exposed:

Processing algorithm with the flag FlagHideFromToolbox set will not be exposed as WPS process.

Parameters with the flag FlagHidden set won't be exposed in a DescribeProcess request

References

• OGC standards
• OGC Api processes
• Introduction to WPS
• Py-qgis-server at FOSS4G 2019.