6.9.2

# ql2sc

QuakeLink (gempa GmbH) to SeisComP event parameter exchange.

## Description

ql2sc manages the import of SeisComP objects from one or several QuakeLink servers into a SeisComP system in real time. Like scimex but contrary to scimport the exchange of the SeisComP objects is event based. This means no messages will be exchanged until the exporting system has produced an event.

The user may control at various levels which information to import. Whenever possible server-side filters should be preferred to reduce both the network bandwidth consumption as well as the CPU and memory utilization on the local machine.

Note

ql2sc does not delete events at the import system although quakelink allows the deletion of events. Deleted events are ignored by ql2sc and kept in the SeisComP database.

## Server-Side Event Filter

QuakeLink provides a filter syntax similar to SQL-WHERE clauses which may be used to filter interesting events on the server side:

    clause    := condition[ AND|OR [(]clause[)]]
    condition := MAG|DEPTH|LAT|LON|PHASES|DIST(lat,lon) op {float} |
                 DIST(lat,lon) IN [{float}, {float}] |
                 UPDATED|OTIME op time |
                 AGENCY|AUTHOR|STATUS|ESTATUS|EMODE|TYPE|REGION|MAG_T op 'string' |
                 MAG|DEPTH|LAT|LON|PHASES|OTIME|UPDATED IS [NOT] NULL
    op        := =|!=|>|>=|<|<=|eq|gt|ge|lt|ge
    time      := %Y,%m,%d[,%H,%M,%S,%f]

E.g., the following filter string would select only those events with a minimum magnitude of 6, detected by at least 10 stations and which are shallower than 100km:

    MAG >= 6.0 AND PHASES >= 10 AND DEPTH < 100

Note

The supported filter commands depend on the specific QuakeLink version. To list all available options you may connect to the server, e.g., using telnet localhost 18010, and request the help page of the SELECT command using help select.

## Server-Side Object Filter

QuakeLink provides a coarse object filter for the most relevant SeisComP objects:

Option | Impact
--- | ---
picks | include picks
amplitudes | include amplitudes
arrivals | include origin arrivals
staMags | include origin station magnitudes
staMts | include moment tensor station contributions and phase settings
preferred | include only preferred origin and magnitude information

## Local Object Filter and Routing

Subsequent to the server-side filters a routing table defines which objects to import and to which message group to send them. Depending on the SeisComP modules listening to the specified message groups an object may be further processed. Typically no modules (other than scmaster) is connected to the `IMPORT_GROUP` so that objects sent to this group are just stored to the database. If an object should be discarded, the special group identifier `NULL` may be used.

The routing table is defined as a comma-separated list of `object name:group name` pairs. Also the routing rules are inherited recursively within the SeisComP object tree. If no explicit rule exists for an object, the routing of its parent is evaluated up to the `EventParameters` root node.

### Examples

    EventParameters:IMPORT_GROUP

Imports everything

    EventParameters:IMPORT_GROUP,Comment:NULL

Imports everything except comments

    Origin:LOCATION,StationMagnitude:MAGNITUDE,Magnitude:MAGNITUDE

Sends origins and it’s children arrival, origin uncertainty to the `LOCATION` group but the magnitude children to the `MAGNITUDE` group. Skips picks, amplitudes, focal mechanisms and events.

### Default routing table

The default use case of ql2sc is to import earthquake solutions from other data centers or in-house redundant SeisComP systems. The intention is not to reprocess the solution but to add them to the local catalog.

By default we route:

- Picks and Amplitudes to the `IMPORT_GROUP` group to prevent processing by the local locator and amplitude processor

- Origins (including its StationMagnitude and Magnitude children) to the `LOCATION` to allow event association.

- FocalMechanisms to the `FOCMECH` group to trigger processing by specialized applications, e.g., graphical user interfaces for strong motion analysis or tsunami risk assessment.

We don’t route events at all. With the help of scevent locations are either associated to existing events or will create new events with local settings.

We don’t route StationMagnitudes and Magnitude to the `MAGNITDUE` group because scmag subscribes to `LOCATION` and `MAGNITUDE`. Separated groups might lead to duplicated magnitude types in case a manual magnitude solution is imported. In this case the foreign Origin with its Magnitudes would be split into at least two messages, the first one containing the Origin, the second one the Magnitude. The Origin message immediately triggers magnitude calculation, potentially for a magnitude type which is received with the second message.

The default routing table is set as given in `host.$name.routingTable`.

## Agency List Filter

In addition to the local object filter the user may choose to accept only those objects originating from a set of trusted agencies. If at least one agency is defined in the `processing.whitelist.agencies` or `processing.blacklist.agencies` configuration option, then the `creationInfo.agencyID` of amplitudes, arrivals, comments, events, focal mechanisms, magnitudes, moment tensors, origins, picks and station magnitudes is evaluated. Objects with unmatched or unset agency information are filtered out. If objects with unset agency ID should match, then empty string `""` has to be added to the white list.

The agency filter is applied on remote as well as local objects. In this way remote objects may be excluded from import and local objects my be protected from overriding or removing. Also the filter is applied recursively. If parent object (e.g., an origin) is filtered out, all of its children (e.g., magnitudes) are also skipped even if they carry a different agency ID.

Note

The agency white list filter might be essential to avoid circular event updates between cross-connected SeisComP systems.

## PublicID Prefix Filter

In addition to the agency filter incoming or local objects can be skipped by checking their publicID prefix. It behaves similar to the agency filter but checks the `publicID` attribute rather than the `creationInfo.agencyID` attribute. Prefixes can be configure as white- or blacklist with `processing.whitelist.publicIDs = ...` and `processing.blacklist.publicIDs = ...`.

## Workflow

Each event update received from a QuakeLink host is parsed and analyzed for differences to the local database. The comparison starts at the level of the top-level elements in the following order: picks, amplitudes, origins, focal mechanisms, events.

For each top-level element the object tree is traversed in a depth-first search order. Objects on the same level are processed in the order of their appearance. The differences are collected as a list of notifier objects with the following operation types:

Type | Description
--- | ---
ADD | The object does not exist locally
UPDATE | The object does exist locally but differs from the remote one
REMOVE | The object exist locally but not remotely

The `ADD` and `REMOVE` operation always generates notifies of the same type for all children of the current object. `ADD` notifiers are collected top-down, `REMOVE` notifiers are collected bottom-up.

Because the order of child objects is arbitrary, e.g., the arrivals of an origin, each object on the remote side has to be found in the set of local objects. For public objects (e.g., origins, magnitudes, magnitudes..), the `publicID` property is used for comparison. All other objects are compared by looking at their index properties. For e.g., arrivals this is the `pickID` property, for comments the `id` property.

Ones all notifiers are collected they are send to the local messaging system. For performance reasons and because of the processing logic of listening SeisComP modules ql2sc tries to batch as many notifiers as possible into one notifier message. A separate notifier message is created if the target message group changes between successive notifiers or if the configurable `batchSize` limit is reached.

Note

Care must be taken when configuring the `batchSize` limit. If the value is to big the overall message size limit (default: 1MB) may be exceeded resulting in an undeliverable message. On the other hand a much to small value will create unwanted results in the SeisComP processing chain. If for instance picks are routed to the `PICK` group and the pick set is split into several notifier messages the local scautoloc might create locations based on an incomplete dataset.

## Event Attributes

It might be desirable to synchronize event attributes set at the source with the local system. In particular the event type, the type uncertainty, event descriptions and comments might be of interest. Because it is not advisable to route events and let scevent associate imported origins it can happen that the imported event ID is different from the event ID of the local system. The input host configuration parameter `syncEventAttributes` controls that behaviour. It is set to true by default which means that imported event attributes are going to be imported as well. ql2sc does not update directly the attributes but commandates scevent in as many cases as possible to do so. To find the matching local event it takes the first occurrence which has associated the currently imported preferred origin.

### Limitations

There are limitations to this process to avoid infinite loops when cross connecting two systems. Prior to sending the commands to scevent to change a particular attribute ql2sc checks if that attribute has been set already by another module (via JournalEntry database table). If not, then ql2sc is allowed to request an attribute change otherwise not. To illustrate the issue take the following example:

scolv connected to system `A` changes the event type to ‘earthquake’. ql2sc of system `B` checks if the event type of the local event has been changed already which is not the case and it requests that change. System `A` changes the event type again to ‘unset’. ql2sc of system `B` notices that someone has already changed the event type and it was ql2sc itself. It requests again a change.

scolv connected to system `B` changes the event type to ‘earthquake’ again. ql2sc of system `A` notices that `scolv@A` has already changed the event type and ignores the request.

That simple case would not create an infinite loop even if `ql2sc@A` would accept the last change. The situation changes immediately if two subsequent attribute changes are being received by `ql2sc@B` while both of them are already applied on system `A`. `ql2sc@B` would “restore” the old state due to the first received update and then apply the “final” state due to the second update. Each update triggers again an update at system `A` and the states start flapping. Without the described check there wouldn’t be a well defined exit condition.

## Caveats

Specific combinations of remote and local object filters may result in the loss of data. If for instance origins are imported from system `A` to `B` and additional magnitudes for the received origins are calculated on `B`, care must be taken. Without protection a new event update containing the same origin will `REMOVE` all newly calculated magnitudes on `B` since they are not included in the magnitude set sent by `A`.

To avoid losing these local magnitudes one may decide to block magnitudes from import by routing them to `NULL`. If magnitudes from `A` and from `B` should be available, an agency filter or publicID filter may be defined.

Make sure `A` and `B` use either distinct agency IDs or distinct publicID patterns and add the agency ID of `B` to `processing.blacklist.agencies` or the publicID prefix of `B` to `processing.blacklist.publicIDs`.

## Module Configuration

`etc/defaults/global.cfg`
`etc/defaults/ql2sc.cfg`
`etc/global.cfg`
`etc/ql2sc.cfg`
`~/.seiscomp/global.cfg`
`~/.seiscomp/ql2sc.cfg`

ql2sc inherits global options.

backLog

  Default: `1800`

  Type: int

  Unit: s

  Number of seconds to fetch missed updates on start up.

cacheSize

  Default: `5000`

  Type: int

  Number of public objects to cache.

batchSize

  Default: `2000`

  Type: int

  Maximum number of notifiers to batch in one message. If set to 0 no size limit is enforced. Make sure to not hit the overall message size limited of 16MiB which is enforced by the messaging system.

eventAssociationTimeout

  Default: `10`

  Type: int

  Unit: s

  If event synchronisation is enabled and an incoming origin is not yet associated with an event on the target machine, then this timeout defines the maximum number of seconds to wait for an association.

hosts

  Type: list:string

  Registration of the host profiles defining the connection parameters to the QuakeLink hosts.

Note

host.* Definition of host profiles. For each host profile a connection to one QuakeLink server can established. The profiles must be registered in ‘hosts’ to apply them.

Note

host.$name.* Provide the connection parameters to one QuakeLink server. $name is a placeholder for the name to be used and needs to be added to `hosts` to become active.

    hosts = a,b
    host.a.value1 = ...
    host.b.value1 = ...
    # c is not active because it has not been added
    # to the list of hosts
    host.c.value1 = ...

host.$name.url

  Default: `ql://localhost:18010`

  Type: string

  URL of the QuakeLink service, the scheme ‘qls’ enables SSL.

  Format: [ql[s]://][user:pwd@][host][:port].

  If set to an empty string the application will run without any QuakeLink connection attempt.

host.$name.gzip

  Default: `false`

  Type: boolean

  Enable/disable GZip (GNU zip) compression.

host.$name.native

  Default: `false`

  Type: boolean

  Request native data instead of XML format. Native data export may be disabled on some hosts.

host.$name.syncEventAttributes

  Default: `true`

  Type: boolean

  Try to update the event attributes of the target event with the attributes of the source event which includes event type and event certainty. It will not import events but tries to find the associated event of the input preferred origin at the target system and will update the event attributes via journaling.

host.$name.syncPreferred

  Default: `false`

  Type: boolean

  Synchronize the preferred origin and preferred magnitude selection if different from the imported selection. ql2sc will wait for the event association of an imported origin and check if the preferred origin or preferred magnitude is different from the imported Quakelink event. If so it will send a journal to force selection of the preferred origin and selection of the preferred magnitude type. These are the same operations as within scolv to fix an origin and a particular magnitude type.

host.$name.syncEventDelay

  Default: `0`

  Type: int

  Delays the synchronization of event attributes in seconds if set to a value greater than zero.

host.$name.keepAlive

  Default: `false`

  Type: boolean

  Request server to send keep alive message every 30s to prevent connection reset by firewalls on long idle periods. If activated the client will reset the connection if no alive message is received within 60s.

host.$name.filter

  Type: string

  SQL like WHERE clause to filter the result set.

  clause := condition[ AND|OR [(]clause[)]] __ condition := MAG|DEPTH|LAT|LON|PHASES|OTIME|UPDATED [op float|time]|[IS [NOT] NULL] __ op := =|>|>=|<|<=|eq|gt|ge|lt|ge __ time := %Y,%m,%d[,%H,%M,%S,%f]

host.$name.routingTable

  Default: `Pick:IMPORT_GROUP,Amplitude:IMPORT_GROUP,FocalMechanism:EVENT,Origin:EVENT`

  Type: list:string

  Map datamodel class names to messaging groups. For unmapped objects the mapping of their parent objects is evaluated recursively. Objects may be excluded by mapping them to ‘NULL’.

Note

host.$name.data.* Specify the XML components to fetch. Note: These options are not used if ‘native’ data is requested.

host.$name.data.picks

  Default: `true`

  Type: boolean

  Include picks

host.$name.data.amplitudes

  Default: `true`

  Type: boolean

  Include amplitudes

host.$name.data.arrivals

  Default: `true`

  Type: boolean

  Include origin arrivals

host.$name.data.staMags

  Default: `true`

  Type: boolean

  Include origin station magnitudes

host.$name.data.staMts

  Default: `true`

  Type: boolean

  Include moment tensor station contributions and phase settings

host.$name.data.preferred

  Default: `true`

  Type: boolean

  Include only preferred origin and magnitude information

processing.blacklist.publicIDs

  Type: list:string

  Defines a blacklist of publicID prefixes that are not allowed for processing. Separate items by comma.

processing.whitelist.publicIDs

  Type: list:string

  Defines a whitelist of publicID prefixes that are allowed for processing. Separate items by comma.

## Command-Line Options

ql2sc [options]

### Generic

-h, --help

  Show help message.

-V, --version

  Show version information.

--config-file arg

  Use alternative configuration file. When this option is used the loading of all stages is disabled. Only the given configuration file is parsed and used. To use another name for the configuration create a symbolic link of the application or copy it. Example: scautopick -> scautopick2.

--plugins arg

  Load given plugins.

-D, --daemon

  Run as daemon. This means the application will fork itself and doesn’t need to be started with &.

### Verbosity

--verbosity arg

  Verbosity level [0..4]. 0:quiet, 1:error, 2:warning, 3:info, 4:debug.

-v, --v

  Increase verbosity level (may be repeated, eg. -vv).

-q, --quiet

  Quiet mode: no logging output.

--print-component arg

  For each log entry print the component right after the log level. By default the component output is enabled for file output but disabled for console output.

--component arg

  Limit the logging to a certain component. This option can be given more than once.

-s, --syslog

  Use syslog logging backend. The output usually goes to /var/lib/messages.

-l, --lockfile arg

  Path to lock file.

--console arg

  Send log output to stdout.

--debug

  Execute in debug mode. Equivalent to --verbosity=4 --console=1 .

--trace

  Execute in trace mode. Equivalent to --verbosity=4 --console=1 --print-component=1 --print-context=1 .

--log-file arg

  Use alternative log file.

### Messaging

-u, --user arg

  Overrides configuration parameter `connection.username`.

-H, --host arg

  Overrides configuration parameter `connection.server`.

-t, --timeout arg

  Overrides configuration parameter `connection.timeout`.

-g, --primary-group arg

  Overrides configuration parameter `connection.primaryGroup`.

-S, --subscribe-group arg

  A group to subscribe to. This option can be given more than once.

--content-type arg

  Overrides configuration parameter `connection.contentType`.

--start-stop-msg arg

  Set sending of a start and a stop message.

### Database

--db-driver-list

  List all supported database drivers.

-d, --database arg

  The database connection string, format: service://user:pwd@host/database. "service" is the name of the database driver which can be queried with "--db-driver-list".