Remote Supervision is the transmission, aggregation and evaluation of data from multiple systems on a Remote Supvervision Master Server. For smaller systems, especially Rapid Prototyping Systems, typically the master is also an OGEMA instance.

Overview and Concepts

The remote supvision uses the RemoteRESTConnector to synchronize data between client and server. Furthermore zipped files are transmitted from the client to the server - in the standard case data log files (slotsDB) and backup files of the resource database generated by the ResAdmin-App.

Synchronization based on RemoteRESTConnector

The data model used for synchronization is GatewayTransferInfo. A resource named RemoteSupervisionList of type ResourceList with element type GatewayTransferInfo is created by the bundle remotesupervision-util. The list contains a single element, which has the gatewayID as name. On the server the respective ResourceList is named controlledTestGateways. It contains an entry for all clients connected to the server.

On the client the single element of the ResourceList has a decorator of type RestConnection (which currently is a custom type of the RemoteRESTConnector). This decorator is NOT transmitted to the master. All connections are configured and initiated by the client, so "pull" means getting data from the server, "push" means sending data to the server.

A problem of the concept is that with every synchronization between client and server the entire data has to be resent. So the amount of data and the update frequeny has to be set with care.

Until appox. 11/2018 the synchronization configuration and coordination was implemented in the bundle remotesupervision-util, which is wrapped into an OGEMA application by the app remote-gateway-v2. Usage of these bundles is not recommended anymore. The configuration of the RemoteSuperVisionList and the sub resource of type RestConnection should be done by a custom application or simply by importing ogj or ogx files with preconfigured resources. The resource of type RestConnection is the sub resource with name restConfigGatewayTest in the resource tree example below.

Configuration of File Transfer: Resource Database Backup

The resource RundirBackup is of type SCPDataCollectionAction. It is processed by the app backup-install and it defines the transmission of the backup of the entire resource tree. It triggers the resAdmin configuration named DailyBackupToSend via the ExtensionPoint mechanism via OGEMA Action. The address configured here is also somehow used by remotesupervision-util to find out the server address. The file system path given in both configuration resources (in the example below data/semabox_01/extBackup) contains a backup of the resource data base in form of unzipped .ogx files. This is overwritten daily. The zipping naming to generalBackup<Date_Time>.zip is performed in the app backup-install-v2 that also sends the file via SCP.

Currently in sema the resources are configured for transmission of resource backups every 12 hours.

Note that until approx. 11/2018 the messages generated via the OGEMA / SLF4J logger from the <rundir>/logs directory were also transferred as Zip-file. This was triggered by the resource EventLogBackup, also of type SCPDataCollectionAction and also processed by the app backup-install. The usage of this mechanism is not recommended anymore. Instead event logs are transferred with the log-transfer app that also organizes transfer of data logs.

The following structure gives an overview on the configurations required besides the RemoteSupeversionList:

RundirBackup.png

Configuration of File Transfer: Slotsdb / Event Log Transfer

This transfer is organized via the app log-transfer or backup-install-sp. The app log-transfer requires an OSGi configuration for the Jetty server on the OGEMA server receiving the data, but also installation of the same app log-transfer also on the server (TODO: Add documentation). The app backup-install-sp assumes that the app file-upload-servlet receives its data on the server. For each log to be transferred a resource has to be defined where the last successful transfer is stored. Only relevant files of the log that are changed after this time are transferred, which shall make sure that all data is transferred, but no unnnecessary transfer is performed.

Transfer of Event Logs: a copy of the data/logs directory is created on the server. The single log files may be gz-compressed.

Transfer of SlotsDB: The element of type GatewayTransferInfo contains an element datalogs, which defines which datalogs shall be transferred. Synchronization of this resource should be disabled or only be done occasionally / on changes as it usually grows quite large.

The RemoteSupervisionList could look like this:

RemoteSupervisionList.png

Add data logs to be transferred

It should be sufficient to add an entry to RemoteSupervisionList.dataLogs to let the client transfer the data to the master. There is currently no way to send a request for e.g. all data logs of a certain type by the master to the clients. The entry must have active fields clientLocation (the location of the resource being logged) and the field transferInterval/timeIntervalLength/type (and the resources in these paths also need to be active). For daily transmission of the log data set type to 10. The other elements will be created by the RemoteSupversion apps:

DataLogTransferInfo.png

Alternative slotsDB transfer if ports for client are blocked

In some cases the gateway can set up a reverse proxy tunnel to the server, but cannot connect to other ports like 8447. In this case the gateway can be configured to generate a daily zip file, which has to be collected by the server and the content is integrated into the remote-slotsDB file structure. To configure zip file generation on the gateway the property org.ogema.eval.utilextended.slotsbackupretardmilli=0 has to be set. With this property the daily zip-file will be generated at 00:30 each day for the previous day. If the generation shall take place later, the value can be increased.

Furthmore you have to setup a backup configuration in the Resource Backup and Cleaner app (resadmin-v2) that contains ./data/semabox_01/extBackup/generalBackup as destinationDirectory. A configuration file that can be used to create this via replay-on-clean is provided as B000_resAdminConfig.ogj. You have to install rsync on the server and on the box. If the ports of the box are very limited sudo apt-get install rsync may fail on the box. In this case you should download the respective package directly from the mirror server / path given by apt-get in its error message, transfer it to the gateway and install with sudo dpkg -i rsync_<version-String>.deb . Note that the name of the file on the server may be slightly different but must contain the version and end on .deb .

To enable the server to access the gateway via SSH and tunnel without a passwort prompt, generate a key file and install on the gateway. For this it is recommended to create a directory for the box communication on the server, which in the following is given as ~/<boxDirectory> . The reverse proxy port via which the gateway can be reached from the server is <gw-reverse-proxy-port>. Then execute on the server like this:

cd ~/<boxDirectory>
ssh-keygen -t rsa -C "login <boxId>" -f ~/<boxDirectory>/id_rsa
ssh-copy-id -p <gw-reverse-proxy-port> -i ~/<boxDirectory>/id_rsa pi@localhost

You can test the connection with

ssh -p <gw-reverse-proxy-port> -i ~/<boxDirectory>/id_rsa pi@localhost

You should then execute a script like the following on server daily with a Cronjob configured like this:

0 3 * * * ~/<boxDirectory>/collectRemote.sh

Script:

#!/bin/sh
cd ~/<boxDirectory>
/usr/bin/rsync --remove-source-files -azvh -e "ssh -p <gw-reverse-proxy-port> -i id_rsa" pi@localhost:ogema/data/backupzip/remoteSlots*.zip .
/usr/bin/unzip -o 'remoteSlots*.zip'
mv data/semabox_01/extBackup/generalBackup*.zip data
rm remoteSlots*.zip
cp -R data/* ../ogemaCollect/rest/<boxID>/

Note that this is still experimental. The implementation needs to save the information of the last full day transferred and zip all full days availabel since then into the file. There should also be a possibility to trigger this manually via the console. Triggering via server GUI would also be great, but this requires a communication back to the gateway.

Management of information which log data has been transmitted already (deprecated, not relevant when using log-transfer)

When a file is transmitted for a certain data log time series and a certain day an entry is added into the datesSentCompletely list of the respective DataLogTransferInfo entry. When all data logs are transferred the respective date is inserted into the datasSentCompletelyAllLogs list and the entries in the single data logs are removed in order to avoid the creation of too many resources. TODO: At the moment moving a date to datasSentCompletelyAllLogs is blocked if a single data log cannot be transferrred, e.g. because the resource configured is lost. This can have a significant impact on the overall system performance. A date should be set to be complete if 3 earlier days have been transmitted successfully for at least 50% of the data logs.

Server configuration Essentials

FendoDB configuration

When using FendoDB you have to make sure on the server that the slotsDB file structure is reloaded when new data has been transferred to the server. Usually you have to set the property org.smartrplace.logging.fendo.reloaddays_interval, e.g org.smartrplace.logging.fendo.reloaddays_interval=900000 for a reload every 15 minutes. You can also call the method org.smartrplace.logging.fendodb.CloseableDataRecorder.reloadDays() to trigger the reload.

Server configuration (partially deprecated)

Reading data from files (slotsDB / Database backup)

  • Zentraler Access Point: Service GatewayBackupAnalysis (implemented in server-backup-analysis)
    This app uses apps backup-parser and slotsdb.standalone (these apps do the real parsing)
  • It also provides GUI to view the data accessible
  • GUI größten Teil nicht mehr notwendig: slotsdb-visualisation (hat Zusatzfunktionen, z.B. Anlegen neuer Slotdsb-Datenbanken)

Evaluations

Setting up a Server Rundir on a local PC

  • Copy Rundir from Git to place on harddisk
    Change project name (directory and in .project file)
  • Add/Change ogemaCollect folder with content from semaServer. Usually only copy content partially (gateways / days) to avoid too much data to be processed all the time during development
  • Copy backup of controlledTestGateways (usually *controlledTestGateways.ogx or .ogj) from server into replay-on-clean of your rundir
    Note that this structure in productve operation is created by clients connecting to the server. It is required for correct operation of the apps calculating points etc.
  • Adapt setting in ogema.properties: org.smartrplace.tools.upload.folder, org.smartrplace.analysis.backup.parser.basepath, de.iwes.sema.test.gateways

Client Configuration

The Rundir requires:

         <bundle dir="bin/smartrplace" groupId="org.smartrplace.apps" artifactId="smartrplace-util" version="&smartrplace-version;" startLevel="30" />
        <bundle dir="bin/ogema" groupId="org.ogema.model" artifactId="smartrplace-proposed" version="&smartrplace-version;" startLevel="10"/>
       <bundle dir="bin/apps" groupId="org.smartrplace.tools" artifactId="remotesupervision-util" version="&smartrplace-version;" startLevel="30" />
         <bundle dir="bin/apps" groupId="org.smartrplace.tools" artifactId="remote-gateway-v2" version="&smartrplace-version;" startLevel="30" />

Furthermore you have to add information to ogema.properties, e.g.

org.smartrplace.remotesupervision.gateway.id=semabox_01debug
org.smartrplace.remotesupervision.server=https://sema.iwes.fraunhofer.de
org.smartrplace.remotesupervision.server.port=8443

Create information for RemoteRESTConnector

The properties server / port are only evaluated in the app sema-fieldtest-config. They are used to create the information in:

  • In any resource GatewayTransferInfo.getSubResource(restConfigGatewayTest, RestConnection.class), particularly in RemoteSuperVisionList/<listElement>/restConfigGatewayTest :
    • restConfigGatewayTest.remotePath
  • SCPDataCollectionAction.host / port if name field of the SCPDataCollectionAction is "RundirBackup"
    Note that such a resource is created as top-level resource and the host is set accordingly, but the port is set to 8443 by default (may be changed by the pattern later on)

This information is used to create

Control PollingIntervals

Properties:

  • de.iwes.sema.supervision.pushinterval : Is used if GatewayTransferInfo.getSubResource("restConfigGatewayTest", RestConnection.class).pushInterval is not acitve / positive or greater then (- 10) minutes.
    The negative value means that the push interval timer is not set by the RemoteRESTConnector, but by remote-supervision-util and the push in the RemoteRESTConnector is trigged via triggerPush.
  • Note that pullInterval is not set or touched (probably) but it seems that still deprecated pollingInterval is used instead. This is set to 30 seconds if de.iwes.sema.devmode-property is true, otherwise to 10 minutes if not already set larger in sema-fieldtest-config
    remote-supervision-util creates pullconfigs-entries for the fields "connectionInterval" and "masterToClientData".
    sema-fieldtest-config creates pullconfigs-entries for a field of type SemaPointsInfo. Here more details are set like the depth of 9, but not an individual pullRate.
  • de.iwes.sema.devmode : see above

Set gateway-ID to be used

The id from ogema.properties should only be used if not set in OGEMA_Gateway/id - and OGEMA_Gateway/id  should be created by ResAdminApp based on ogema.properties if not available (see also ). Furthermore such information is stored in RundirBackup, which is a separate Action from the RemoteSupervision and transfers a backup of the resource database to the server once a day.

App and Data Structures

remote-gateway-v2 contains just a single class that extends RemoteSupervisionClientBase from  remotesupervision-util to make it non-abstract and allow to use it as OGEMA App class. It can be replaced by a custom implementation, typically by replacing the generics type GatewayTransferInfo of RemoteSupervisionClientBase to an extended custom type if information shall be transferred that is not modeled in the standard type.

The app creates a resource RemoteSuperVisionList of type ResourceList<GatewayTransferInfo>. In the package org.ogema.model.gateway.remotesupervision  (fhg-proposed) you find a set of resources that are used to transfer information from the client to the master. The resource list should contain only a single element on each client that is synchronized with the same element on the master via the RemoteRESTConnection app. This transfer is also configured automatically by RemoteSupervisionClientBase. The following information is part of the standard GatewayTransferInfo:

  • SingleValueTransferInfo: Currently only FloatResources as source are supported as RemoteRESTConnection does not support a generic SingleValueResource declaration here.
  • ScheduleTransferInfo: Schedules for evaluation are especially important as results of pre-evaluation on the client in resources of type StatisticalAggregation
  • DataLogTransferInfo: For slotsdb data only information which RecordedData is transferred and which RecordedData is available for transfer is part of the RemoteRESTConnectot transfer. The transfer of the slotsdb data itself is done via file transfer of the REST servlet of the master, but RemoteSuperVisionList also organizes this based on the configuration in RemoteSuperVisionList.
  • For single values and schedules also a structure is foreseen to transfer information from master to client (masterToClientData).
  • ResourceList<EventTransferInfo> events() is not implemented yet. Event notifications will be quite important for RemoteSupervision, though. There is also currently not mechanism to automatically transfer the message log files from client to master and provide tools for evaluation. A suitable strategy is to be discussed.

TODO: Example of a Slotsdb configuration on client

When a new RecordedData is registered as recommended in the Data Analysis Tutorial you can hand over a reference to the GatewayTransferInfo and let the util register the transfer of the slotsdb data right away.

Master Configuration

The master needs the following bundles:

        <bundle dir="bin/smartrplace" groupId="org.smartrplace.apps" artifactId="smartrplace-util" version="&smartrplace-version;" startLevel="30" />
        <bundle dir="bin/ogema" groupId="org.ogema.model" artifactId="smartrplace-proposed" version="&smartrplace-version;" startLevel="10"/>
        <bundle dir="bin/apps" groupId="org.ogema.tools" artifactId="connected-gateway-viewer" version="&widgets-version;" startLevel="30" />

        <bundle dir="bin/apps" groupId="org.smartrplace.tools" artifactId="gateway-master-v2" version="&smartrplace-version;" startLevel="30" />
        <bundle dir="bin/apps" groupId="org.smartrplace.tests" artifactId="file-upload-servlet" version="&smartrplace-version;" startLevel="30" />
        <bundle dir="bin/apps" groupId="org.smartrplace.external" artifactId="slotsdb-standalone" version="&smartrplace-version;" startLevel="30" />
        <bundle dir="bin/apps" groupId="org.smartrplace.external" artifactId="slotsdb-visualisation" version="&smartrplace-version;" startLevel="30" />
       <bundle dir="bin/sema" groupId="de.iwes.sema" artifactId="weatherdata-connector" version="ogema-apps-iwes-version;" startLevel="20" />

Usually you also want to enable access and evaluation of the data:

        <bundle dir="bin/smartrplace" groupId="org.smartrplace.analysis" artifactId="server-backup-analysis" version="&smartrplace-version;" startLevel="30" />
        <bundle dir="bin/smartrplace" groupId="org.smartrplace.tools" artifactId="schedule-management" version="&smartrplace-version;" startLevel="30" />
        <bundle dir="bin/commons" groupId="org.apache.commons" artifactId="commons-csv" version="1.4" startLevel="30" />

        <bundle dir="bin/apps" groupId="de.iwes.tools" artifactId="timeseries-eval-base" version="&widgets-version;" startLevel="30" />
        <bundle dir="bin/apps" groupId="de.iwes.tools" artifactId="timeseries-eval-viz" version="&widgets-version;" startLevel="30" />
        <bundle dir="bin/apps" groupId="de.iwes.tools" artifactId="eval-schedule-viewer" version="&widgets-version;" startLevel="30" />
        <bundle dir="bin/apps" groupId="de.iwes.tools" artifactId="timeseries-aggregation" version="&widgets-version;" startLevel="30" />    
        <bundle dir="bin/apps" groupId="de.iwes.tools" artifactId="timeseries-analysis" version="&widgets-version;" startLevel="30" />
        <bundle dir="bin/apps" groupId="de.iwes.tools" artifactId="server-timeseries-source" version="&smartrplace-version;" startLevel="30" />    
        <bundle dir="bin/apps" groupId="de.iwes.tools" artifactId="server-resource-source" version="&smartrplace-version;" startLevel="30" />


ogema.properties like this:

org.ogema.tools.gatewayviewer.script=/home/sema/ogema/ssh_lsof.sh
org.ogema.tools.gatewayviewer.interval=20000
org.smartrplace.tools.upload.folder=ogemaCollect

Configure Email notification on master

The master sends a summary email to all Recipients configured for priority "low" once a day. For this reason an email sender needs to be configured as well as recipient(s).

App and resource structures

On the master you will find a ResourceList controlledTestGateways, which contains the information directly received from the clients vie RemoteRESTConnector. Furthmore there is a resource of type RemoteSupversionMaster called gatewayInfos that contains information on each gateway generated on the master. Furthermore this resource contains an element "aggregated data", which represents a "virtual remote gateways" that contains aggregated information (e.g. the average on all average temperature values from the gateways).

The app gateway-master-v2 shall be adapted to the evaluation requirements of each project or shall collect evaluations from different projects (to be discussed / checked during project developments).

Note that the Master still contains a lot of project specific and sometimes outdated code that most likely will require further structuring.

Network state management: Handling connections problems

During the automated transfer of data via RemoteRESTConnector and File Upload Servlet network states and errors have to be handled appropriatly to allow for optimal automated transmission. This is a collection of relevant information:

  • The source file/directory is not present. This is a common issue with sending SlotsDB data when not data is available for a certain time stamp/data row. This should be fixed in remote-supervision-util
  • No connection is possible to the server
    • No network connection at all: With an IP address as destination this gives a NoRouteToHostException, with an URL an UnknownHostException
    • No internet connection: With an IP address this gives an HttpHostConnectException, with an URL an UnknownHostException
      On the server this gives a com.sun.mail.util.MailConnectException.
    • Server unreachable: This case has been tested and produces an HttpHostConnectException, which is caught in FileUploadUtil.sendFile. It remains to be checked what happens without network/internet connection.
  • The server refuses to accept the data: The server used not to accept files that are already present (e.g. sent before; gave HTTP 500 error), but that should be fixed.

sema-specific: SEMA-Punkteberechnung

  • The sema-api defines service interfaces for LevelsCalculation and PointsCalculation. The API-bundle is required on the client and on the server.
  • Central Logic for calculation of levels and points is the bonus-server-app. It triggers the services for level/point calculation, gets the data for them from server-backup-analysis and writes results into the remote supervision resources.
    If more than one service is registered for a function (e.g. heating point calculation) it uses the service with the highest priority, declared by the service like:
    @Property(name=Constants.SERVICE_RANKING,intValue=Integer.MIN_VALUE)
  • Standard services actually implementing the level/point calculation are defined in sema-point-calculation
  • The main GUI for triggering level/point calculations manually is sema-test-app/AlgoPage
  • On sever property de.iwes.sema.points.interval.length=10800000 (3 hours) required
Tags:
Created by David Nestle on 2018/04/06 08:47