Remote Telemetry Data Acquisition

A full stack remote data acquisition project monitoring temperature every 5 seconds and using a WiFi link, the remote telemetry unit (RTU) connects to an Internet based data acquisition server (could be MQTT or HTTPS) to register the unit id, temperature and the time of the transmissions from the RTU.

An Internet facing API is available to retrieve and consume services based on the registered data and make it available for monitoring systems like mobile applications to get the current temperature and an oversight of how it changed looking back in time, or be alerted when temperatures exceed defined settings.

Remote Telemetry Reception

The section above is about reading data sensors, formatting the data into JSON data stream and dispatching it to a remote server. This is about receiving data from remote telemetry sensors.

There are many ways to develop APIs like web services frameworks and SOAP, however these are much to complex and cumbersome for this purpose. Considering we already have a firewall and surrounding processes monitoring inbound traffic as well as using HTTPS as transport medium, a single class with an Internet aware interface will do just fine.

Inbound data handler and decoder class

Simple straight forward JSON data telemetry receiver and decoder. The complete class can be downloaded here.

How it works: when requests to the API URI are received by the server, they are routed to this class for initial processing. The data stream from the request is read in (note the web server is monitoring for max data length and buffer overflow), checked if it is JSON data and then decoded according to the telemetry data construct (ENUM). Should the data decode correctly it is then offered to the database service for storage in the database. The response back to the sending party if a good telemetry data packet was received is an 'OK' for success or a 'NOK' for a failure to register the data.

Public Sub ProcessRequest(context As HttpContext) Implements IHttpHandler.ProcessRequest
            context.Response.Clear()
            Try
                '// Check for 'signature' in header of request, if found continue else report IP to interactive FW and drop request
                '// ....
                Dim strJson As String = New StreamReader(context.Request.InputStream).ReadToEnd()
                If Not String.IsNullOrEmpty(strJson) Then
                    Dim objMeas As MeasureInfo = JsonConvert.DeserializeObject(Of MeasureInfo)(strJson)
                    If DataService.PutLoggerData(objMeas.mac, objMeas.ip, objMeas.temp) Then
                        context.Response.Write("Ok")
                    Else
                        context.Response.Write("Nok")
                    End If
                Else
                    context.Response.Write("No Data")
                End If
                context.Response.ContentType = "text/plain"
                context.Response.Cache.SetCacheability(HttpCacheability.NoCache)
                context.Response.Cache.SetAllowResponseInBrowserHistory(False)
                context.Response.Flush()
                context.Response.Close()
            Catch ex As Exception
                context.Response.Flush()
                context.Response.Close()
            End Try
        End Sub

Data processing and forwarding

As the data receiver class is potentially connected directly to the Internet, isolating it from processes deeper in the systems is a smart idea. In order the get the checked and verified data into the database another (data) class that utilizes data base stored procedures is implemented. This prevents any direct access to the database from Internet facing processes. This is the call to

DataService.PutLoggerData(objMeas.mac, objMeas.ip, objMeas.temp)

Public Shared Function PutLoggerData(mac As String, ip As String, value As String) As Boolean
        Dim ok As Boolean = False
        Try
            Dim ConnectionString As String = ConfigurationManager.ConnectionStrings("YOUR_DB_CONNECTIONSTRING").ConnectionString
            Using Connection As SqlConnection = New SqlConnection(ConnectionString)
 
                Using Command As SqlCommand = New SqlCommand("put_logger", Connection)
                    Command.CommandTimeout = 2
                    Command.Parameters.AddWithValue("@mac", mac)
                    Command.Parameters.AddWithValue("@ip", ip)
                    Command.Parameters.AddWithValue("@value", value)
                    Command.CommandType = CommandType.StoredProcedure
                    If Connection.State = ConnectionState.Closed Then
                        Connection.Open()
                    End If
                    Command.ExecuteNonQuery()
                    ok = True
                End Using
            End Using
        Catch ex As Exception
            Errorhandler.ErrorHandler(0, ex.ToString)
        End Try
        Return ok
    End Function

Database Storage

To store the telemetry data, at this point two things are needed. A database table to store the information and the stored procedure to put the data into the data table. In this instance a MS SQL backend server is used located on the local LAN network.

Data example

Example of logged data as stored in the data table on the database.

The 'id' column is the primary key with a self incrementing number. This way we can guarantee that if the numbers are consecutive, the database is intact and no data has been removed.

The date is populated with the UCT date time of the data record entry into the database. If it is important to register the time the data value was actually measured, then that will need to be sent from the remote telemetry unit.

The MAC and IP addresses serve to identify from which unit the data transmissions origionated from.

The product id, not used at present, is to associate the measured reading to a particular product, location, whatever.

As time goes by and more data is collected other processes can start using the historic data, like daily, weekly, monthly and yearly averages with the ability to compare current values to historic values, analyse the differences and determine trends.

Environmental monitoring proof of concept.

My better half who is studying applied science and in her final year for getting her degree, as I write this, questioned me about monitoring CO² gas concentrations in a small closed environment for a period of about 20 weeks. The first thing that interested me was, how do you measure the amount of CO² in the atmosphere accurately. Surprizingly Wikipedia about CO² has information on the gas itself, but no information on detection techniques and how to measure it in atmosphere on tera firma, unless you have a spacecraft, and I don't.

Turned my research to environmental engineering and found that there were multiple ways to do this, with the 'most' accurate and cost effective method using the fact that the gas CO² absorbes infrared light at a specific frequency, now there is something we can work with! Again to my surprise, "CO² sensor modules" were readily available, even with direct I²C communication options, and calibrated as well (?), ready to be integrated with any microcontroller. Could not resist to try and get some operational real data...

The graph below is the result of combining a SCD43 I²C gas sensor to a ESP8266 programmed to take reading every 5 minutes and send these via WiFi to a network server which checks and registers the values in a database. From here an interface via a web server is available to request the results the graph displays.

Quick explanation of the graphic: The sensor is in a room 6 x 4 meters with a single pane window of 3 x 4 meters. The room has permanent inhabitants that appeciate a temperature not under 16°C. Regular drop and rize in temperature (with humidity following) are the result of an ATmega328P connected to a DHT22 sensor and using a solidstate relay to control a small 230 Vac fan heater. If you see regular short term spikes in humidity and temperature, it means it's winter season at this location. Regarding the CO² levels. When the room is occupied by one or more humans, mainly myself for a period of time, these levels rise. Once the room is vacated, they drop.

Real world application

Near real time track and trace system for vehicles using location and data telemetry for monitoring and management. This system uses GPS (Global Positioning System) for location detection, an accelerometer for measuring gravitational forces and has an interface to the vehicle CAN bus. Telemetry dispatches are determined by predefined conditions such as distance, change in rotation angle, accelerometer values and time. When any of the dispatch conditions are met, telemetry is dispatched via cellular networks to a telemetry endpoint server. Here the data is checked, decoded and processed before sending it to a data base server for storage and later processing.

Processed telemetry overview

Collected data is processed and stored, from here reporting and analytics can be undertaken to produce information about the whereabouts of the vehicle as well as fuel use, speed limit adherence, max and average speed for a trip, the amount of CO2 produced and how long the trip took from star address to stop address. From this overview it is possible to drill down on various options, like where exactly were the speed limits exceeded, or where was the sharp corner that registered .57G. The option to view the trip on the map showing where and how the vehicle drove from point A (start trip) to point B (end trip)

Advantages

• The distance the vehicle traveled determined by the odometer at the start and end of the trip, using GPS telemetry point locater coordinates to determine distance is not accurate enough.
• Fuel consumption. Accurate fuel consumption from the vehicle motor management system ensures correct fuel readings as these can vary per trip depending on weather conditions, terrain, and the (heavy) foot of the driver.
• Time management. How long it took to get from point A to point B. It could be there was a traffic jam or roadworks, making it difficult to predict when a vehicle will arrive at it's destination. Real time telemetry allow insight into where the present location of the vehicle is as well as its speed.
• Register points of interest, like speeding in a sharp corner (the G-forces), Abnormal acceleration (pulling away with screeching tires) or standing hard on the brakes to avoid an accident and last but not least registering an accident as it happens.

Telemetry mapping overview scope

Fast and effective overview of all telemetry units with easy identity and physical location projected on a map. The ability to select a unit and see its trip progress from where it started the current trip. Also having the ability to show all transmission points for said unit for the complete day to date.

Telemetry trip points

Show interactive trip point transmissions. These are colour coded and can have icons. The colours indicate whether the situation is normal and within all set boundaries (green), if boundary settings are reached or slightly higher (orange) or if boundaries have been exceeded (red), like speeding more than 5 kilometers above the set legal limit.

Telemetry data details

Each received telemetry data packet can contain a host of information and is determined by the use and value of each data item in the telemetry message constructor. It is a smart device that saves of fuel use, extends tire life and saves on maintenance. It also allows for insight in the motor management system of the vehicle and provide information on the actual state of the vehicle.

This process operates making use of Microsoft Server, SQL server, IIS, ASP.NET, JSript and C#.

Telemetry and hydroponics

Small scale remote bidirectional telemetry using microcontrollers, sensors, actuators, embedded web server and web sockets to monitor and manage the cultivation of plants.

This process operates making use of remote DataBase, ESP32 RTOS, JSript, C++ and C#.