Tag Archives: IoT

IoT Workshop (Part 3)

In the first session of the IoT workshop we presented a general introduction to the LoRaWAN IoT radio and The Things Network. In the second session we looked in more detail at getting an Arduino powered node connected to The Things Network and transmitting data packets.

In the third session, we are taking the data and looking at how it can be processed, in this case using the NodeRed tool.

Installation of Node-RED

Node-RED is a web server program which uses Node.js, which can be installed on a home computer, server, Raspberry Pi or in the Cloud. It can be used to collect and process the data collected by The Things Network[1].) We will be using MQTT to get access sensor data.

For supported Operating Systems and installation instructions, see: https://nodered.org/docs/getting-started/

On Linux Operating Systems (Ubuntu, Debian, Endebian etc.), Node-RED can be installed with

bash <(curl -sL https://raw.githubusercontent.com/node-red/linux-installers/master/deb/update-nodejs-and-nodered)

sudo systemctl enable nodered.service

It also appears to be installable on Windows 10 using the Windows Subsystem for Linux (WSL)[2].

Once installed, ot can be accessed via web browser connecting to port 1880. If it is installed on the local computer, ot can be accessed by visiting the URL – http://localhost:1880/

A User Guide with Tutorials are available from the Node-RED website[3]. If you have access to a working Node-RED server then the Tutorials are a good way to get started.

One final note. Flows are stored on the computer under the computers name (hostname). It that is ever changed, then the flows will not be loaded and it will look like your configuration has dissapeared. It is still there, and will reappear if the hostname is changed back. The easiest way to fix this is to export your flows to a file first (json) and then import them afterwards. This option is under the main menu.

Notes:
[1] Node-RED is a general purpose automation framework and can do a lot more than mentioned here,
[2] https://schellingerhout.github.io/bash%20on%20ubuntu%20on%20windows/nodered-windows/
[2] https://nodered.org/docs/

Working with Node-RED

Once Node-RED has been installed it can be accessed via port 1880.

Some additional packages are required. These are installed via the Manage Palette menu.

node-red-contrib-persist
node-red-dashboard

Configuration is done by connecting processing nodes with virtual wires to create flows. This allows messages to be processed as they pass through the system.

A Node-RED flow configured to process IoT data.

Data collected by The Things Network is pulled into Node-RED by using the MQTT node. This requires the name of The Things Network application and an application key/password.

Add an MQTT node and add a new mqtt-broker (server)

  • Name: Something memorable eg. Meshed
  • Connection
    • Server: thethings.meshed.com.au
    • Port: 1883
    • SSL/TLS: (Unchecked)
    • Use Legacy MQTT 3.1 support: (Checked)
  • Security
    • Username: enfieldlibrary_iot_trial
    • Password: (application-key copied from TTN website, starts with ‘ttn-account-v2’)

The topic is the MQTT channel to subscribe to where ‘+’ is a wildcard and will match any device name.

enfieldlibrary_iot_trial/devices/+/up

There are a range of channels defined in the The Things Network API[1] covering different types of messages. Eg. up, down.

Node Decription

MQTT client

This node connects to The Things Network and subscribes to the defined topic. Any data sent by our LoRaWAN devices will be sent out this node.

JSON

This node converts a text payload to JSON format. This is useful for allow later nodes to access individual parts of the payload.

Switch

Diverts messages between different  processing flows. In the above flow, it is being used to send device data to different dashboard labels. This allows us to display the latest data coming from each sensor.


Function

Code can be written in JavaScript to manipulate the messages passed to the node.

In our example we take the data transmitted by our device and received by The Things Network (eg. Sensor: 0x0267)[2], extract the data (substring) and convert to a meaningful number (moisture). This is passed directly to the next node in the payload of the message (the ‘msg’ variable).

var payload = msg.payload.payload_fields.receivedString
payload = payload.substring(10)
var integer = parseInt(payload,16)
var moisture = integer/1024.0 * 100.0
msg.payload = moisture

// To allow plotting of multiple series
msg.label = "sensor-1"

return msg;

Dashboard Label

Creates label elements to display on the dashboard. These are updated by messages.

Dashboard Graph

Displays a graph from data. By default, the data is not permanantly stored and will not persist between flow deployments. The graph data can be stored persistantly by including save and restore nodes.

Persist (Save Data / Restore Data)

These nodes are used to ensure that the graphed data is persistent between reboots. Under particular conditions the date is written to a non-volatile location, and is read back into the system when starting up.

Multiple sets of data can be saved, but the save and restore nodes need to match.

Notes:
[1] https://www.thethingsnetwork.org/docs/applications/mqtt/api.html
[2] For the workshop there was a deliberate choise made to use ASCII strings to encode the data. This is inefficient, but allows the data to be easily viewed for training and debugging purposes. Have a look at Cayenne Low Power Payload (Cayenne LPP) for one encoding method to improve this.

Summary

The nodes above can be used in Node-RED to receive, display, graph and store data from LoRaWAN enabled sensors. It is a general purpose system and as such is missing some of the whistles and bells of some other systems.

Improvements can be made by hooking in some additional software packages. In particular, data can be stored in a database specifically designed to hold time series of sensor data (eg. InfluxDB[1]) and a more functional dashboard can be added for better graphing and analysis (eg. Grafana[2]).

Notes:
[1] https://www.influxdata.com/
[2] https://grafana.com/

IoT Workshop (Part 2)

In the previous post we discussed the LoRaWAN network and gave an overview of how it works end to end. This post looks in more detail and describes what’s necessary to get data into the network from individual sensors.

The LoRa Radio and LoRaWAN

The LoRa radio module transmits in the Industrial , Scientific and Medical (ISM) unlicensed radio band[1]. In Australia (and New Zealand) LoRa transmits between 915-928 MHz[2]. For uploading data, the band is divided into 64 narrow bandwidth channels of 125 kHz, with an additional 8 overlapping broader channels of 500 kHz.

The data is encoded using a chirp signal. The rate that this frequency changes is defined by a ‘spreading factor’ (SF), and a doubling of the spreading factor approximately doubles the amount of time it take to transmit data[3].

LoRaWAN Channel Allocation for Australia and New Zealand

Notes:
[1] ISM Band – https://en.wikipedia.org/wiki/ISM_band
[2] https://lora-alliance.org/sites/default/files/2018-05/lorawan_regional_parameters_v1.0.2_final_1944_1.pdf
[3] https://medium.com/home-wireless/testing-lora-radios-with-the-limesdr-mini-part-2-37fa481217ff

Hardware

Dragino Arduino Shield

To use LoRa and LoRaWAN we are using the following LoRaWAN Arduino Shield available from the local suppliers. It is supplied to transmit and receive in the 915 MHz. As purchased the jumpers were not set, and needed to be installed as indicated in the picture below.

When developing hardware, it is only possible to use digital input/outputs 3,4 and 5 (3 lines). Digital IO 0 and 1 are used by serial IO and interfere with programmng. The digital lines D11-13 are used to control the LoRa radio.

The available Arduino pins for projects are

  • D3-5 (3 digital lines) – Can be sed for Digital IO
  • D0-1 (2 digital lines) – Can be used for Digital IO but interferes with in circuit programming. Needs to be disconnected first.
  • A0-5 – Canbe used for Analog Input or Digital IO
  • SCL,SDA – Can be used for Digital IO and Analog In on Arduni Uno.
  • AREF – Used as analog reference voltage
  • GND – Ground

Software

Arduino Library for LoRaWAN (LMIC)

The Arduino LMIC library that we are using is being maintained by Thomas Laurenson and can be found on Github[1]. The original LMIC Library was written by IBM and has been ported for use with the Arduino board.

Once downloaded and installed in the Ardiuino IDE library directory[2], the software needs to be configured to use the Australian frequency plan which is done by editing the file ‘project_config/lmic_project_config.h’. For some reason the Australian configuration is called CFG_au921.

// project-specific definitions
//#define CFG_eu868 1
//#define CFG_us915 1
define CFG_au921 1
//#define CFG_as923 1
// #define LMIC_COUNTRY_CODE LMIC_COUNTRY_CODE_JP       /* for s923-JP */
//#define CFG_in866 1
define CFG_sx1276_radio 1
//#define LMIC_USE_INTERRUPTS

Notes:
[1] Github: https://github.com/thomaslaurenson/arduino-lmic
[2] When unzipping the software, there may be duplicated recursive top level directory names (eg. arduino-lmic/arduino-lmic). Ensure that only one level is copied into Arduino library folder.

Example Arduino Sketch

An example Arduino sketch also available from the GitHub repository, which can be used to connect to the LoRaWAN network. See arduino-lmic/examples/ttn-otaa-dragino-lorashield-au915[1].

When connecting to the LoRaWAN network there are two authentication methods “Authtication by Personalisation” (ABP) and “Over the Air Authentication” (OTAA). While it is a little more complicated, OTAA is preferred as it allows the network more control over the authentication process, but requires two-way communication with the end-node. ABP does not require the end-node to receive data from the network but the network needs to know when the end-node is ever reset[2].

For OTAA, there are three pieces of data that need to be copied into the sketch, and are obtained from The Things Network console[3].

  • Device EUI (DEVEUI) – This is the unique identifier for this device on the network. This can be changed. This needs to be used in the sketch in Least Significant Byte (lsb) order.
  • Application EUI – This is the application identifier, and is unique across the LoRAWAN network. This needs to be used in the sketch in Least Significant Byte (lsb) order.
  • Application Key – This is an initial encryption key shared between the Node and Application. It can be changed. This needs to be used in sketch in Most Significant Byte (msb) order.

The webpage tries to make this as easy as possible by allowing you to adjust the formatting and the copy to clipboard button.

Device confiuration parameters in The Things Network

These details are copied into the Arduino sketch in the FILLMEIN locations as shown below.

// This EUI must be in little-endian format, so least-significant-byte 
// first. When copying an EUI from ttnctl output, this means to reverse
// the bytes. For TTN issued EUIs the last bytes should be 0xD5, 0xB3,
// 0x70.
static const u1_t PROGMEM APPEUI[8]= { FILLMEIN };
void os_getArtEui (u1_t* buf) { memcpy_P(buf, APPEUI, 8);}

// This should also be in little endian format, see above.
static const u1_t PROGMEM DEVEUI[8]= { FILLMEIN };
void os_getDevEui (u1_t* buf) { memcpy_P(buf, DEVEUI, 8);

// This key should be in big endian format (or, since it is not really a
// number but a block of memory, endianness does not really apply). In
// practice, a key taken from the TTN console can be copied as-is.
static const u1_t PROGMEM APPKEY[16] = { FILLMEIN };
void os_getDevKey (u1_t* buf) { memcpy_P(buf, APPKEY, 16);}

In the sketch, the following line can also be changed.

static uint8_t mydata[] = "OTAA" 

This is the data that is going to transmitted over the network. This short string can be edited to something more meaningful. The sketch can then be compiled and installed, and if a LoRaWAN Gateway is within range, transmitted data will start to be collected by The Things Network Application.

Notes:
[1] Github: https://github.com/thomaslaurenson/arduino-lmic/tree/master/examples
[2] The packet counters need to be reset. This is used to stop data ‘replay’ attacks.
[3] Console->Applicaion->Devices

LoRaWAN Data

The transmitted packet data will appear under the Application Data on The Things Network.

Data packets being received from an IoT device via LoRaWAN

The information includes the data transmitted by the IoT device and metadata from the network about the transmission. Expanding the packet entry shows this detail.

In the case above, to assist with debugging a custom Payload Format decoder has been added which converts the bytes to a string with can be seen in the ‘receivedString’ field[1].

// Decode an uplink message from a buffer
// (array) of bytes to an object of fields.
function Decoder(bytes, port) {
// var decoded = {};
// if (port === 1) decoded.led = bytes[0];
// return decoded;

// Decode plain text; for testing only
return {
receivedString: String.fromCharCode.apply(null, bytes)
};
}

The raw transmitted data can be seen in the Payload.

The metadata fields show the frequency used (917.8 MHz), the datarate (SF7BW125 – Spreading Factor 7, Bandwidth 125 kHz) and the gateways which heard the transmission and their details.

Notes:
[1] https://core-electronics.com.au/tutorials/encoding-and-decoding-payloads-on-the-things-network.html

Next Time (Part 3)

The next step will be to look at how to make use of the collected data. NodeRed is a software package will be used to process and manipulate the data as it is received.

IoT Workshop

On Thursday (24 Oct 2019) we started running the first session of a new IoT workshop, being held at the local libray (Enfield Branch of the Port Adelaide Enfield Council). This workshop aims to be an introduction to creating sensors and nodes for the LoRaWAN IoT network, using The Things Network website.

This first sesssion (of four) was a whirlwind dump of details which we will unpack and work on through the remainder of the sessions, with hands on electroncs and programming.

We went though the architecture of the system and got everyone to create an account on The Things Network, and set them up so that they could see the IoT data that was being transmitted over the network.

Architecture of LoRAWAN Network (from The Things Network)

Technolgy discussed..

A quick summary of the various bits-n-pieces that are being used follows.

LoRa

The radio technology that is used by sensors for ‘Long Range’, low bandwidth communication. These radios may be programmed to be used in either point-to-point or point-to-gateway (LoRaWAN) mode, configurable with software. They use the unlicensed ISM radio bands, 911MHz in Australia.

https://en.wikipedia.org/wiki/LoRa

LoRaWAN

The radio network and infrastructure used to collect sensor data, via publically accessible gateways.

Arduino

A microprocessor board and programming tools, which use the Atmel (and other) microprocessor. The original aim was to create a programming system which was easy to use for hobbyists and members of the maker community. Arduino microprocessor boards can be used with a LoRa radio to create sensors which can transmit data via LoRAWAN and the internet to be collected and used by other applications. https://www.arduino.cc/

Arduino Shield

This is additional hardware that has been designed to plug directly into an Arduino board and allow it to be easy to use.

Arduino IDE

The Interactive Development Environment (IDE) used to program an Arduino microprocessor board. It is available as a downloadable Java program, and as a web based application. The software is available from the Arduino website.

Arduino Libraries

Additional software packages which can be installed to extend Arduino programs. For example, the LMIC library is used to allow our Arduino programs to use the LoRaWAN radio shield.  

Libraries can be downloaded from within the Arduino IDE, or downloaded and installed manually.

LoRaWAN Sensor Node

A device which collects data and transmits to the LoRaWAN network. It uses a LoRa radio. They need to be registered and authenticated with the LoRaWAN network to transmit correctly.

LoRaWAN Gateway

A device which listens for LoRaWAN radio transmissions from nodes in its area and forwards them to the servers on the LoRaWAN network. They need to be connected to the internet and be registered to operate.

The Things Network

A free website (registration required) which allows access to data collected by LoRaWAN sensors. It is used to register LoRaWAN gateways on the network, and create LoRaWAN applications.

https://www.thethingsnetwork.org/

MQTT

MQ Telemetry Transport. This is the communication software and protocol  used to allow sensor data to be distributed and used once it has been collected by The Things Network.

NodeRed

A web server program, which can be installed on a home computer or server (or Raspberry Pi) which can collect and use sensor data collected. It uses MQTT to access sensor data.

NodeRed is an automation environment and is useful for a lot more than collecting and displaying our IoT data. See the website for more details.

https://nodered.org/

TTNmapper

Mobile application which can be used to map LoRaWAN coverage.  https://ttnmapper.org/

Sensor/Node Example

The following are some example of the sensors that will be built in the workshop.

Moisture Sensor

A soil moisture sensor which sends measurements via LoRaWAN to a monitoring application created NodeRed

The data is displayed in a web browser as the following.

Next Post: IoT Workshop (Part 2)

LoRaWAN Node (Update)

LoRa Node consisting of an Arduino Uno (bottom), Duinotech LoRa Radio shield (middle) and Freetronics prototyping board (top)

The following is a quick update on the construction of LoRaWAN node., which was perviously described here.

After the previous work and the messing around with the uneven Arduino pins, I ordered some Freetronics prototyping boards and continued with those.

With the first prototype, I had attached LEDs to the first two digital pins (D0,D1) which are also used by the USB Serial connection. These made them kind of useless for monitoring any status, but also interfeared with the Arduino sketch programming. Before a new program could be successfully loaded in this configuration, the board first needed to be removed. This got annoying very quickly, particularly when I enjoy having a fast development process.

In the next design I removed these LED’s and connected them up to the supposibly free and available digital pins (D11,D12 and D13). It appears as though the LoRa radio board also uses at least one of these pins out of the box, with configuration jumpers that use the others. If the configuration jumpers are changed, the software would also need to be changed. The board works as expected out of the box with the available Arduino libraries, so it would be a pity to mess this up. Also, the documentation on the boards jumpers, and their orientation isn’t the clearest, so it would be best just to leave this well alone.

The remnants of this work can be seen on the bottom three resistors seen in the picture above. These resistors were connected with hookup wire underneath to pin D11, D12 and D13.

The Arduinio analog input pins, which were still unuised, can be switch to be used as digital outputs. The three LEDs that were connected earlier to the digital pins, were now connected to Analog pins A0, A1 and A2, with addional LEDs (with resistors, more about that sortly) to A3, A4, and A5. They are connected by more hookup wire underneath the prototype board.

Spot the difference.

A sketch was written to test the connected LED’s and was uploaded, without having to remove the prototype board first. This showed that everything was working as expected, except for one thing. The three new LEDs where significantly dimmer than the first ones.

The three new resistors I had grabbed quickly from my resistor box were 1K (Brown-Black-Black-Brown/Brown), rather than 100 Ohm (Brown-Black-Black-Black/Brown). This ment that the current through the LEDs was less, and hence their brightness was less as well. This has not been fixed in this prototype and probably wont be, although I could solder another suitably size resistor in parallel to the 1K ones to made the combined resistance close to the desired 100 Ohm amount.

The uploaded sketch was still contained the ability to set a couple of LEDs based on a byte of data received from the LoRaWAN radio. This was shown to work from The Things Network web, with the next step being able to drive this from custom software. Data from the LoRaWAN node is accesses via the MQTT protocol, and data can be uplinked to the node using the same method.

I have created several utility scripts to test all of this, which can be found on GitHub at: https://github.com/PaulSchulz/ttn-utils

LoRaWAN Node and Gateway

LoRaWAN Node and Gateway

With assistance of the STEM Education resources from the Port Adelaide Enfield Libraries, I have been investigating LoRaWAN devices, how to build them and how they can be used.

There is more investigation to be done, but the node shown in the picture above is able to do the following:

  • Connect and Authenticate to a LoRaWAN network (The Things Network) via Over The Air Authentication (OTAA); and
  • Send data (Uplink) and Receive data (Downlink) via the LoRaWAN Gateway(s). (Downlink data is queued, and is received in a window after Uplink data has been received by the network. This reduces power usage in the node.)

The Node used was the Gragino Arduino Shield (or clone from duinotech)

The Gateway is the uGateway LoRaWAN from Core Electronics. Having a local gateway is not necessary if there is another one in the neighbourhood, within range of your node. There are currently no nodes in my area so an internal gateway was required for development and testing. The other option was to take the Node for a drive in the car after a development session, but this made debugging very tiresom, very quickly.

There are a couple of other useful tools worth mentioning.

TTN Mapper is an Android application that will monitor your device on your smart phone, via The Things Network. It then used your phones GPS to log the position of your device. The result is that you can map the coverage of nearby gateways by using your LoRaWAN node and phone in a similar way to how Wireless Access points used to be mapped (WarDriving). This is useful for knowing where your nodes have TTN access, and where additional gateways might be required.