Tag Archives: LoRaWAN

Getting the Heltec Wireless Lite V3 with SX1262 LoRa Radio Chip to work with LoRaWAN

This article is part 1.5 of the series – Developing a component for ESPHome, for the SX1262 LoRa Radio Chip. The process getting the LoRaWAN functionality of this development board to work with The Things Network proved to be an excercise infrustration management, so the following is a write up of what was required to get it to work. The following is my account of setting up the Heltec Wireless Lite V3 for
connecting to the LoRaWAN network. There were several false starts, but in
the end it was possible to send data and decode it correctly.

Development Environment

ArduinoIDE and the Heltec Board Manager extention

The Heltec boards are defined by the following file (link) which is added through the menu in Preferences…->Additional boards manager URLs

https://github.com/Heltec-Aaron-Lee/WiFi_Kit_series/releases/download/0.0.7/package_heltec_esp32_index.json

This file specifies that esptool 3.3.0 is used for writing the firmware to the Heltec board. There is as issue with usung this version for the Heltec V3 boards, in that the firmware compiles and installs, but puts the ESP32-S3 processor into a boot loop.

The details of this issue is here: https://github.com/Heltec-Aaron-Lee/WiFi_Kit_series/issues/159#issuecomment-1842269504

The fix is to upgrade esptool 4.6.1. The following fix works on Ubuntu, but
it’s a hack. The correct fix would be to change the JSON definition file.

cd ~/.arduino15/packages/Heltec-esp32/tools/esptool_py
mv 3.3.0 3.3.0-bak
wget https://github.com/espressif/esptool/archive/refs/tags/v4.6.1.zip
unzip v4.6.1.zip
rm v4.6.1.zip
mv esptool-4.6.1 3.3.0

LoRaWAN Library

Install the ‘SX126X-Arduino’ library with the library manager (currently version
2.4.0).

A small fix is required. Edit the ‘src/mac/Commissioning.h’ file and comment out the check for whether the LoRaWAN Region has already being set (Line 40).

Example Code – LoRaWAN.ino

From the LoRaWAN library, the LoRaWAN example was copied and modified to get the
system to work.

Modifications:

  • Copy the LoRaWAN.ino file, and remove other unnecessary files. Fix the source to remove the dependancies.
  • Simplify the code. eg. Remove the code sections used for other processors.
  • Set the authentication mode to OTAA, the device type and the Frequency bandplan region to AU915 by adding the optional parameters to ‘lmh_init() – true, CLASS_A and LORAMAC_REGION_AU915.
  • Ensure that the correct LoRaWAN sub-band is being used for AU915 by calling ‘lmh_setSubBandChannels(2)’.
  • Additional Serial.println() output added to trace the execution of the firmware, including the output of the LoRaWAN parameters.

The LoRaWAN parameters (AppEUI, DevEUI and AppKey) are set in the following section.

The Things Network

The Heltec V3 Devices have not been configured as known devices in The Things
Network
, and need to be specifically setup.

Select “Enter end device specifics manually”.

Add a new device and set the following Device Parameters

  • Frequency Plan: Australia 915-928 MHz, FSB 2 (used by TTN)
  • LoRaWAN Version: LoRaWAN Specification 1.0.2
  • Regional Parameters version: RP001 Regional Parameters 1.0.2 revision B

Create a JoinEUI (previously known as the AppEUI). This can be freely set any 8-byte value, but cannot be changed once set.

The other parameters that need to be set are:

  • DevEUI
  • AppKey

The easiest way to to get these values is to allow them to be automatically generated by The Things Network interface. They can be specifically set if the manufacturer of a device has preprogrammed a device with a LoRaWAN configuration.

Build and Install Firmware on Device

Update the definitions of the LoRaWAN parameters in the code to reflect the values just generated. It is possible to directly produce the desired strings (in ‘msb’ format) from The Things Network device interface.

Paste these into the code

uint8_t nodeDeviceEUI[8] = {…};
uint8_t nodeAppEUI[8] = {…};
uint8_t nodeAppKey[16] = {…};

Ensure that these parameters are being set in the ‘setup()’ function with the following:

lmh_setDevEui(nodeDeviceEUI);
lmh_setAppEui(nodeAppEUI);
lmh_setAppKey(nodeAppKey);

It is also helpful to also have these values displayed when the board os booted.

The firmware compiles and can be installed onto the Heltec board. On boot, it should
connect to the LoRaWAN network and start transmitting packets of data.

On The Things Network, to check that the uploaded data is correct, the following JavaScript payload filter can be installed for the device, which will decode the data and
(hopefully) display the correct string.

function decodeUplink(input) {
var result;
result = String.fromCharCode.apply(null, input.bytes);
return {
  data: {
    bytes: input.bytes,
    result: result
  },
  warnings: [],
  errors: []
};
}

Notes, Issues and Errors

  • When selecting and playing around with different LoRaWAN Versions and Regonal Parameters, it appears as though The Things Network does not reset LoRaWAN values properly. Changing the LoRaWAN version to 1.1.0 required the additional parameter ‘NwsKey’. If the LoRaWAN version is then set back to 1.0.2, this value becomes the one that is used for the AppKey. There is a note about this in The Things Network interface, but it is not obvious what is required.
  • Care needs to be taken when setting the nodeDevEUI and nodeAppEUI not to set them the wrong way around. If connection packets are being received from the device, but it isn’t able to authenticate, then the nodeDevEUI will be correct, but the other parameters (including LoRaWAN Version) may be wrong.
  • If only intermittent transmitted packets are seen in The Things Network, and the device is not able to authenticate, check that the firmware has selected and is using the correct LoRaWAN sub-band.
  • Selecting other LoRaWAN versions eg. 1.1.0 will require additional parameters to be set., namely NwsKey. This setting is not removed if the LoRaWAN version is set back to 1.0.2, and this value is then used instead of the nodeAppKey. This is a trap. The only way to fix this is to delete the device definition in The Things Network and start over.

Next Steps

  • Convert the Arduino LoRaWAN firmware to ESPHome. This will allow LoRaWAN to be able to be used with the other ESPHome functions, including support for all of the existing supported sensors and outputs, and also operate with Home Assistant. (This will be Part 2 of the “Developing a component for ESPHome, for the SX1262 LoRa Radio Chip” series.)
  • Test with the Helium network. Register the device on the Helium network and check that data is routed correctly.
  • Implement as Class B and Class C LoRaWAN device. The example code currently only
    uploads data. Class B and Class C deviced allow data to be downloaded as well, for device control.

Developing a component for ESPHome, for the SX1262 LoRa Radio Chip – Part 1

Introduction

This post describes “Work in Progress”. I have deliberately decided to put these details together now as it has been possible to build something that ‘sort of’ works, in that all the pieces are there, ot nearly there. The end is in sight, albeit a reasonably long way away in time and effort.

The original aim was to make the SX1282 LoRa radio chip available, for both LoRa and LoRaWAN modes, directly in ESPHome. It is hoped that the use of the SX126X series of radio chips can be as easy as possible to use and in most cases wholly driven from the ESPHome YAML files. When creating an IoT device with a new dev board, this has been one of the frustrating pieces of the excercise, particularly when using the Arduino IDE and programmer.

The Github repository of the component is here: https://github.com/PaulSchulz/esphome-lora-sx126x

These notes assume that you have ESPHome installed (

To include this external component in your own ESPHome yaml file, see below, or read the repository’s documentation (which will be newer than this).

Hardware

This ESPHome component is being developed on the Heltec V3 Development Boards which use the LoRa SX1262 chip. These dev boards are:

At the present time, these boards are not available for selecting directly in the
ESPHome YAML file (when using the Arduino framework). In order to make these
boards work with the available version of ESPHome, the following YAML is required:

esp32:
  board: esp32-s3-devkitc-1
  framework:
    type: arduino

These board definitions are imported from the platform-espessif32 project here.

The LoRa Arduino Library

The Arduino library being used is beegee-tokyo/SX126X-Arduino, which itself is based on several other libraries. It appear to currently be the best and most maintained LoRa library available for these radio chips.

The following header file contains the descriptions of the functions and fields used by the library: https://github.com/beegee-tokyo/SX126x-Arduino/blob/master/src/radio/radio.h

Testing and Usage

The files lora-sx126x.yaml and secret.yaml provide an example of how to use the
component. To use with a working version of ESPHome, edit secret.yaml to add
your local Wifi details, then run:

esphome run lora-sx126x.yaml

Development Comments

To get the component into a working state, this ESPHome component currently does some things which are not encouraged by the ESPHome Core and Component developers (see below).

It is always hoped that this component will become compliant enough to be
included in ESPHome, but in the meantime, it can be included in your ESPHome
build by either cloning this repository locally and adding an external_component
with a local source; or by including the github repository directly as an
external_component. See the External Components documentation for more details

If downloading and using as a local source for external component:

external_components:
- source:
    type: local
    path: esphome/components
  components: ["lora_sx126x"]

and, using directly from the Github source for an external component

external_components:
- source:
    type: git
    url: https://github.com/PaulSchulz/esphome-lora-sc126x
    ref: main
  components: ["lora_sx126x"]

Considerations / Things to fix

Direct use of SPI and SX126x-Arduino libraries

If possible, the SX126x-Arduino library needs to be implemented natively in
ESPHome, to make use of the native ESPHome SPI code.

By using the Library directly, it is uncertain at the moment whether this component can be used generally with other devices that use the same SPI interface.

Example YAML

The following is the proposed example YAML configuration, for a ESPHome image that will enable the SX126X radio.

esphome:
  name: "lora-sx126x"
  libraries:
    - "SPI"
    - "Ticker"
    - "SX126x-Arduino"

...

external_components:
  - source:
      type: local
      path: esphome/components

...

lora_sx126x:
  # The frequency to use needs to be specified as will depend on the
  # hardware and the region in which the it is being used.
  frequency: 915000000

Development

Proposed YAML Options

The following is an example of the proposed full option list, for using the LoRa radio chip.

lora_sx126x:
  # optional, with sensile defaults, if possible from board id.
  pin_lora_reset: 12
  pin_lora_dio_1: 14
  pin_lora_busy: 13
  pin_lora_nss: 8
  pin_lora_sclk: 9
  pin_lora_miso: 11
  pin_lora_mosi: 10
  radio_txen: -1
  radio_rxen: -1
  use_dio2_ant_switch: true
  use_dio3_tcx0: true
  use_dxo3_ant_switch: false

  # required - depends on region and frequency band being used
  rf_frequency:           915000000
  # optional (sensible defaults)
  tx_output_power:               22
  lora_bandwidth:                 0
  lora_spreading_factor:          7
  lora_codingrate:                1
  lora_preamble_length:           8
  lora_symbol_timeout:            0
  lora_fix_length_layload_on: false
  lora_iq_inversion_on:       false
  rx_timeout_value:            3000
  tx_timeout_value:            3000

It should then be possible to use the radio with the various builtin types. This has yet to be implemented.

text_sensor:
  - platform: lora_sx126x
    id: lora_message
    name: LoRa Message

# Is there a component for this in ESPHome?
# Sending a string to a component?
text_message:
  - platform: lora_sx126x
    id: send_message
    name: "Send LoRa Message"

binary_sensor:
  - platform: lora_sx126x
    id: lora_sensor
    name: LoRa Sensor
    on_string: "@+++"
    off_string: "@---"

switch:
  - platform: lora_sx126x
    id: lora_switch
    name: LoRa Switch
    on_string: "@^^^"
    off_string: "@vvv"

binary_input:
 - platform: lora_sx126x
   id: lora_input
   name: LoRa Binary Input
   on_string: "@***"
   off_string: "@..."
  
binary_output:
 - platform: lora_sx126x
   id: lora_output
   name: LoRa Binary Ouput
   on_string: "@>>>"
   off_string: "@<<<"

Setup and Configuration of a Commercial LoRaWAN Temperature Sensor

Introducton

This document is part of the PAE-IoT Parks and Gardens Project.

The aim of this post is to document the configuration details needed to setup a new temperature sensor which was deployed as part of the Port Adelaide-Enfield Library/Parks and Gardens collaboration, for monitoring the environment around a newly built sports ground and playing surface.

It is a fairly long document and aims to explicitly describe the code that is needed to set up the new data path. It is built on top of the existing data handling system (The Things Network, NodeRED, InfluxDB and Grafana) which has been described in previous posts, so it isn’t a complete explaination of this system.

Grafana graph from installed temperature sensors

More details about the project, including the live sensor data, can be found on the project pages:

The Hardware – Temperature Sensor

General information (from The Things Network)

Device Id:       eui-a840416eb183975f
Description:     LSN50V2-D23 LORAWAN WATERPROOF TEMPERATURE SENSOR (3 PROBES)
Frequency plan:  Australia 915-928 MHz, FSB 2 (used by TTN)
LoRaWAN version: LoRaWAN Specification 1.0.3
Regional Parameters version: RP001 Regional Parameters 1.0.3 revision A

Hardware

Brand:            dragino
Model:            lsn50v2-d20-d22-d23
Hardware version: unknown_hw_version
Firmware version: lsn50v2-d20-d22-d23 1.7.5

Note: The LoRaWAN Regional Parameters is set to “Version 1.0.3 Revision A”. This is different to what was found to work with custom developed sensors. It is unclear at this stage what the actual difference is.

The Things Network (TTN) Configuraton

The sensor was easily registered with The Things Network, as the hardware has been pre-configured with a hardware profile from The Things Network Device Repository. This configures the network parameters (as described in the previous section) as well as data formatting functions.

The current Formatter Code returns four data points (Temp_Black, Temp_Red, Temp White, and BatV) and two status values (Work_Mode and ALARM_status).

The code for this uplink data formatting function is as follows:

function decodeUplink(input) {
	var port = input.fPort;
	var bytes = input.bytes;
	var mode=(bytes[6] & 0x7C)>>2;
	var data = {};
	 switch (input.fPort) {
		 case 2:
if(mode=='3')
{
  data.Work_mode="DS18B20";
  data.BatV=(bytes[0]<<8 | bytes[1])/1000;
  data.ALARM_status=(bytes[6] & 0x01)? "TRUE":"FALSE";
  
  if((bytes[2]==0xff)&& (bytes[3]==0xff))
  {
    data.Temp_Red="NULL";
  }
  else
  {
    data.Temp_Red= parseFloat(((bytes[2]<<24>>16 | bytes[3])/10).toFixed(1));
  }

  if((bytes[7]==0xff)&& (bytes[8]==0xff))
  {
    data.Temp_White="NULL";
  }
  else
  {
  	data.Temp_White=parseFloat(((bytes[7]<<24>>16 | bytes[8])/10).toFixed(1));
  }
  
  if((bytes[9]==0xff)&& (bytes[10]==0xff))
  {
    data.Temp_Black="NULL";
  }
  else
  {
  	data.Temp_Black=parseFloat(((bytes[9]<<8 | bytes[10])/10) .toFixed(1)); 
  }
}
else if(mode=='31')
{
  data.Work_mode="ALARM";
  data.Temp_Red_MIN= bytes[4]<<24>>24;
  data.Temp_Red_MAX= bytes[5]<<24>>24; 
  data.Temp_White_MIN= bytes[7]<<24>>24;
  data.Temp_White_MAX= bytes[8]<<24>>24; 
  data.Temp_Black_MIN= bytes[9]<<24>>24;
  data.Temp_Black_MAX= bytes[10]<<24>>24;  
}

  if(bytes.length==11)
  return {
      data: data,
    }
	break;
default:
    return {
      errors: ["unknown FPort"]
    }
  }
}

The data that is then sent through to NodeRED via the MQTT service is a text formatted JSON data structure.

Comments

In previous PAEIoT projects, the decoding of the LoRaWAN uplink data has been done at a later step, in NodeRed. It was done like this for two reasons: 1) The encoded LoRaWAN packets were smaller for transmitting over the network; and 2) Understanding how to decode and process the data could be done separately to anything that TTN does.

Both of these conditions have changed.

In Version 3 of the TTN service, a lot more meta-data about the network is sent with the sensor data, so any advantages of savings made with minimising the size of the internet packet is lost.

As the uplink data formatting code is automatically included by the manucaturer, is available and open-source, provided thatit works, there is no advantage to not using it.

NodeRed and InfluxDB – Data Procesing and Storage

The following flow diagram describes the new path and changes added to support the new temperature sensor.

Specifically, the following changes and details were added.

TTN Decode (minor change)

The “TTN Decode” node now also passes the entire “uplink_message”, rather than just “payload_raw”. This means that the sensor location (longitude, latitude, and elevation) can be set via the registraion of the device in TTN.

payload.dev_id – Switch (New Output Added)

Added switch to separate flow for the Parks and Garden device (dev_id=eui-a840416eb183975f).

Note: When additional sensors are added, this switch node will need to be changed to also select the new sensor.

Parks Decode – Function (New)

This is the main function which pulls out and renames the data. The ‘msg.payload’ contains this data, and ‘msg.meta’ contains tags for the data.

// Parks Decode
// Decode temperature sensing device
// LSN50V2-D23 LORAWAN WATERPROOF TEMPERATURE SENSOR (3 PROBES)

var device_id = msg.payload.dev_id;

var batv       = msg.payload.uplink_message.decoded_payload.BatV;
var temp_black = msg.payload.uplink_message.decoded_payload.Temp_Black;
var temp_red   = msg.payload.uplink_message.decoded_payload.Temp_Red;
var temp_white = msg.payload.uplink_message.decoded_payload.Temp_White;

var latitude  = msg.payload.uplink_message.locations.user.latitude;
var longitude = msg.payload.uplink_message.locations.user.latitude;
var altitude  = msg.payload.uplink_message.locations.user.altitude;

var data = {};
data.batv       = batv;

data.temp_black = temp_black;
data.temp_red   = temp_red;
data.temp_white = temp_white;

data.temp_air     = temp_white;
data.temp_surface = temp_red;
data.temp_ground  = temp_black;

data.latitude   = latitude;
data.longitude  = longitude;
data.altitude   = altitude;

msg.payload = data;
msg.meta = {
    "device_id": device_id,
};

return msg;

InfluxDB Encode – Function (Copied)

This function is the same as prevously used to format the payload prior to sending it to the InfluxDB node (and into the database).

// InfluxDB Encode
var bucket    = "paeiot-bucket"

var device_id = msg.meta.device_id;
var fields    = msg.payload;

msg.payload = {   
  bucket: bucket,
  data: [{
    measurement: 'sensor',
    tags: {
      "dev_id":  device_id
    }, 
    fields: fields
  }]};

return msg;

InfluxDB – Database Node (Copied)

This uses the existing configured node without any changes.

Grafana Dashboard

A new folder and dashboardwas created to display the temperature data.

Parks and Gardens / Playing Ground Temperatures

The query used to return the air temperature (temp_air) is the following:

from(bucket: "paeiot-bucket")
 |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
 |> filter(fn: (r) => r["_measurement"] == "sensor")
 |> filter(fn: (r) => r["dev_id"] == "eui-a840416eb183975f") 
 |> filter(fn: (r) => r["_field"] == "temp_air")
 |> drop(columns: ["dev_id", "latitude", "longitude", "altitude"])
 |> aggregateWindow(every: v.windowPeriod, fn: mean)
 |> yield(name: "mean")

An addtional two queries pull out the data for the surface temperature (temp_surface) and ground temperature (temp_ground) with the filter for the ‘_field’ value being changed respecfully. eg.

 filter(fn: (r) => r["_field"] == "temp_surface")
 filter(fn: (r) => r["_field"] == "temp_ground")

The resulting graph (as shown at the top of this post) also has some additional formatting changes (overrides), as pseudo-code:

Where field has name "temp_air", change Display Name to  "Air";
Where field has name "temp_surface", change Display Name to  "Surface";
Where field has name "temp_ground", change Display Name to  "Ground";

Conclusion

The existing PAE-IoT data processing system was modified to allow data from a newly added, commercially available temperature sensor, to be stored, displayed and dynamically updated.

A moderate amount of changes were required due to choices made by the hardware and system vendors. The system design choices that were made were informed by the desire to minimise additional changes if more sensors are added or if the system is extended in other straight forward ways.

This documentation is part of this process.

New Toy (the reMarkable2) and a discussion about LoRaWAN and IoT

As a birthday present for myself earlier in the year I purchased a “reMarkable 2” e-ink tablet and it’s been awesome. More details about the device below (for those that are interested) but I recently had the chance to use it in a 1-1 meeting covering the PAE IoT Workshop and it worked really well as a paper notebook substitute. Afterwards I was able to go over the details, tidy them up, and rearrange them into a more sensible layout (rather than the original messy stream of consciousness).

Drawing on the reMarkable2

The tablet allows hand written text and drawings to be systematically edited, rearranged and replaced. There is the option on having layers (which I didn”t really use) and showing an underlying templete (lots to choose from) which is a useful guide for keeping everything nicely lined up, and which can be removed afterwards (replaced with the blank one).

The final result can emailed as PDF, PNG, or SVG formatted files, and the result is below.

About IoT Data Logging using LoRaWAN

The following is an example of the path which might be taken by a packet of date, created on a LoRaWAN IoT device, as it makes it’s way from the sensor to be displayed on a web-based dashboard on the Internet.

About the reMarkable2

There is quite a lot to talk about with the reMarkable2 but where is a vary brief summary – it’s a Linux based ARM system, which has Wifi access and drives a e-Ink display with stylus for drawing and input. The display and pen software is closed source, but everything else appears to be Free and Open Source Software. It is possible to SSH into the device from both Wifi and USB-C, with a password that is displayed from the settings menu. If you wanted to you could use that the change any of the software on the device but beware – there be dragons.

reMarkable (the company) currently send out software updates regularly (almost monthly), as these also include new features and tweeks.

There are also third party projects (the reHackable project) which make the tablet even more useful, and support the use of the reMarkable in many more ways than the manufacturer may have intended, including replacing the entire operating system. I have not considered doing this just yet.

IoT Workshop (Part 3)

In the first session of the IoT workshop we presented a general introduction to the LoRaWAN IoT radio and The Things Network. In the second session we looked in more detail at getting an Arduino powered node connected to The Things Network and transmitting data packets.

In the third session, we are taking the data and looking at how it can be processed, in this case using the NodeRed tool.

Installation of Node-RED

Node-RED is a web server program which uses Node.js, which can be installed on a home computer, server, Raspberry Pi or in the Cloud. It can be used to collect and process the data collected by The Things Network[1].) We will be using MQTT to get access sensor data.

For supported Operating Systems and installation instructions, see: https://nodered.org/docs/getting-started/

On Linux Operating Systems (Ubuntu, Debian, Endebian etc.), Node-RED can be installed with

bash <(curl -sL https://raw.githubusercontent.com/node-red/linux-installers/master/deb/update-nodejs-and-nodered)

sudo systemctl enable nodered.service

It also appears to be installable on Windows 10 using the Windows Subsystem for Linux (WSL)[2].

Once installed, ot can be accessed via web browser connecting to port 1880. If it is installed on the local computer, ot can be accessed by visiting the URL – http://localhost:1880/

A User Guide with Tutorials are available from the Node-RED website[3]. If you have access to a working Node-RED server then the Tutorials are a good way to get started.

One final note. Flows are stored on the computer under the computers name (hostname). It that is ever changed, then the flows will not be loaded and it will look like your configuration has dissapeared. It is still there, and will reappear if the hostname is changed back. The easiest way to fix this is to export your flows to a file first (json) and then import them afterwards. This option is under the main menu.

Notes:
[1] Node-RED is a general purpose automation framework and can do a lot more than mentioned here,
[2] https://schellingerhout.github.io/bash%20on%20ubuntu%20on%20windows/nodered-windows/
[2] https://nodered.org/docs/

Working with Node-RED

Once Node-RED has been installed it can be accessed via port 1880.

Some additional packages are required. These are installed via the Manage Palette menu.

node-red-contrib-persist
node-red-dashboard

Configuration is done by connecting processing nodes with virtual wires to create flows. This allows messages to be processed as they pass through the system.

A Node-RED flow configured to process IoT data.

Data collected by The Things Network is pulled into Node-RED by using the MQTT node. This requires the name of The Things Network application and an application key/password.

Add an MQTT node and add a new mqtt-broker (server)

  • Name: Something memorable eg. Meshed
  • Connection
    • Server: thethings.meshed.com.au
    • Port: 1883
    • SSL/TLS: (Unchecked)
    • Use Legacy MQTT 3.1 support: (Checked)
  • Security
    • Username: enfieldlibrary_iot_trial
    • Password: (application-key copied from TTN website, starts with ‘ttn-account-v2’)

The topic is the MQTT channel to subscribe to where ‘+’ is a wildcard and will match any device name.

enfieldlibrary_iot_trial/devices/+/up

There are a range of channels defined in the The Things Network API[1] covering different types of messages. Eg. up, down.

Node Decription

MQTT client

This node connects to The Things Network and subscribes to the defined topic. Any data sent by our LoRaWAN devices will be sent out this node.

JSON

This node converts a text payload to JSON format. This is useful for allow later nodes to access individual parts of the payload.

Switch

Diverts messages between different  processing flows. In the above flow, it is being used to send device data to different dashboard labels. This allows us to display the latest data coming from each sensor.


Function

Code can be written in JavaScript to manipulate the messages passed to the node.

In our example we take the data transmitted by our device and received by The Things Network (eg. Sensor: 0x0267)[2], extract the data (substring) and convert to a meaningful number (moisture). This is passed directly to the next node in the payload of the message (the ‘msg’ variable).

var payload = msg.payload.payload_fields.receivedString
payload = payload.substring(10)
var integer = parseInt(payload,16)
var moisture = integer/1024.0 * 100.0
msg.payload = moisture

// To allow plotting of multiple series
msg.label = "sensor-1"

return msg;

Dashboard Label

Creates label elements to display on the dashboard. These are updated by messages.

Dashboard Graph

Displays a graph from data. By default, the data is not permanantly stored and will not persist between flow deployments. The graph data can be stored persistantly by including save and restore nodes.

Persist (Save Data / Restore Data)

These nodes are used to ensure that the graphed data is persistent between reboots. Under particular conditions the date is written to a non-volatile location, and is read back into the system when starting up.

Multiple sets of data can be saved, but the save and restore nodes need to match.

Notes:
[1] https://www.thethingsnetwork.org/docs/applications/mqtt/api.html
[2] For the workshop there was a deliberate choise made to use ASCII strings to encode the data. This is inefficient, but allows the data to be easily viewed for training and debugging purposes. Have a look at Cayenne Low Power Payload (Cayenne LPP) for one encoding method to improve this.

Summary

The nodes above can be used in Node-RED to receive, display, graph and store data from LoRaWAN enabled sensors. It is a general purpose system and as such is missing some of the whistles and bells of some other systems.

Improvements can be made by hooking in some additional software packages. In particular, data can be stored in a database specifically designed to hold time series of sensor data (eg. InfluxDB[1]) and a more functional dashboard can be added for better graphing and analysis (eg. Grafana[2]).

Notes:
[1] https://www.influxdata.com/
[2] https://grafana.com/

IoT Workshop (Part 2)

In the previous post we discussed the LoRaWAN network and gave an overview of how it works end to end. This post looks in more detail and describes what’s necessary to get data into the network from individual sensors.

The LoRa Radio and LoRaWAN

The LoRa radio module transmits in the Industrial , Scientific and Medical (ISM) unlicensed radio band[1]. In Australia (and New Zealand) LoRa transmits between 915-928 MHz[2]. For uploading data, the band is divided into 64 narrow bandwidth channels of 125 kHz, with an additional 8 overlapping broader channels of 500 kHz.

The data is encoded using a chirp signal. The rate that this frequency changes is defined by a ‘spreading factor’ (SF), and a doubling of the spreading factor approximately doubles the amount of time it take to transmit data[3].

LoRaWAN Channel Allocation for Australia and New Zealand

Notes:
[1] ISM Band – https://en.wikipedia.org/wiki/ISM_band
[2] https://lora-alliance.org/sites/default/files/2018-05/lorawan_regional_parameters_v1.0.2_final_1944_1.pdf
[3] https://medium.com/home-wireless/testing-lora-radios-with-the-limesdr-mini-part-2-37fa481217ff

Hardware

Dragino Arduino Shield

To use LoRa and LoRaWAN we are using the following LoRaWAN Arduino Shield available from the local suppliers. It is supplied to transmit and receive in the 915 MHz. As purchased the jumpers were not set, and needed to be installed as indicated in the picture below.

When developing hardware, it is only possible to use digital input/outputs 3,4 and 5 (3 lines). Digital IO 0 and 1 are used by serial IO and interfere with programmng. The digital lines D11-13 are used to control the LoRa radio.

The available Arduino pins for projects are

  • D3-5 (3 digital lines) – Can be sed for Digital IO
  • D0-1 (2 digital lines) – Can be used for Digital IO but interferes with in circuit programming. Needs to be disconnected first.
  • A0-5 – Canbe used for Analog Input or Digital IO
  • SCL,SDA – Can be used for Digital IO and Analog In on Arduni Uno.
  • AREF – Used as analog reference voltage
  • GND – Ground

Software

Arduino Library for LoRaWAN (LMIC)

The Arduino LMIC library that we are using is being maintained by Thomas Laurenson and can be found on Github[1]. The original LMIC Library was written by IBM and has been ported for use with the Arduino board.

Once downloaded and installed in the Ardiuino IDE library directory[2], the software needs to be configured to use the Australian frequency plan which is done by editing the file ‘project_config/lmic_project_config.h’. For some reason the Australian configuration is called CFG_au921.

// project-specific definitions
//#define CFG_eu868 1
//#define CFG_us915 1
define CFG_au921 1
//#define CFG_as923 1
// #define LMIC_COUNTRY_CODE LMIC_COUNTRY_CODE_JP       /* for s923-JP */
//#define CFG_in866 1
define CFG_sx1276_radio 1
//#define LMIC_USE_INTERRUPTS

Notes:
[1] Github: https://github.com/thomaslaurenson/arduino-lmic
[2] When unzipping the software, there may be duplicated recursive top level directory names (eg. arduino-lmic/arduino-lmic). Ensure that only one level is copied into Arduino library folder.

Example Arduino Sketch

An example Arduino sketch also available from the GitHub repository, which can be used to connect to the LoRaWAN network. See arduino-lmic/examples/ttn-otaa-dragino-lorashield-au915[1].

When connecting to the LoRaWAN network there are two authentication methods “Authtication by Personalisation” (ABP) and “Over the Air Authentication” (OTAA). While it is a little more complicated, OTAA is preferred as it allows the network more control over the authentication process, but requires two-way communication with the end-node. ABP does not require the end-node to receive data from the network but the network needs to know when the end-node is ever reset[2].

For OTAA, there are three pieces of data that need to be copied into the sketch, and are obtained from The Things Network console[3].

  • Device EUI (DEVEUI) – This is the unique identifier for this device on the network. This can be changed. This needs to be used in the sketch in Least Significant Byte (lsb) order.
  • Application EUI – This is the application identifier, and is unique across the LoRAWAN network. This needs to be used in the sketch in Least Significant Byte (lsb) order.
  • Application Key – This is an initial encryption key shared between the Node and Application. It can be changed. This needs to be used in sketch in Most Significant Byte (msb) order.

The webpage tries to make this as easy as possible by allowing you to adjust the formatting and the copy to clipboard button.

Device confiuration parameters in The Things Network

These details are copied into the Arduino sketch in the FILLMEIN locations as shown below.

// This EUI must be in little-endian format, so least-significant-byte 
// first. When copying an EUI from ttnctl output, this means to reverse
// the bytes. For TTN issued EUIs the last bytes should be 0xD5, 0xB3,
// 0x70.
static const u1_t PROGMEM APPEUI[8]= { FILLMEIN };
void os_getArtEui (u1_t* buf) { memcpy_P(buf, APPEUI, 8);}

// This should also be in little endian format, see above.
static const u1_t PROGMEM DEVEUI[8]= { FILLMEIN };
void os_getDevEui (u1_t* buf) { memcpy_P(buf, DEVEUI, 8);

// This key should be in big endian format (or, since it is not really a
// number but a block of memory, endianness does not really apply). In
// practice, a key taken from the TTN console can be copied as-is.
static const u1_t PROGMEM APPKEY[16] = { FILLMEIN };
void os_getDevKey (u1_t* buf) { memcpy_P(buf, APPKEY, 16);}

In the sketch, the following line can also be changed.

static uint8_t mydata[] = "OTAA" 

This is the data that is going to transmitted over the network. This short string can be edited to something more meaningful. The sketch can then be compiled and installed, and if a LoRaWAN Gateway is within range, transmitted data will start to be collected by The Things Network Application.

Notes:
[1] Github: https://github.com/thomaslaurenson/arduino-lmic/tree/master/examples
[2] The packet counters need to be reset. This is used to stop data ‘replay’ attacks.
[3] Console->Applicaion->Devices

LoRaWAN Data

The transmitted packet data will appear under the Application Data on The Things Network.

Data packets being received from an IoT device via LoRaWAN

The information includes the data transmitted by the IoT device and metadata from the network about the transmission. Expanding the packet entry shows this detail.

In the case above, to assist with debugging a custom Payload Format decoder has been added which converts the bytes to a string with can be seen in the ‘receivedString’ field[1].

// Decode an uplink message from a buffer
// (array) of bytes to an object of fields.
function Decoder(bytes, port) {
// var decoded = {};
// if (port === 1) decoded.led = bytes[0];
// return decoded;

// Decode plain text; for testing only
return {
receivedString: String.fromCharCode.apply(null, bytes)
};
}

The raw transmitted data can be seen in the Payload.

The metadata fields show the frequency used (917.8 MHz), the datarate (SF7BW125 – Spreading Factor 7, Bandwidth 125 kHz) and the gateways which heard the transmission and their details.

Notes:
[1] https://core-electronics.com.au/tutorials/encoding-and-decoding-payloads-on-the-things-network.html

Next Time (Part 3)

The next step will be to look at how to make use of the collected data. NodeRed is a software package will be used to process and manipulate the data as it is received.

IoT Workshop

On Thursday (24 Oct 2019) we started running the first session of a new IoT workshop, being held at the local libray (Enfield Branch of the Port Adelaide Enfield Council). This workshop aims to be an introduction to creating sensors and nodes for the LoRaWAN IoT network, using The Things Network website.

This first sesssion (of four) was a whirlwind dump of details which we will unpack and work on through the remainder of the sessions, with hands on electroncs and programming.

We went though the architecture of the system and got everyone to create an account on The Things Network, and set them up so that they could see the IoT data that was being transmitted over the network.

Architecture of LoRAWAN Network (from The Things Network)

Technolgy discussed..

A quick summary of the various bits-n-pieces that are being used follows.

LoRa

The radio technology that is used by sensors for ‘Long Range’, low bandwidth communication. These radios may be programmed to be used in either point-to-point or point-to-gateway (LoRaWAN) mode, configurable with software. They use the unlicensed ISM radio bands, 911MHz in Australia.

https://en.wikipedia.org/wiki/LoRa

LoRaWAN

The radio network and infrastructure used to collect sensor data, via publically accessible gateways.

Arduino

A microprocessor board and programming tools, which use the Atmel (and other) microprocessor. The original aim was to create a programming system which was easy to use for hobbyists and members of the maker community. Arduino microprocessor boards can be used with a LoRa radio to create sensors which can transmit data via LoRAWAN and the internet to be collected and used by other applications. https://www.arduino.cc/

Arduino Shield

This is additional hardware that has been designed to plug directly into an Arduino board and allow it to be easy to use.

Arduino IDE

The Interactive Development Environment (IDE) used to program an Arduino microprocessor board. It is available as a downloadable Java program, and as a web based application. The software is available from the Arduino website.

Arduino Libraries

Additional software packages which can be installed to extend Arduino programs. For example, the LMIC library is used to allow our Arduino programs to use the LoRaWAN radio shield.  

Libraries can be downloaded from within the Arduino IDE, or downloaded and installed manually.

LoRaWAN Sensor Node

A device which collects data and transmits to the LoRaWAN network. It uses a LoRa radio. They need to be registered and authenticated with the LoRaWAN network to transmit correctly.

LoRaWAN Gateway

A device which listens for LoRaWAN radio transmissions from nodes in its area and forwards them to the servers on the LoRaWAN network. They need to be connected to the internet and be registered to operate.

The Things Network

A free website (registration required) which allows access to data collected by LoRaWAN sensors. It is used to register LoRaWAN gateways on the network, and create LoRaWAN applications.

https://www.thethingsnetwork.org/

MQTT

MQ Telemetry Transport. This is the communication software and protocol  used to allow sensor data to be distributed and used once it has been collected by The Things Network.

NodeRed

A web server program, which can be installed on a home computer or server (or Raspberry Pi) which can collect and use sensor data collected. It uses MQTT to access sensor data.

NodeRed is an automation environment and is useful for a lot more than collecting and displaying our IoT data. See the website for more details.

https://nodered.org/

TTNmapper

Mobile application which can be used to map LoRaWAN coverage.  https://ttnmapper.org/

Sensor/Node Example

The following are some example of the sensors that will be built in the workshop.

Moisture Sensor

A soil moisture sensor which sends measurements via LoRaWAN to a monitoring application created NodeRed

The data is displayed in a web browser as the following.

Next Post: IoT Workshop (Part 2)

Adding a Dev case out of Lego

While the kids had the Lego out, I thought that I would pinch some pieces and build a case for my LoRaWAN node. Nothing fancy, random colours but enought to protect the boards from random knocks and bumps.

Blinking LEDs, improved with Lego
Blinking LEDs, improved with Lego

Technical Specifications

  • Pieces: 55
  • Colour: Various
  • Dimensions: 9 studs x 12 studs x 5.2 layers

LoRaWAN Node (Update)

LoRa Node consisting of an Arduino Uno (bottom), Duinotech LoRa Radio shield (middle) and Freetronics prototyping board (top)

The following is a quick update on the construction of LoRaWAN node., which was perviously described here.

After the previous work and the messing around with the uneven Arduino pins, I ordered some Freetronics prototyping boards and continued with those.

With the first prototype, I had attached LEDs to the first two digital pins (D0,D1) which are also used by the USB Serial connection. These made them kind of useless for monitoring any status, but also interfeared with the Arduino sketch programming. Before a new program could be successfully loaded in this configuration, the board first needed to be removed. This got annoying very quickly, particularly when I enjoy having a fast development process.

In the next design I removed these LED’s and connected them up to the supposibly free and available digital pins (D11,D12 and D13). It appears as though the LoRa radio board also uses at least one of these pins out of the box, with configuration jumpers that use the others. If the configuration jumpers are changed, the software would also need to be changed. The board works as expected out of the box with the available Arduino libraries, so it would be a pity to mess this up. Also, the documentation on the boards jumpers, and their orientation isn’t the clearest, so it would be best just to leave this well alone.

The remnants of this work can be seen on the bottom three resistors seen in the picture above. These resistors were connected with hookup wire underneath to pin D11, D12 and D13.

The Arduinio analog input pins, which were still unuised, can be switch to be used as digital outputs. The three LEDs that were connected earlier to the digital pins, were now connected to Analog pins A0, A1 and A2, with addional LEDs (with resistors, more about that sortly) to A3, A4, and A5. They are connected by more hookup wire underneath the prototype board.

Spot the difference.

A sketch was written to test the connected LED’s and was uploaded, without having to remove the prototype board first. This showed that everything was working as expected, except for one thing. The three new LEDs where significantly dimmer than the first ones.

The three new resistors I had grabbed quickly from my resistor box were 1K (Brown-Black-Black-Brown/Brown), rather than 100 Ohm (Brown-Black-Black-Black/Brown). This ment that the current through the LEDs was less, and hence their brightness was less as well. This has not been fixed in this prototype and probably wont be, although I could solder another suitably size resistor in parallel to the 1K ones to made the combined resistance close to the desired 100 Ohm amount.

The uploaded sketch was still contained the ability to set a couple of LEDs based on a byte of data received from the LoRaWAN radio. This was shown to work from The Things Network web, with the next step being able to drive this from custom software. Data from the LoRaWAN node is accesses via the MQTT protocol, and data can be uplinked to the node using the same method.

I have created several utility scripts to test all of this, which can be found on GitHub at: https://github.com/PaulSchulz/ttn-utils

LoRaWAN Node and Gateway

LoRaWAN Node and Gateway

With assistance of the STEM Education resources from the Port Adelaide Enfield Libraries, I have been investigating LoRaWAN devices, how to build them and how they can be used.

There is more investigation to be done, but the node shown in the picture above is able to do the following:

  • Connect and Authenticate to a LoRaWAN network (The Things Network) via Over The Air Authentication (OTAA); and
  • Send data (Uplink) and Receive data (Downlink) via the LoRaWAN Gateway(s). (Downlink data is queued, and is received in a window after Uplink data has been received by the network. This reduces power usage in the node.)

The Node used was the Gragino Arduino Shield (or clone from duinotech)

The Gateway is the uGateway LoRaWAN from Core Electronics. Having a local gateway is not necessary if there is another one in the neighbourhood, within range of your node. There are currently no nodes in my area so an internal gateway was required for development and testing. The other option was to take the Node for a drive in the car after a development session, but this made debugging very tiresom, very quickly.

There are a couple of other useful tools worth mentioning.

TTN Mapper is an Android application that will monitor your device on your smart phone, via The Things Network. It then used your phones GPS to log the position of your device. The result is that you can map the coverage of nearby gateways by using your LoRaWAN node and phone in a similar way to how Wireless Access points used to be mapped (WarDriving). This is useful for knowing where your nodes have TTN access, and where additional gateways might be required.