Integrate IoT Device with AWS IoT using Python — Part I: upload data to MQTT topic

Workflow of uploading data from IoT device to AWS IoT MQTT topic

Introduction

This is part I of a series discussing one way to integrate IoT device with AWS IoT using Python. Other parts of the series are listed below:

  • Part II: command-and-response
  • Part III: command-and-response using API
  • Part IV: remote SSH login with ngrok
  • Part V: over-the-air software update

Before You Start

It is assumed that you are familiar with Python 3 and know how to code and run a Python program on an IoT device. It is also assumed that you have an AWS account (free tier is sufficient), are familiar with the AWS console, and know the basics of AWS IoT. It is highly recommended that you download the GitHub repo for this article and parse the code along the way. Although the code was developed on a Raspberry Pi 4, it shall run on any Unix system.

git clone https://github.com/FanchenBao/aws_iot_integration.git
cd aws_iot_integration
git checkout -b upload_data
.
├── Pipfile
├── Pipfile.lock
├── config.py
├── credentials
│ └── readme.md
├── main.py
├── pyproject.toml
├── setup.cfg
└── src
├── __init__.py
├── aws
│ ├── __init__.py
│ └── aws_iot_client_wrapper.py
├── child_processes
│ ├── __init__.py
│ └── child_processes.py
├── clients
│ ├── __init__.py
│ └── upload.py
├── errors
│ ├── __init__.py
│ └── network_connection_error.py
├── logger
│ ├── __init__.py
│ ├── logger_config.py
│ └── ouput.py
└── vehicle_detector
├── __init__.py
└── detect_vehicle.py
pipenv install

Set up AWS IoT

Before diving into the source code, we need to set up the AWS IoT first. The tutorials provided by AWS IoT documentation, such as this one, is a good place to familiarize yourself with the workflow of AWS IoT.

Create a thing type

A thing type can be attached to an IoT thing and provides more searchable attributes and tags for organizational purpose. To create a thing type, go to AWS IoT Core → Manage → Types → Create. Under Name, input VehicleDetectorUpload . Under Description, input “The upload type for vehicle detector. Vehicle detector in this type handles uploading the live data to the MQTT topic.” Then Click Create thing type.

Create a thing group

A thing group is similar to a thing type in regards to its ability to organize IoT things. However, it is more powerful because it has its own ARN and can attach policies, which will be forced onto the IoT things contained within the group. These features allow us to use thing group to apply global policies to all IoT things that share the same characteristics, without having to repeatedly attach the same policy over and over again.

Create policies

Generally speaking, the policies are usually set as the most permissive for tutorial purpose. However, let’s try to make them as specific as possible. We need to create two policies. One for each individual IoT thing to allow the actual IoT device to connect to the IoT thing on AWS; the other for the VehicleDetectorUpload group we have just created.

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iot:Connect"
],
"Resource": [
"arn:aws:iot:[region]:[aws_account]:client/${iot:Connection.Thing.ThingName}"
]
}
]
}
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"iot:Publish"
],
"Resource": [
"arn:aws:iot:[region]:[aws_account]:topic/vehicle_detector/test/raw",
"arn:aws:iot:[region]:[aws_account]:topic/vehicle_detector/dev/raw",
"arn:aws:iot:[region]:[aws_account]:topic/vehicle_detector/prod/raw"
]
}
]
}

Create IoT thing

We have finally reached the stage of creating the AWS IoT thing. It is an entity to which the actual IoT device connects in order to transfer data. To create an IoT thing, go to AWS IoT Core → Manage → Things→ Create → Create a single thing. Under Name, input vehicle_detector_1_UPLOAD . Under Apply a type to this thing, choose VehicleDetectorUpload from the drop down. Under Add this thing to a group, click Change and choose VehicleDetectorUpload from the drop down. Under Set searchable thing attributes, input id for Attribute key and 1 for Value. Click Next.

Attach group policy

The last step is to attach the vehicle_detector_upload_group_policy to the VehicleDetectorUpload thing group. Go to AWS IoT Core → Thing groups → VehicleDetectorUpload → Security → Edit. Under Select a policy to attach to this group, click Select and choose vehicle_detector_upload_group_policy . Click Save.

Summary

In this section, we have set up the AWS IoT by creating an IoT thing named vehicle_detector_1_UPLOAD , tagged it with the VehicleDetectorUpload thing type, added it to the VehicleDetectorUpload thing group, created and downloaded the associated credentials, and attached the vehicle_detector_connect_policy policy to the credentials. In addition, we have created the vehicle_detector_upload_group_policy policy and attached it to the VehicleDetectorUpload thing group.

Set up Credentials

The four credential files downloaded earlier (a certificate, a private key, a public key, and an Amazon Root CA file) must be transferred to the Python application folder under credentials/ . It is crucial that none of these files (except for the Amazon Root CA file) shall ever be exposed to the public. They must only reside in the IoT device. If they are exposed, discard them and create new ones from AWS IoT console.

Set up Python Application Configuration

Source code: config.py

# AWS IoT config
sensor_name=vehicle_detector_1
endpoint=[iot_thing_https_endpoint]
port=8883
root_ca=AmazonRootCA1.pem
upload_private_key=[credential_id]-private.pem.key
upload_cert_file=[credential_id]-certificate.pem.crt
remote_private_key=
remote_cert_file=
upload_topic=vehicle_detector/test/raw
# Main program config
total_iterations=5
debug=False

Set up AWS IoT Upload Client

The design for the Upload client is to first create a generic AWS IoT client wrapper class, and then inherit the wrapper class in the Upload client. The reason for this design is that in part II of this series, we will create another client called Remote . It uses almost the same set up as Upload. Thus it is easier for code maintenance to set up a generic wrapper class, from which both the Upload and Remote class can inherit.

Set up the generic AWS IoT client wrapper

Source code: src/aws/aws_iot_client_wrapper.py

Set up the Upload client

Source code: src/clients/upload.py

Set up The Mock Vehicle Detection Function

Source code: src/vehicle_detector/detect_vehicle.py

Set up Logging

Source code: src/logger/output.py and src/logger/logger_config.py

Set up A Class to Handle All Child Processes

Source code: src/child_processes/child_processes.py

Set up The Main Entry Point

Source code: main.py

  1. Initialize data_q and logger_q.
  2. Use child_processes to create and start the output_logger process. Also set up the queue_logger.
  3. Use child_processes to create and start the detect_vehicle process.
  4. Run a session of detect_vehicle process, which includes uploading five instances of data to AWS IoT.
  5. Use child_processes to terminate the detect_vehicle and the output_logger process when the session ends.

Run The Program And Observe The Logs

In your, run command pipenv run python3 main.py. You can also choose to use python3 main.py directly, but you must make sure that you are in the virtual environment and that the environment variables have been updated with the .env file in your current shell.

$ pipenv run python3 main.py
Loading .env environment variables…
2020-08-03 11:44:36,921 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - MqttCore initialized
2020-08-03 11:44:36,922 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Client id: vehicle_detector_1_UPLOAD
2020-08-03 11:44:36,923 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Protocol version: MQTTv3.1.1
2020-08-03 11:44:36,923 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Authentication type: TLSv1.2 certificate based Mutual Auth.
2020-08-03 11:44:36,923 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Configuring offline requests queueing: max queue size: 0
2020-08-03 11:44:36,924 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Configuring offline requests queue draining interval: 0.100000 sec
2020-08-03 11:44:36,925 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Configuring endpoint...
2020-08-03 11:44:36,925 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Configuring certificates...
2020-08-03 11:44:36,921 - src.vehicle_detector.detect_vehicle - INFO - Vehicle detected! Current number of vehicles: 1
2020-08-03 11:44:36,927 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Configuring reconnect back off timing...
2020-08-03 11:44:36,927 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Base quiet time: 1.000000 sec
2020-08-03 11:44:36,927 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Max quiet time: 32.000000 sec
2020-08-03 11:44:36,927 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Stable connection time: 20.000000 sec
2020-08-03 11:44:36,928 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Configuring connect/disconnect time out: 10.000000 sec
2020-08-03 11:44:36,928 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Configuring MQTT operation time out: 5.000000 sec
2020-08-03 11:44:36,928 - root - INFO - Publishing data to topic vehicle_detector/test/raw...
2020-08-03 11:44:36,928 - src.aws.aws_iot_client_wrapper - INFO - Connecting vehicle_detector_1_UPLOAD to MQTT client...
2020-08-03 11:44:36,929 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Performing sync connect...
2020-08-03 11:44:36,929 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Performing async connect...
2020-08-03 11:44:36,929 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Keep-alive: 600.000000 sec
2020-08-03 11:44:37,285 - src.aws.aws_iot_client_wrapper - INFO - vehicle_detector_1_UPLOAD ONLINE.
2020-08-03 11:44:37,286 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Performing sync publish...
2020-08-03 11:44:39,928 - src.vehicle_detector.detect_vehicle - INFO - Vehicle detected! Current number of vehicles: 2
2020-08-03 11:44:39,929 - root - INFO - Publishing data to topic vehicle_detector/test/raw...
2020-08-03 11:44:39,930 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Performing sync publish...
2020-08-03 11:44:41,930 - src.vehicle_detector.detect_vehicle - INFO - Vehicle detected! Current number of vehicles: 3
2020-08-03 11:44:41,932 - root - INFO - Publishing data to topic vehicle_detector/test/raw...
2020-08-03 11:44:41,932 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Performing sync publish...
2020-08-03 11:44:45,935 - src.vehicle_detector.detect_vehicle - INFO - Vehicle detected! Current number of vehicles: 4
2020-08-03 11:44:45,936 - root - INFO - Publishing data to topic vehicle_detector/test/raw...
2020-08-03 11:44:45,937 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Performing sync publish...
2020-08-03 11:44:50,941 - src.vehicle_detector.detect_vehicle - INFO - Vehicle detected! Current number of vehicles: 5
2020-08-03 11:44:50,942 - root - INFO - Publishing data to topic vehicle_detector/test/raw...
2020-08-03 11:44:50,942 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Performing sync publish...
2020-08-03 11:44:51,013 - root - INFO - Program ended
2020-08-03 11:44:52,943 - src.vehicle_detector.detect_vehicle - INFO - Vehicle detection terminated.
2020-08-03 11:44:53,947 - root - INFO - Logging terminated.

Run The Program And Observe The Uploaded Data in Real Time

On AWS console, go to AWS IoT Core → Test. Under Subscription topic, input vehicle_detector/test/raw, and then click Subscribe to topic.

{
"timestamp": 1596470034182,
"cur_vehicle_count": 1
}

Summary

In this article, we have shown a non-trivial example of how to set up AWS IoT and a Python program to upload data from an IoT device to an AWS IoT MQTT topic.

Hi, I am from the Earth. And you?