Integrate IoT Device with AWS IoT using Python — Part II: command-and-response
This is Part II of a series discussing one way to integrate IoT device with AWS IoT using Python. Other parts of the series are listed below:
- Part I: upload data to MQTT topic
- Part II: command-and-response (this article)
- Part III: command-and-response using API
- Part IV: remote SSH login with ngrok
- Part V: over-the-air software update
For background information of this series, please refer to the Introduction section of Part I.
Before You Start
It is highly recommended that you go through Part I to set up your development environment and configure AWS. This article assumes that you have already downloaded the source code from the GitHub repo and set up AWS IoT for
vehicle_detector_1_UPLOAD. We will introduce new AWS services such as AWS IoT Jobs and AWS S3. Although no prior knowledge of these services is needed to get through the article, if you are completely new to them, some sections might be a bit much to swallow. If in doubt, the AWS documentation is always your best friend.
Since you already have the GitHub repo, you just need to sync it with the
master branch and switch the tag to
git pull origin master
git checkout -b command_and_response
The directory tree looks like this:
│ └── readme.md
│ ├── __init__.py
│ ├── script_config.py
│ └── update_lambda_code.py
│ ├── __init__.py
│ └── aws_iot_client_wrapper.py
│ ├── __init__.py
│ └── child_processes.py
│ ├── __init__.py
│ ├── remote.py
│ └── upload.py
│ ├── __init__.py
│ └── network_connection_error.py
│ ├── __init__.py
│ └── remote_control_api
│ ├── __init__.py
│ └── lambda_function.py
│ ├── __init__.py
│ ├── logger_config.py
│ └── ouput.py
│ ├── __init__.py
│ ├── command.py
│ └── process.py
If you compare the directory tree with that in Part I, you will notice a few additions, including a
remote.py file in the
clients module and a
remote_control module. These are the additional code needed for the IoT device to enable remote command-and-response.
There is also a
lambda_func module and a
scripts folder. They will be used in Part III of the series, where we will focus on how to perform command-and-response via a simple API call.
What does “command-and-response” mean?
Command-and-response in our context means that a user sends a remote command to an IoT device, which is physically unreachable but connected to the Internet, and gets a response from it after the command has been executed. This feature is very useful to dynamically check the current state of the IoT device (e.g. report current battery level, CPU temperature, data collection status, etc.), request a SSH login address (we will discuss this in Part IV), and perform over-the-air (OTA) software update. It is a must-have to ensure full control of an IoT device after it is deployed.
Workflow of AWS IoT Jobs for command-and-response
The workflow of using AWS IoT Jobs (documentation) for command-and-response is shown below:
- User prepares a job document that contains all the information needed for the IoT device to execute the command. This document is a json file and should be stored in a private AWS S3 bucket.
- User creates a job, where the job id, target AWS IoT thing, source of the job document, etc. must be specified. This job is now in the QUEUED status.
- If the IoT device is not currently engaged in executing a job, it will receive a notification that a new job is available.
- The IoT device expresses intent to start the new job. This will change the job’s status from QUEUED to IN_PROGRESS.
- Once the job’s status is successfully changed to IN_PROGRESS, AWS IoT sends another notification to the device regarding the details of the job.
- Upon receiving the job whose status is now IN_PROGRESS, the IoT device reads the job document and performs the specified command.
- Once the command is executed, the IoT device produces two pieces of items: job execution status and additional information (e.g. any output from the job execution or error message). The job execution status can be SUCCEEDED or FAILED.
- The IoT device then updates the job’s status with the job execution status. In addition, it also uploads the additional information. This will change the job status from IN_PROGRESS to either SUCCEEDED or FAILED.
- Once the job’s status is successfully changed to either SUCCEEDED or FAILED, AWS IoT sends a confirmation message. This concludes the full operation on the IoT device’s side regarding command-and-response.
In the following, I will show step by step how this workflow is implemented.
Create A Job Document And Save It on S3
A job document is a simple json file. It does not have any format requirement, as long as a presigned S3 URL is not included. A presigned S3 URL is useful if the IoT device needs to download other files from the S3 bucket, e.g. when it is performing an OTA software update. However, for simplicity purpose, our job document will not include a presigned S3 URL.
The job document can be as simple as this:
This document contains one key-value pair, specifying that the command is called
"greeting". We will later define what this
"greeting" command actually signify for the IoT device. Save this file locally as
greeting.json. Note that it is crucial that the file name is exactly the same as the command itself.
Then let’s create an S3 bucket. If you already know how to do it, go ahead and create a private bucket and upload the
greeting.json file in it.
If you haven’t used S3 before, go to AWS console → S3 → Create bucket. Under Bucket name, input any name that is allowed by AWS. Click Next. Tick Versioning. Click Next. Keep Block all public access ticked. Click Next. Click Create bucket.
Click the name of the bucket to view its content, which is nothing for the moment. Click Upload and upload
greeting.json to the bucket.
If you want to reduce manual mouse clicking, I recommend you invest some time in AWS CLI (documentation). It can vastly speed up the creation and modification of most AWS resources, especially when such action is repetitive and requires a lot of configuration (i.e. a lot of mouse clicking on the console).
Set up AWS IoT for The Remote Client
The procedure to set up AWS IoT for the
Remote client is generally the same as the one for the
Upload client, so please refer to Part I for detailed explanation. Briefly, we first create a thing type and a thing group with the same name
VehicleDetectorRemote. Unlike the
Upload case, there is no policy attached to the
VehicleDetectorRemote thing group.
Then we create a policy called
vehicle_detector_remote_individual_policy. It is specific to each individual
Remote client . Its content is as follows:
Make sure to change the
[aws_account] to fit your AWS account profile. This policy specifies which MQTT topics the
Remote client subscribes and publishes to. These topics are managed by AWS IoT. They allow the
Remote client to receive new job notification and update job status.
Finally, we create an AWS IoT thing called
vehicle_detector_1_REMOTE, generate its certificates and key pairs, and download the credential files to the
credentials/ folder in the repo. When generating the certificates, we need to attach two policies,
vehicle_detector_connect_policy and the newly created
vehicle_detector_connect_policy was created in Part I, which allows the device to connect to AWS IoT.
Set up The Remote Client on The Source Code
The code for the
Remote client resides in two places:
This module defines a
Remote class similar to the
Upload. The class inherits from the
AWSIoTMQTTClientWrapper with the
client_type set to
REMOTE. This allows the IoT device to handle connection to AWS IoT. It then establishes an
AWSIoTMQTTThingJobsClient, which is responsible for subscribing to the necessary AWS IoT Job-related topics to get notification when a new job is coming (see
_subscribe() method for details), and publishing updates to the job’s status. The
Remote class also defines callback functions for each of the subscription. They are fired when a message is received. For instance, the
_new_job_received() callback is fired when the IoT device is notified of the arrival of a new, currently QUEUED job. It immediately expresses intents to accept the job by calling the
sendJobsStartNext() method. This will turn the job status from QUEUED to IN_PROGRESS. Once this change is successful, AWS IoT sends a message to the
JOB_START_NEXT_TOPIC, which will cause the callback function
_start_next_in_progress() to fire. This function executes the command specified by the job document using the
execute_command() function, and updates the job status. The rest of the callback functions are all for logging purposes.
This module specifies what each command does and how it is executed by the IoT device.
COMMAND_DICTS records the actions for each command. As you can see, the command
"greeting" is defined to return a string
"Hello from [sensor_name]". For our current device, this means once it receives the
"greeting" command, it executes the function associated with the
"greeting" key in the
COMMAND_DICTS, which produces the output
"Hello from vehicle_detector_1". The function
execute_command() also specifies how error is handled if the command execution fails, and what additional information (
status_details) is to be uploaded to the job.
This module defines a very simple function
remote_control() that instantiates the
Remote client, and then enters a forever loop. This is the entry point to the child process that runs the
Remote client exclusively. It is important NOT to run the
Remote client with the main process, because we want the
Remote client to remain functional in case the main process fails. Since the
remote_control() function follows the guidelines of the
ChildProcesses, we can easily start it up with
cp.create_and_start('remote_control', remote_control) in the
We need to add the credential file names for the
Remote client to the configuration. Your
.env file shall look like this:
# AWS IoT config
upload_topic=vehicle_detector/test/raw# Main program config
Notice that we have also changed
1000. This is to extend the running time of the IoT device, such that we have a long window to test command-and-response.
aws_iot_integration folder, run command
pipenv run python3 main.py (or
python3 main.py if the virtual environment is on and the
.env file has been loaded to the environment variables).
Check the content of the logging messages at the beginning of the run. You shall verify that there are messages saying
vehicle_detector_1_UPLOAD ONLINE. and
vehicle_detector_1_REMOTE ONLINE.. These tell you both clients are up and running. You shall also see five consecutive instances of
Performing sync subscribe… near
vehicle_detector_1_REMOTE ONLINE.. This tells you the
Remote client has subscribed to all the necessary topics for AWS IoT Jobs. You can set
debug=True in the
.env file and test run again to see what specific MQTT topics the
Remote client subscribes to.
If any error pops up, the most likely culprit is incorrect set up of the
.env file, not loading the
.env file to the environment variables, or incorrect set up of the
Command-and-Response Live Demo
Now that both the AWS IoT and the IoT device source code are ready, we can run a live demo on command-and-response.
First, keep the IoT device running the same way as the Test run.
Then go to AWS IoT Core → Manage → Jobs → Create → Create custom job. Under Job ID, input
test1. Under Select devices to update, choose
vehicle_detector_1_REMOTE. Under Add a job file, choose the S3 bucket you have created earlier and
greeting.json. Keep everything else as default, and click Next. In the following page, keep everything as default, but before clicking Create, make sure you can see the IoT device’s logging messages in a separate window. While keeping an eye on the logging messages, click Create. In about one to two seconds, you shall see these logging messages show up:
2020-08-05 19:55:24,456 - src.clients.remote - INFO - Recieved command "greeting", executing now...
2020-08-05 19:55:24,456 - src.clients.remote - INFO - Execution result: Hello from vehicle_detector_1
2020-08-05 19:55:24,457 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Performing sync publish...
2020-08-05 19:55:24,543 - src.clients.remote - INFO - Job status (4, 'SUCCEEDED') update SUCCESS for cmd "greeting"
This indicates that the IoT device has received the job, identifies that the command to execute is
"greeting", executes the command, updates the job status, and uploads additional information to the job.
Now head to the Jobs page of AWS console and refresh it. You shall see a card called
test1 there. Click the card and verify that the job’s status is set to
SUCCEEDED. You can also view the job document under Details → View stored job file. However, you cannot see the additional information uploaded by the IoT device. To view that and truly unlock the power of command-and-response, we need to step up the game and use an API call to create a job and retrieve all of its info. That will be the focus of Part III.
In this article, we have shown a working example of using AWS IoT Jobs to enable command-and-response with an IoT device. We have also performed a command-and-response live demo by creating a job on the AWS IoT Console and observing the device receiving and executing the job in real time. However, we realize that using the AWS IoT Console is not the best way to do command-and-response, because the response from the IoT device is not available.
In Part III, we will remedy this problem by creating a job and retrieving the full info with an API call. Using a simple API call to perform command-and-response unlocks all kinds of possibilities to integrate with the IoT device, such as a web app, a mobile app, or a CI/CD pipeline.