Integrate IoT Device with AWS IoT using Python — Part III: command-and-response using API

15 min readAug 17, 2021

Workflow of command-rand-response with API

Introduction

After more than a year of hiatus, I am back on this series. This is Part III of the series discussing one way to integrate IoT device with AWS IoT using Python. Other parts of the series are listed below:

Part I: upload data to MQTT topic
Part II: command-and-response
Part III: command-and-response using API (this article)
Part IV: remote SSH login with OpenVPN
Part V: over-the-air software update

The goal of this article is to set up a simple REST API using AWS resources such that we can conduct command-and-response without relying on the AWS IoT console. One benefit of creating an API for this is that the API can be called from anywhere. Thus, it opens up the possibility of building other technology on top of the API to customize the process of conducting command-and-response (e.g. web app, mobile app, CLI, etc.). Another benefit is that we are able to obtain the response uploaded by the IoT device after it executes the command. Such response is not available from the AWS IoT console.

Before You Start

It is highly recommended that you go through Part I and II to set up your development environment and configure AWS. This article assumes that you have already downloaded the source code from the GitHub repo and got the live demo of command-and-resoon to work on vehicle_detector_1_REMOTE. We will introduce new AWS services such as AWS API Gateway and AWS Lambda. Although no prior knowledge of these services is needed to get through the article, if you are completely new to them, some sections might be a bit hard to follow. If in doubt, the AWS documentation is always your best friend.

Since you already have the repo locally, sync it with themaster branch, create a new branch, and sync with the tag command_and_response_API:

git pull origin master
git checkout -b new_branch_name command_and_response_API

The directory tree looks like this:

.
├── config.py
├── credentials
│   ├── [upload_credential_id]-certificate.pem.crt
│   ├── [upload_credential_id]-private.pem.key
│   ├── [upload_credential_id]-public.pem.key
│   ├── AmazonRootCA1.pem
│   ├── [remote_credential_id]-certificate.pem.crt
│   ├── [remote_credential_id]-private.pem.key
│   ├── [remote_credential_id]-public.pem.key
│   └── readme.md
├── main.py
├── Pipfile
├── Pipfile.lock
├── pyproject.toml
├── scripts
│   ├── __init__.py
│   ├── script_config.py
│   └── update_lambda_code.py
├── setup.cfg
└── src
    ├── aws
    │   ├── aws_iot_client_wrapper.py
    │   └── __init__.py
    ├── child_processes
    │   ├── child_processes.py
    │   └── __init__.py
    ├── clients
    │   ├── __init__.py
    │   ├── remote.py
    │   └── upload.py
    ├── errors
    │   ├── __init__.py
    │   └── network_connection_error.py
    ├── __init__.py
    ├── lambda_func
    │   ├── __init__.py
    │   └── remote_control_api
    │       ├── __init__.py
    │       └── lambda_function.py
    ├── logger
    │   ├── __init__.py
    │   ├── logger_config.py
    │   └── ouput.py
    ├── remote_control
    │   ├── command.py
    │   ├── __init__.py
    │   └── process.py
    └── vehicle_detector
        ├── detect_vehicle.py
        └── __init__.py

It is almost exactly the same as the one shown in Part II, except that you shall have credential files under credentials/ folder from going through Part I and II.

Workflow of command-and-response using API

We create an API using AWS API Gateway. API Gateway serves as the “front-end”. It accepts our API call, parses any query parameters (query parameters to an API is very similar to how arguments are to a function), and hands over the execution to a “back-end”. We use AWS Lambda as the“back-end”, which, upon receiving the hand-over from API Gateway, grabs all the necessary information, executes command-and-response programmatically, and returns the outcome to API Gateway. After receiving the response from Lambda, API Gateway forwards it back to where the API call is initiated and we will have access to the response.

In the following sections, we will go through how to set up Lambda and API Gateway for this workflow.

Set up IAM Role

For AWS Lambda to function properly, it must be granted an IAM role that allows it to perform the command-and-response tasks. An IAM role is akin to a container of policies (a policy defines what action is permitted). When an AWS service, such as Lambda, is granted an IAM role, it inherits all the permissions specified in the policies contained within the role. To perform command-and-response via AWS IoT job, we need to create two policies manage_iot_jobs_policy and s3_access_policy.

Create `manage_iot_jobs_policy`

Go to AWS console → IAM → Policies → Create Policy

Under the JSON tab, paste the following policy document (replace [aws_account] with your AWS account number, which is a 12-digit number). Click Next: Tags and then click Next: Review

Before moving on, a bit more explanation on the policy itself. The policy contains one statement.It has the effect of “Allow”, which means the statement allows all the actions specified in the “Action” field to be carried out on resources specified in the “Resource” field. In the “Action” field, we can see that the actions to be allowed are all related to manage IoT job, including creating, canceling, describing, and listing jobs. The resources listed in the “Resource” field uses wild card character * to describe ALL the IoT things, jobs and thing groups under the AWS account. We will see later how these actions and resources are related to Lambda’s handling of command-and-response.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "iot:CreateJob",
                "iot:CancelJob",
                "iot:DescribeJob",
                "iot:DescribeJobExecution",
                "iot:ListJobExecutionsForThing"
            ],
            "Resource": [
                "arn:aws:iot:*:[aws_account]:thing/*",
                "arn:aws:iot:*:[aws_account]:job/*",
                "arn:aws:iot:*:[aws_account]:thinggroup/*"
            ]
        }
    ]
}

Under Name, put manage_iot_jobs_policy. Click Create policy. The policy is now created.

Create s3_access_policy

Follow the steps as above to create s3_access_policy. The policy content is shown below, where bucket_name is the name of the bucket created in Part II. This policy allows downloading all the objects in the specified bucket. Our Lambda requires this policy because it needs to download the job document stored in the bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::[bucket_name]/*"
        }
    ]
}

Attach both policies to the lambda role

Next, we need to create remote_control_api_lambda_role and attachboth manage_iot_jobs_policy and s3_access_policy to it. Go to AWS console → IAM → Roles → Create role

Select AWS service → Lambda, and click Next: Permissions

Under Filter policies, choose Customer managed. Select both manage_iot_jobs_policy and s3_access_policy.

Under Filter policies, choose AWS managed, search for AWSLambdaBasicExecutionRole, and select it. Click Next: Tags. Then click Next: Reviews.

Under Role name, put remote_control_api_lambda_role. Click Create role. The role is now created.

Set up AWS Lambda

AWS Lambda is a serverless computation resource that can handle easy computation tasks (i.e. short execution time) at a low cost. Essentially, it is an AWS-managed platform to run almost any custom function coded in one of the allowed runtimes. We can certainly spin up a virtual machine in EC2 to handle the computation, but for our command-and-response purpose, which does not require complex and prolonged computation, Lambda is the most appropriate.

Go to AWS Console → Lambda → Create function

Input remote_control_api as Funciton name and choose Python 3.9 as Runtime. Expand Change default execution role and choose remote_control_api_lambda_role as the role assigned to the Lambda. Click Create function.

If Lambda creation is successful, you shall see something similar to this on the console.

Notice that under Code source, we have an empty function called lambda_handler. This is where we put the API backend code. When this Lambda is triggered by the API Gateway, lambda_handler is called to perform the actual command-and-response tasks.

Follow the steps below to set up lambda_handler function.

Copy the code in ./src/lambda_func/remote_control_api/lambda_function.py and paste it under Code source and click Deploy

Under Configuration → Environment variables, click Edit

Add two environment variables as shown below. Note that bucket_name refers to the bucket created in Part II (for me, it is aws-iot-integration-jobs; for you, it will be something different). Click Save.

IOT_PREFIX=arn:aws:iot:[region]:[aws_account]
S3_PREFIX=arn:aws:s3:::[bucket_name]

Explanation of The Lambda Code

In this section, we will give a high-level overview of the Lambda code, such that you have a general idea what the function is doing. For more details, refer to the doc-string and comments in the source code.

lambda_handler( )

As mentioned earlier, this function is the entrance of the Lambda code. It first extracts all the query string parameters and multi-value query string parameters and validate them via the validate() function. Recall that query string parameters to API are akin to arguments to function. We validate whether the query string parameters satisfy our requirements. If they do, we use them as arguments for subsequent function calls. If not, we terminate the function and return error message.

Once the validation passes, we check which action to handle. If the action is command, we first cancel all on-going jobs and then issue a new job associated with the given command. If the action is response, we wait for the response of a particular job as specified by the query string parameters. If the action is cancel, we cancel the target job.

validate()

This function validates all query string parameters and multi-value query string parameters. The rule of the query strings is specified in the doc-string. We will discuss the query strings later when setting up API Gateway.

cancel()

Cancel a target job. Note that we also include a busy wait loop to ensure that the function returns if and only if the target job has been fully canceled.

respond()

Obtain the response from an IoT thing after a command has been executed. This response is NOT available from the AWS IoT job console, but we can access it by using the describe_job_execution API. Note that we include a busy wait loop in case the job has not been completed by the IoT thing when we call the respond() function. There is also a timeout option (timeout is an optional query string parameter) to terminate the wait. The default timeout is three seconds.

command()

Create a job for all the IoT things specified in the multi-value query string parameter thingNames associated with the given command.

Test remote_control_api Lambda

Test command

In the Lambda console, under Code source, click the drop down of Test and click Configure test event. Under Event name, put testCommand. Then paste the following test case. The test case is equivalent to an API call that sends the command "greeting" to vehicle_detector_1_REMOTE. Click Create to create the test case (note that when creating jobs, we can issue the same command to a list of IoT things).

{
  "queryStringParameters": {
      "action": "cmd",
      "cmd": "greeting"
  },
  "multiValueQueryStringParameters": {
      "thingNames": [
          "vehicle_detector_1_REMOTE"
    ]
  }
}

Under the local repo aws_iot_integration folder, run command pipenv run python3 main.py (or python3 main.py if the virtual environment is on and the .envfile has been loaded to the environment variables). While watching the logs on the terminal, click the Test button on the Lambda console. You shall see these messages from the terminal

2021-08-16 18:39:17,258 - src.clients.remote - INFO - Recieved command "greeting", executing now...
2021-08-16 18:39:17,258 - src.clients.remote - INFO - Execution result: Hello from vehicle_detector_1
2021-08-16 18:39:17,259 - AWSIoTPythonSDK.core.protocol.mqtt_core - INFO - Performing sync publish...
2021-08-16 18:39:17,347 - src.clients.remote - INFO - Job status (4, 'SUCCEEDED') update SUCCESS for cmd "greeting"

and this output from the Lambda

If you can see both, that means you have successfully set up the Lambda for creating a job.

Test response

Create a new test case called testResponse as shown below (note that when querying for response, we can only do one IoT thing at a time). Regarding jobId, I am using the same one from Part II. You can use any jobId that has been completed by vehicle_detector_1_REMOTE. You can find all jobId in the AWS IoT job console.

{
  "queryStringParameters": {
      "action": "resp",
      "thingName": "vehicle_detector_1_REMOTE",
      "jobId": "test1"
  },
  "multiValueQueryStringParameters": []
}

Test the Lambda. If you see an output similar to the one shown below, you have successfully set up the Lambda for querying the response of a job.

Test cancel

A job can only be canceled if it is either in the QUEUED or IN_PROGRESS status. Thus, we have to test cancel when the IoT thing is not running. Run the testCommand test case without the IoT thing running, and examine that a new job is created and is in IN_PROGRESS status.

Create a new test case called testCancel as shown below. Note that the jobId must match the newly created IN_PROGRESS job.

{
  "queryStringParameters": {
      "action": "cancel",
      "jobId": "f1834612-fee4-11eb-b54a-919eefa6b4af"
  },
  "multiValueQueryStringParameters": []
}

Run testCancel test case. If you see output similar to the ones shown below, you have successfully set up the Lambda for cancelling a job.

Set up API Gateway

This is the final step; we are in the home stretch. Go to AWS console → API Gateway. Click Build under REST API

Configure the API settings as shown below. Under API name, input remote_contorl_api. Click Create API

Under Actions dropdown choose Create Method. In the method dropdown, choose GET, and then click the check mark. You shall be presented with this screen.

Check Use Lambda Proxy integration and choose remote_control_api as the Lambda function for the proxy. This means all the traffic going through this API will be handed over to the Lambda function for further processing. Click Save. And then click OK in the next pop up to add permission to the Lambda.

In the next screen, click Method request

Set Request Validator to Validate query string parameters and headers. Set API Key Required to true. Add action to URL Query String Parameters and check Required and Caching (we are asking API Gateway to check the existence of action query string parameter. All the other query string parameters are validated by the Lambda). Then click Method Execution to go back to the previous screen.

Enable CORS such that this API can be used from any source (for details about CORS, refer to the documentation). We can allow CORS because our API will be protected by API key. Click the Actions dropdown and click Enable CORS, then click Enable CORS and replace existing CORS headers. Then click Yes, replacing existing values

Once CORS is enabled, you can see an extra method OPTION below GET. The API is now ready for testing. Clicking TEST

Follow the same procedures as described previously to test the command, response, and cancel functionalities of the API, except that we will be triggering the Lambda via API Gateway instead of using test cases. To trigger test from API Gateway, input the following string to Query Strings one at a time and click Test. Remember to run pipenv run main.py when testing the command functionality. If all three tests pass, the integration between API Gateway and Lambda is successful.

Test command

action=cmd&cmd=greeting&thingNames=vehicle_detector_1_REMOTE

Test response

action=resp&thingName=vehicle_detector_1_REMOTE&jobId=test1

Test cancel

action=cancel&jobId=[some_job_id]

Now that our API is working, we need to deploy it and create an API key to protect it. To deploy the API, keep the GET method selected and click the Actions dropdown. Click Deploy API. In the pop-up window, choose [New Stage] for Deployment stage and input test for Stage name. CLick Deploy

In the next screen, you shall see Invoke URL on top of the screen. That URL is the one we will use to call the API.

But we cannot call it yet, because we haven’t set up API key yet. To set up API key, we must first create a usage plan and then associate add the API key to the usage plan. Click Usage Plans on the left panel and configure its settings as shown below. Click Next

Click Add API Stage, choose remote_control_api under API and test under Stage, and click the check mark. Then click Next

Click Create API Key and add to Usage Plan. In the pop-up window, input remote_control_api_key under Name. Click Save. Then click Done

We have thus deployed the API and configured an API key for it. To obtain the API key, click API Keys and remote_control_api_key. The key will show up after clicking Show.

Final Test

We will conduct the same three test as before, but this time, we will trigger the API from command line using curl

Test command

Run the following command in a separate terminal after running pipenv run main.py. Note that [your_api_key] should be the API key created above, and [your_api_endpoint] is the URL presented after deployment. Also recall that thingNames correspond to a multi-value query string parameter, which means we can put multiple thing names under thingNames. To do so on the query string, we can simply chain multiple thingNames together such as thingNames=iot_thing_1&thingNames=iot_thing_2&thingNames=iot_thing_3 .

curl -X GET -H 'x-api-key:[your_api_key]' -H 'Accept:application/json' "https://[your_api_endpoint]/test?action=cmd&cmd=greeting&thingNames=vehicle_detector_1_REMOTE"

You should get the following as response

{"jobId": "021d673d-fef4-11eb-9200-0d5b5ed6a46e", "message": "cmd='greeting' SUCCEEDED!"}

Test response

Run the following command in a terminal. Note that we use the same jobId as the one returned from the previous test.

curl -X GET -H 'x-api-key:[your_api_key]' -H 'Accept:application/json' "https://[your_api_endpoint]/test?action=resp&thingName=vehicle_detector_1_REMOTE&jobId=021d673d-fef4-11eb-9200-0d5b5ed6a46e"

You should get the following response:

{"output": "Hello from vehicle_detector_1", "message": "Respond SUCCEEDED!"}

Test cancel

While not running main.py, we first issue a command

curl -X GET -H 'x-api-key:[your_api_key]' -H 'Accept:application/json' "https://[your_api_endpoint]/test?action=cmd&cmd=greeting&thingNames=vehicle_detector_1_REMOTE"

The command API returns

{"jobId": "61afd4af-fef6-11eb-8d37-3fa1f0109e21", "message": "cmd='greeting' SUCCEEDED!"}

Then we issue a cancel, using the same jobId as above

curl -X GET -H 'x-api-key:[your_api_key]' -H 'Accept:application/json' "https://[your_api_endpoint]/test?action=cancel&jobId=61afd4af-fef6-11eb-8d37-3fa1f0109e21"

The cancel API returns

{"message": "Cancel job 61afd4af-fef6-11eb-8d37-3fa1f0109e21 SUCCEEDED!"}

Robustness tests

Now that we have the API set up, we can run different types of test without having to do mouse clicks. It is also possible to put together a script to automatically test different situations and see how the API responds. While we will not elaborate how the robustness tests should be conducted, we can share some ideas.

Possible tests to run

Wrong endpoint
Wrong/missing API key
Some query string parameter is missing
Wrong/missing thingName
Wrong/missing jobId
Wrong/missing cmd
Wrong/missing action
Use thingName instead of thingNames in the command call
Use thingNames instead of thingName in the response call
Use a method other than GET to call the API

A robust API must return meaningful success or fail messages under all situations.

Summary

In this article, we have gone through detailed steps to create a simple cloud architecture that enables the use of an API to perform command-and-response with an IoT device. This API not only makes it easier to communicate with the IoT device, but also allows us to integrate the IoT device with other technologies. Now we can build any IoT device and write a web and/or mobile app to remotely control it.

However, there is still something missing. Maybe you have noticed it already, but all the command-and-response we can do right now is based on hard-coded scripts already imbedded in the IoT device. What if we need to run some task on the IoT device that hasn’t been hard-coded before? What if we just want to check in on the device to see how it is doing? To satisfy this need, we must be able to SSH into the IoT device at any time. In the next article, we will discuss how remote SSH can be achieved using OpenVPN, even if the IoT device does not have a static IP or is hidden behind a firewall.

Integrate IoT Device with AWS IoT using Python — Part III: command-and-response using API

Introduction

Before You Start

Workflow of command-and-response using API

Set up IAM Role

Create manage_iot_jobs_policy

Create s3_access_policy

Attach both policies to the lambda role

Set up AWS Lambda

Explanation of The Lambda Code

lambda_handler( )

validate()

cancel()

respond()

command()

Test remote_control_api Lambda

Test command

Test response

Test cancel

Set up API Gateway

Test command

Test response

Test cancel

Final Test

Test command

Test response

Test cancel

Robustness tests

Summary

Written by FanchenBao

Create `manage_iot_jobs_policy`