Set up MQTT publisher and subscriber

Introduction to MQTT

MQTT (Message Queuing Telemetry Transport) is a lightweight, publish-subscribe network protocol that transports messages between devices. It was designed to be simple and easy to implement, which makes it an ideal protocol for IoT (Internet of Things) scenarios where bandwidth and power are at a premium. MQTT operates over TCP/IP, making it reliable in environments where devices need to communicate over a network.

Originally developed by IBM, MQTT has become the de facto standard for IoT communication due to its efficiency and low overhead. It allows for real-time messaging between a wide range of devices, from tiny sensors to full-fledged industrial machines, ensuring that information is delivered in a timely and reliable manner.

Key Concepts

  • Broker: The central server that manages the routing of messages between clients, ensuring that messages published to a particular topic are delivered to all clients subscribed to that topic. The broker can handle thousands of devices, making it scalable for large IoT deployments. It also manages the sessions, keeps track of the clients, and ensures that messages are delivered according to the specified QoS levels.

  • Publisher: A device or application that sends (publishes) messages to a specific topic on the broker. Publishers can be any IoT device, such as sensors, smartphones, or computers. They do not need to know who the subscribers are; they just send the message to a topic, and the broker handles the delivery.

  • Subscriber: A device or application that receives (subscribes to) messages from a specific topic on the broker. Subscribers can also be any device that needs to consume the data, such as a dashboard, a control system, or another sensor. Subscribers can filter the topics they are interested in, allowing them to only receive relevant information.

  • Topic: A hierarchical string that the broker uses to filter messages for each connected client. Topics can be structured in a tree-like format, allowing for granular control over message routing. For example, a topic could be home/livingroom/temperature, where home is the root, livingroom is a sub-topic, and temperature is the specific topic.

  • QoS (Quality of Service): MQTT defines three levels of QoS to ensure the reliability of message delivery between the publisher and the subscriber:

    • QoS 0: At most once (fire and forget). The message is delivered once without any confirmation, and there is no guarantee that the message will arrive at the subscriber. This is suitable for scenarios where occasional message loss is acceptable, such as environmental monitoring where updates are frequent.

    • QoS 1: At least once (message is delivered at least one time, but could be delivered more than once). The message is guaranteed to be delivered at least once to the subscriber, but it may arrive multiple times if acknowledgments are lost. This level is used when it is important to ensure that the message is received, even if duplicates occur.

    • QoS 2: Exactly once (message is delivered exactly once). The highest level of QoS, ensuring that the message is delivered exactly once. This is crucial in financial or critical systems where duplicate messages could cause serious problems.

Setting Up MQTT with AWS IoT Core

AWS IoT Core is a managed cloud service provided by Amazon Web Services that enables connected devices to easily and securely interact with cloud applications and other devices. It supports all the major IoT protocols, including MQTT, HTTP, and WebSockets, making it a versatile choice for IoT solutions. AWS IoT Core manages all the infrastructure required to handle the scalability, security, and availability of your IoT application, allowing you to focus on the device and application logic.

Step 1: Create an IoT Thing

Fortunately, the IoT core for AIMPF is already set up and running. Therefore, we will skip the setup of the IoT core and focus on setting up the MQTT publisher and subscriber to interact with the IoT core.

Step 2: Generate Certificates

For secure communication between your device and AWS IoT Core, you need to generate and configure security certificates. AWS IoT Core uses X.509 certificates to authenticate devices that attempt to connect to your IoT Core endpoint.

As the administrator of the IoT core, you can generate the certificates from the AWS IoT console, activate it and attach a policy to it. The policy defines the permissions your device has within AWS IoT Core. The policy specifies the actions (e.g., connect, receive, publish, subscribe) that the device can perform and the resources (e.g., MQTT topics) these actions apply to.

As a device developer, you will need to download the certificates and keys generated by the administrator and use them to establish a secure connection with AWS IoT Core. To acquire the certificates, please contact Andrew Dugenske (dugenske@gatech.edu) directly.

The full list of security files include:

  • AmazonRootCA1.pem or AmazonRootCA3.pem: The root CA certificate for AWS IoT Core. This is used to verify the server’s certificate during the TLS handshake.

  • <certificate-id>-certificate.pem.crt: The device certificate. This is used to authenticate the device to AWS IoT Core.

  • <certificate-id>-private.pem.key: The private key associated with the device certificate. This is used to establish a secure connection with AWS IoT Core.

  • <certificate-id>-public.pem.key: The public key associated with the device certificate. This is used for encryption and decryption. This is OPTIONAL and not required for the MQTT connection.

Note: You should also receive the endpoint url (or, broker, as specified in the codes below) from Andrew Dugenske.

Step 3: Setting Up MQTT Publisher and Subscriber

In this step, you’ll set up the actual publisher and subscriber components using Python and the Paho MQTT library. The publisher will send messages to a topic, and the subscriber will listen to that topic and print any messages it receives.

Option 1: Using Python with Paho MQTT

Paho is a popular MQTT client library that makes it easy to implement both MQTT publishers and subscribers in Python. It is maintained by the Eclipse Foundation and supports many programming languages.

  • Install Paho MQTT Library:

    The first step is to install the Paho MQTT library using pip. This library provides the necessary functions to connect to an MQTT broker, publish messages, and subscribe to topics.

    pip install paho-mqtt

    After running this command, the Paho MQTT library will be installed and ready to use in your Python scripts.

  • Publisher Example:

    The following Python script sets up an MQTT publisher that connects to AWS IoT Core and sends a message to a topic. This script will use the certificates and keys generated earlier to establish a secure connection.

    pub_aimpf.py
    import paho.mqtt.client as mqtt
import ssl
import os
from datetime import datetime
import threading

# >>>> Update the following details for the IoT device >>>>

# Define the MQTT broker details
broker = "endpoint/url"  # The endpoint URL/alias of your AWS IoT Core
port = 8883  # Use 8883 for AWS IoT Core
topic = "your/topic"  # The topic to publish to

# Define the paths to the CA certificate, client certificate, and client key
ca_cert = os.path.join(os.getcwd(), r"path/to/AmazonRootCA1.pem")
client_cert = os.path.join(os.getcwd(), r"path/to/your/certificate.pem.crt")
client_key = os.path.join(os.getcwd(), r"path/to/your/private-key.pem")

# <<<< Update the above details for the IoT device <<<<

# Create a new MQTT client instance with a unique client_id
client = mqtt.Client(client_id="YourPublisher")

# Configure TLS/SSL with client certificates
client.tls_set(ca_certs=ca_cert, certfile=client_cert, keyfile=client_key, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLS)

def publish_message():
    # This function will be called every 2 seconds
    timestamp = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
    message = f"Hello from MQTT YourPublisher! Timestamp: {timestamp}"
    client.publish(topic, message)
    print(f"Published message: {message} to topic: {topic}")
    # Set up the timer to call this function again after 2 seconds
    threading.Timer(2, publish_message).start()

def on_connect(client, userdata, flags, rc):
    if rc == 0:
        print("Connected successfully")
        # Start publishing messages periodically
        publish_message()
    else:
        print(f"Connection failed with code {rc}")

client.on_connect = on_connect

# Connect to the MQTT broker
client.connect(broker, port, 60)

# Start the network loop in a separate thread
client.loop_start()

# The main thread can now do other tasks or just wait
try:
    while True:
        pass  # Keep the main thread running
except KeyboardInterrupt:
    client.loop_stop()
    print("Publisher stopped")

    This script initializes the MQTT client, configures it with the appropriate security settings, and connects it to the AWS IoT Core endpoint. Upon successful connection, it publishes a message to the topic your/topic.

  • Subscriber Example:

    The following Python script sets up an MQTT subscriber that connects to AWS IoT Core and listens for messages on the your/topic topic. When a message is received, it prints the message payload to the console.

    sub_aimpf.py
    import paho.mqtt.client as mqtt
import ssl
import os

# >>>> Update the following details for the IoT device >>>>

# Define the MQTT broker details
broker = "endpoint/url"  # The endpoint URL/alias of your AWS IoT Core
port = 8883  # Use 8883 for AWS IoT Core
topic = "your/topic"  # The topic to publish to

# Define the paths to the CA certificate, client certificate, and client key
ca_cert = os.path.join(os.getcwd(), r"path/to/AmazonRootCA1.pem")
client_cert = os.path.join(os.getcwd(), r"path/to/your/certificate.pem.crt")
client_key = os.path.join(os.getcwd(), r"path/to/your/private-key.pem")

# <<<< Update the above details for the IoT device <<<<

# Create a new MQTT client instance with a unique client_id
client = mqtt.Client(client_id="YourSubscriber")

# Enable logging to get more insights
client.enable_logger()

# Configure TLS/SSL with client certificates
client.tls_set(ca_certs=ca_cert, certfile=client_cert, keyfile=client_key, cert_reqs=ssl.CERT_REQUIRED, tls_version=ssl.PROTOCOL_TLS)

def on_connect(client, userdata, flags, rc):
    if rc == 0:
        print("Connected successfully")
        client.subscribe(topic)
        print(f"Subscribed to topic: {topic}")
    else:
        print(f"Connection failed with code {rc}")

def on_message(client, userdata, msg):
    print(f"Received message: {msg.payload.decode()} from topic: {msg.topic}")

client.on_connect = on_connect
client.on_message = on_message

# Connect to the MQTT broker
client.connect(broker, port, 60)

# Start the network loop
client.loop_forever()

    This script initializes the MQTT client, configures it with the security settings, and connects it to the AWS IoT Core endpoint. Upon successful connection, it subscribes to the topic your/topic and waits for incoming messages. When a message is received, it is printed to the console.

Note: the broker, or endpoint url, should not contain ‘http://’ nor end with a ‘/’ as typical urls do. This will look something like: uuid.region.amazonaws.com.

Option 2: Using AWS IoT SDKs

For more complex setups or if you prefer to work with AWS-specific tools, AWS provides SDKs for Python, JavaScript, and other languages that simplify the process of connecting to AWS IoT Core. These SDKs handle much of the connection, subscription, and publishing logic automatically, allowing you to focus on your application’s functionality rather than the underlying infrastructure.

To use the AWS IoT SDKs, follow the installation instructions provided by AWS for your preferred language, and refer to their extensive documentation for guidance on how to implement your publisher and subscriber.

Step 4: Testing

Now that both the publisher and subscriber are set up, it’s time to test the communication between them. This step ensures that your device can successfully send and receive messages via AWS IoT Core.

  1. Run the Subscriber Script First: Open a terminal and run the subscriber script.

python3 sub_aimpf.py

OR

python sub_aimpf.py

The script will connect to AWS IoT Core and subscribe to the topic your/topic. It will remain active, waiting to receive messages published to that topic.

  1. Run the Publisher Script: In a separate terminal, run the publisher Script.

python3 pub_aimpf.py

OR

python pub_aimpf.py

The script will connect to AWS IoT Core and publish a message to the topic your/topic. You should see a confirmation in the terminal where the publisher is running, indicating that the message was sent.

  1. Observe the Message Received by the Subscriber: Switch back to the terminal where the subscriber is running. You should see the message that was published by the publisher script. The message will be printed to the console, confirming that the end-to-end communication is working correctly.

  2. Troubleshooting: If you do not see the message on the subscriber’s side, check the following:

    • Ensure that the AWS IoT endpoint, certificates, and keys are correctly specified in both scripts.

    • Verify that the policy attached to the device’s certificate allows the necessary MQTT actions (connect, publish, subscribe).

    • Make sure that the device is properly registered and that the certificate is activated.

Note

For security reasons, AWS IoT Core permits communication from a device only through a single application at a time. If multiple applications attempt to communicate with the device, IoT Core will cease communication with it. To connect a new device, you must either stop the previous application and start the new one or request the creation of a new IoT Thing in IoT Core, along with a new set of certificates for the new device.

Conclusion

This expanded tutorial provides a more detailed walkthrough of the process of setting up MQTT with AWS IoT Core for AIMPF. You now understand the key concepts of MQTT, how to acquire the certificates, and how to implement both a publisher and a subscriber in Python using the Paho MQTT library.

This foundational knowledge can be expanded upon by integrating more complex IoT devices, leveraging additional AWS IoT services like AWS IoT Analytics, AWS IoT Greengrass, and AWS Lambda, and implementing more secure and scalable architectures to meet your specific IoT needs. With this setup, you’re well on your way to building robust and scalable IoT applications using MQTT and AWS IoT Core.