Documentation

MQTT Protocol Data Ingestion

Lightweight messaging protocol ideal for resource-constrained devices and real-time data streaming with CivIoT.

Overview

MQTT (Message Queuing Telemetry Transport) is a lightweight, publish-subscribe messaging protocol designed for IoT devices with limited bandwidth and processing power. It's perfect for battery-powered sensors, remote monitoring systems, and real-time data collection scenarios.

CivIoT's MQTT broker supports QoS levels, TLS encryption, and provides reliable message delivery with automatic reconnection handling for unstable network conditions.

MQTT Broker Configuration

Broker Connection Details
Host: mqtt.civiot.com
Port: 1883 (non-TLS) / 8883 (TLS)
Protocol: MQTT v3.1.1 / v5.0

• Supports both MQTT v3.1.1 and v5.0 protocols

• TLS encryption available for secure connections

• Automatic reconnection with exponential backoff

Authentication
Username: your_device_id
Password: your_device_token
Client ID: unique_device_identifier

• Device-specific credentials for secure access

• Client ID must be unique across all devices

• Automatic token refresh support

Topic Structure

Recommended Topic Hierarchy
civiot/{organization_id}/{device_id}/data
civiot/{organization_id}/{device_id}/status
civiot/{organization_id}/{device_id}/config

data: Sensor readings and device measurements

status: Device health and operational status

config: Configuration updates and commands

Message Format

JSON Payload Structure
{ "timestamp": "2024-01-15T10:30:00Z", "device_id": "sensor_001", "sensors": [ { "sensor_id": "temp_001", "type": "temperature", "value": 23.5, "unit": "celsius", "accuracy": 0.1 }, { "sensor_id": "humidity_001", "type": "humidity", "value": 65.2, "unit": "percent", "accuracy": 0.5 } ], "location": { "latitude": 40.7128, "longitude": -74.0060, "altitude": 10.5 }, "metadata": { "battery_level": 85, "signal_strength": -45, "firmware_version": "1.2.3" } }

Required fields: timestamp, device_id, sensors

Optional fields: location, metadata

Message size: Keep messages under 256KB for optimal performance

Quality of Service (QoS) Levels

QoS 0 - At Most Once
Fire and forget delivery. No acknowledgment or retry mechanism.

• Lowest overhead

• No guarantee of delivery

• Best for non-critical data

QoS 1 - At Least Once
Guaranteed delivery with acknowledgment. May result in duplicate messages.

• Reliable delivery

• Possible duplicates

• Good for most IoT use cases

QoS 2 - Exactly Once
Guaranteed delivery with no duplicates. Highest overhead but most reliable.

• Guaranteed delivery

• No duplicates

• Critical data applications

Code Examples

Python Example (Paho MQTT)
import paho.mqtt.client as mqtt import json import time from datetime import datetime # MQTT Configuration broker = "mqtt.civiot.com" port = 1883 username = "your_device_id" password = "your_device_token" client_id = "unique_device_identifier" # Create client client = mqtt.Client(client_id) client.username_pw_set(username, password) def on_connect(client, userdata, flags, rc): print(f"Connected with result code {rc}") def on_publish(client, userdata, mid): print(f"Message {mid} published") client.on_connect = on_connect client.on_publish = on_publish # Connect to broker client.connect(broker, port, 60) client.loop_start() # Publish data topic = "civiot/org123/device001/data" data = { "timestamp": datetime.utcnow().isoformat() + "Z", "device_id": "device001", "sensors": [ { "sensor_id": "temp_001", "type": "temperature", "value": 23.5, "unit": "celsius" } ] } client.publish(topic, json.dumps(data), qos=1) time.sleep(1) client.loop_stop() client.disconnect()
JavaScript Example (MQTT.js)
const mqtt = require('mqtt'); const client = mqtt.connect('mqtt://mqtt.civiot.com:1883', { clientId: 'unique_device_identifier', username: 'your_device_id', password: 'your_device_token', clean: true, reconnectPeriod: 1000, connectTimeout: 30 * 1000 }); client.on('connect', () => { console.log('Connected to MQTT broker'); // Publish data const topic = 'civiot/org123/device001/data'; const data = { timestamp: new Date().toISOString(), device_id: 'device001', sensors: [ { sensor_id: 'temp_001', type: 'temperature', value: 23.5, unit: 'celsius' } ] }; client.publish(topic, JSON.stringify(data), { qos: 1 }, (err) => { if (err) { console.error('Publish error:', err); } else { console.log('Data published successfully'); } }); }); client.on('error', (err) => { console.error('Connection error:', err); });

Best Practices

Use Appropriate QoS Levels

Choose QoS 1 for most IoT data and QoS 2 only for critical information that cannot tolerate duplicates.

Implement Reconnection Logic

Handle network disconnections gracefully with exponential backoff and automatic reconnection.

Keep Messages Small

Limit message size to improve transmission reliability and reduce bandwidth usage.

Use TLS for Security

Enable TLS encryption when transmitting sensitive data over public networks.

Performance & Limits

Connection Limits
Max concurrent connections: 10,000 per organization
Message rate: 100 messages/second per device
Keep-alive: 60 seconds minimum
Message Limits
Max message size: 256KB
Topic length: 255 characters max
Client ID: 23 characters max

Troubleshooting

Common Issues & Solutions

Connection Refused

Check broker URL, port, and firewall settings. Ensure credentials are correct.

Authentication Failed

Verify username/password and ensure device is properly registered in CivIoT.

Message Not Received

Check topic structure, QoS settings, and ensure client is subscribed to the correct topic.

Frequent Disconnections

Increase keep-alive interval and implement proper reconnection logic.

Need Help with MQTT Integration?

If you encounter any issues with MQTT data ingestion or need assistance with your integration, our support team is here to help.

Contact Support