Documentation

HTTP Data Ingestion

Configure CivIoT to pull data from external APIs and services. CivIoT actively polls external systems to collect sensor data, measurements, and operational information.

Overview

HTTP data ingestion in CivIoT uses a pull-based architecture where CivIoT actively requests data from external systems:

  • • External APIs: Third-party services, IoT platforms, and custom systems expose data through REST APIs
  • • CivIoT Polling: CivIoT server periodically calls these external APIs to pull the latest data
  • • Data Processing: Retrieved data is transformed, validated, and stored in CivIoT's platform
  • • Automated Collection: No manual intervention required - data collection runs on configured schedules

This approach is ideal for integrating with existing systems that already have API interfaces, legacy databases with HTTP endpoints, or third-party data providers.

How It Works

1. Configure
Set up external API endpoints, authentication, and polling schedules in CivIoT
2. Poll APIs
CivIoT automatically calls external APIs at configured intervals to pull data
3. Process Data
Retrieved data is transformed, validated, and stored in CivIoT's infrastructure
4. Access & Analyze
Data becomes available in CivIoT's dashboard, analytics, and reporting tools

External API Configuration

API Endpoint Configuration
External API: https://api.external-system.com/v1/sensors
Configure the external API endpoints that CivIoT will poll for data collection.

• Base URL and endpoint paths

• HTTP method (GET, POST, etc.)

• Query parameters and headers

• Response format specification

Polling Schedule
Polling Interval: Every 5 minutes Cron Expression: */5 * * * *
Set how frequently CivIoT should poll each external API for new data.

• Time-based intervals (every X minutes/hours)

• Cron expressions for complex schedules

• Different schedules for different data sources

• Peak/off-peak timing considerations

Authentication
API Key: X-API-Key: external_api_key_123 Bearer Token: Authorization: Bearer jwt_token_456
Configure authentication methods for accessing external APIs from CivIoT.

• API keys in headers or query params

• Bearer tokens and JWT authentication

• Basic authentication

• OAuth 2.0 flows

Data Transformation

Mapping External Data to CivIoT Format

External API Response

{ "sensor_data": [ { "id": "temp_sensor_01", "reading": 23.5, "unit": "C", "timestamp": "2024-01-15T10:30:00Z", "location": "Building A" } ] }

CivIoT Internal Format

{ "device_id": "temp_sensor_01", "timestamp": "2024-01-15T10:30:00Z", "sensors": [ { "sensor_id": "temp_sensor_01", "type": "temperature", "value": 23.5, "unit": "celsius" } ], "metadata": { "source": "external_api", "location": "Building A" } }

Field Mapping: Configure how external API fields map to CivIoT's internal data structure

Data Validation: Set rules for data quality and required fields

Error Handling: Define what to do with malformed or missing data

Configuration Examples

YAML Configuration
# CivIoT HTTP Ingestion Configuration api_sources: weather_station: endpoint: "https://api.weather.com/v1/station/12345" method: "GET" headers: Authorization: "Bearer weather_api_token" polling_interval: "*/15 * * * *" # Every 15 minutes data_mapping: temperature: "current.temp" humidity: "current.humidity" timestamp: "timestamp" location: "station.location" industrial_sensors: endpoint: "https://factory-api.company.com/sensors" method: "POST" body: query: "SELECT * FROM sensors WHERE updated_at > :last_poll" headers: X-API-Key: "factory_api_key" polling_interval: "*/5 * * * *" # Every 5 minutes
JSON Configuration
{ "api_sources": { "smart_city_sensors": { "endpoint": "https://city-api.gov/sensors", "method": "GET", "headers": { "Authorization": "Bearer city_api_token" }, "polling_interval": "*/10 * * * *", "data_mapping": { "device_id": "sensor.id", "timestamp": "sensor.timestamp", "sensors": [ { "sensor_id": "sensor.id", "type": "sensor.type", "value": "sensor.value", "unit": "sensor.unit" } ] } } } }

Test External API Connection

API Connection Tester
Test your external API endpoints to ensure CivIoT can successfully connect and retrieve data. Enter a URL below and click "Test Connection" to see the raw API response.
💡 Protocol (http:// or https://) will be auto-added if missing
💡 Data loggers may need 30-60 seconds

💡 Tips for testing:

  • • Auto-Protocol: Just enter the domain (e.g., "api.example.com") - http:// will be added automatically
  • • CORS Proxy Fallback: The tester automatically tries multiple CORS proxy services
  • • Data Logger URLs: Works great with Campbell Scientific, Insitu, and other IoT systems
  • • Timeout Settings: Increase timeout for slow-responding industrial systems (30-60 seconds)
  • • Alternative: If proxies fail, test from your backend or use browser dev tools with CORS disabled

Common Use Cases

Third-party IoT Platforms

Pull data from existing IoT platforms like ThingSpeak, Ubidots, or custom sensor networks.

• Weather station APIs

• Industrial monitoring systems

• Smart home platforms

• Agricultural monitoring services

Legacy System Integration

Connect to older systems that expose data through HTTP endpoints or REST APIs.

• SCADA systems with HTTP interfaces

• Database gateways

• Custom enterprise applications

• Equipment monitoring systems

Data Provider Services

Integrate with external data services that provide specialized information.

• Financial market data APIs

• Traffic and transportation APIs

• Environmental monitoring services

• Social media analytics APIs

Multi-site Operations

Collect data from multiple locations, each with their own API endpoints.

• Branch office systems

• Remote monitoring stations

• Partner organization APIs

• Distributed sensor networks

Monitoring & Health Checks

Connection Status
Monitor external API connectivity and response times
Data Quality
Track data validation success rates and error patterns
Error Handling
Monitor failed API calls and data processing errors

Best Practices for HTTP Ingestion

Implement Exponential Backoff

Use intelligent retry logic when external APIs are temporarily unavailable.

Respect Rate Limits

Configure polling intervals that don't exceed external API rate limits.

Data Validation

Implement robust validation rules to ensure data quality from external sources.

Error Logging

Comprehensive logging for debugging API integration issues and data problems.

Incremental Data Collection

Use timestamps to only collect new data since the last successful poll.

Error Handling & Troubleshooting

Common Issues & Solutions
401/403Authentication errors - Check API keys and token expiration
429Rate limiting - Increase polling intervals or implement backoff
500/502External API errors - Monitor external service health
TimeoutConnection issues - Check network connectivity and API response times

Need Help with HTTP Ingestion?

If you encounter issues configuring external API connections or need assistance with data mapping, our support team is here to help.

Contact Support