Documentation

FTP File Upload Data Ingestion

Bulk data import via FTP file uploads. Ideal for historical data migration, batch processing, and large dataset imports.

Overview

FTP (File Transfer Protocol) data ingestion allows you to upload large files containing sensor data, device readings, and historical information to CivIoT. This method is perfect for scenarios where you need to import bulk data, migrate from existing systems, or process large datasets that aren't suitable for real-time streaming.

Our FTP service supports multiple file formats including CSV, JSON, Excel, and custom delimited files, with automatic format detection and data validation.

FTP Server Configuration

FTP Connection Details
Host: ftp.civiot.com
Port: 21 (FTP) / 990 (FTPS)
Protocol: FTP, FTPS (FTP over SSL/TLS)
Mode: Passive mode recommended

• Supports both standard FTP and secure FTPS

• Passive mode for firewall-friendly connections

• Automatic SSL/TLS negotiation for FTPS

Authentication & Security
Username: your_organization_id
Password: your_ftp_token
Encryption: TLS 1.2+ (FTPS)
Key-based auth: Supported

• Organization-specific credentials

• TLS encryption for secure transfers

• SSH key authentication available

File Structure & Organization

Recommended Directory Structure
/incoming/
├── /data/
│ ├── /sensors/
│ ├── /devices/
│ └── /historical/
├── /processed/
└── /failed/

incoming/data: Upload your data files here for processing

processed: Successfully processed files are moved here

failed: Files with processing errors are moved here

Supported File Formats

CSV Files
Comma-separated values with automatic delimiter detection.

• UTF-8 encoding recommended

• Auto-detects delimiters (comma, semicolon, tab)

• Header row support

• Maximum file size: 100MB

JSON Files
JavaScript Object Notation format for structured data.

• Single JSON object or array of objects

• Nested structure support

• Schema validation

• Maximum file size: 50MB

Excel Files
Microsoft Excel (.xlsx) and legacy (.xls) formats.

• Multiple worksheet support

• Auto-detects data ranges

• Formula evaluation

• Maximum file size: 25MB

Custom Delimited
Custom delimiter-separated files with configuration.

• Pipe, tab, or custom delimiters

• Fixed-width format support

• Custom field mapping

• Maximum file size: 100MB

Data Format Examples

CSV Format Example
timestamp,device_id,sensor_id,type,value,unit,latitude,longitude 2024-01-15T10:00:00Z,device001,temp001,temperature,23.5,celsius,40.7128,-74.0060 2024-01-15T10:01:00Z,device001,humidity001,humidity,65.2,percent,40.7128,-74.0060 2024-01-15T10:02:00Z,device001,pressure001,pressure,1013.25,hPa,40.7128,-74.0060

Required columns: timestamp, device_id, sensor_id, type, value, unit

Optional columns: latitude, longitude, altitude, accuracy, metadata

JSON Format Example
[ { "timestamp": "2024-01-15T10:00:00Z", "device_id": "device001", "sensor_id": "temp001", "type": "temperature", "value": 23.5, "unit": "celsius", "location": { "latitude": 40.7128, "longitude": -74.0060 } }, { "timestamp": "2024-01-15T10:01:00Z", "device_id": "device001", "sensor_id": "humidity001", "type": "humidity", "value": 65.2, "unit": "percent", "location": { "latitude": 40.7128, "longitude": -74.0060 } } ]

Upload & Processing Workflow

1. Upload

Upload file via FTP to /incoming/data/

2. Validation

File format and data structure validation

3. Processing

Data parsing and transformation

4. Storage

Data stored in CivIoT database

5. Cleanup

File moved to /processed/ or /failed/

Processing Options & Configuration

Processing Modes
Immediate processing (default)
Scheduled processing
Batch processing
Data Handling
Duplicate detection
Data validation rules
Conflict resolution

Code Examples

Python FTP Upload
from ftplib import FTP import os def upload_file_to_civiot(local_file, remote_path): # Connect to CivIoT FTP server ftp = FTP('ftp.civiot.com') ftp.login('your_org_id', 'your_ftp_token') # Change to data directory ftp.cwd('/incoming/data') # Upload file with open(local_file, 'rb') as file: ftp.storbinary(f'STOR {remote_path}', file) print(f"File {local_file} uploaded successfully") ftp.quit() # Example usage upload_file_to_civiot('sensor_data.csv', 'sensors/sensor_data.csv')
Bash Script Example
#!/bin/bash # FTP configuration FTP_HOST="ftp.civiot.com" FTP_USER="your_org_id" FTP_PASS="your_ftp_token" REMOTE_DIR="/incoming/data" # Upload file ftp -n $FTP_HOST << EOF user $FTP_USER $FTP_PASS cd $REMOTE_DIR put sensor_data.csv bye EOF echo "File uploaded successfully"

Monitoring & Status

Processing Status
Pending - File uploaded, waiting for processing
Processing - Currently being processed
Completed - Successfully processed
Failed - Processing errors occurred
Status Check Methods
Check /processed/ and /failed/ directories
Monitor processing logs via API
Email notifications for completion

Best Practices

Use Descriptive Filenames

Include date, device type, and data type in filenames for easy identification.

Validate Data Before Upload

Ensure data quality and format compliance before uploading to avoid processing failures.

Use FTPS for Security

Always use FTPS (FTP over SSL/TLS) when transmitting sensitive data.

Monitor Processing Status

Regularly check processing status and handle failed uploads promptly.

Need Help with FTP Integration?

If you encounter any issues with FTP data ingestion or need assistance with your file uploads, our support team is here to help.

Contact Support