I’m working on an IoT project where I need to store and process large volumes of time-series data using GridDB. I want to integrate GridDB with Google Cloud Storage to archive historical data and leverage Google’s cloud services for further analysis. I have the following requirements:

1. Store real-time IoT data in GridDB.

2. Periodically transfer older data to Google Cloud Storage.

3. Ensure data integrity and efficient retrieval of archived data for analytics.

Here is the code I have so far to connect GridDB and store data:

import griddb_python as griddb
import datetime

# Initialize GridDB connection
factory = griddb.StoreFactory.get_instance()
gridstore = factory.get_store(
    host='0.0.0.0',
    port=31999,
    cluster_name='Zaigham_Cluster',
    username='*',
    password='*'
)

# Create a time-series container
conInfo = griddb.ContainerInfo(
    "iot_data",
    [
        griddb.ContainerField("timestamp", griddb.Type.TIMESTAMP),
        griddb.ContainerField("sensor_id", griddb.Type.STRING),
        griddb.ContainerField("value", griddb.Type.FLOAT)
    ],
    griddb.ContainerType.TIME_SERIES
)

ts = gridstore.put_container(conInfo)
ts.set_auto_commit(False)

# Insert sample data
now = datetime.datetime.now()
for i in range(10):
    ts.put([now, f"sensor_{i}", i * 10.0])
    now += datetime.timedelta(seconds=1)

ts.commit()

Now, I want to periodically archive data to Google Cloud Storage. Here’s what I have tried using the google-cloud-storage library, but I’m unsure how to efficiently transfer data:

from google.cloud import storage
import csv

# Initialize Google Cloud Storage client
client = storage.Client()
bucket_name = 'zaigham'
bucket = client.bucket(bucket_name)
blob = bucket.blob('archived_iot_data.csv')

# Query and export data from GridDB
query = ts.query("SELECT * WHERE timestamp < NOW() - INTERVAL '1 DAY'")
rs = query.fetch()

# Write data to CSV and upload to Google Cloud Storage
with open('/tmp/archived_iot_data.csv', 'w') as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(["timestamp", "sensor_id", "value"])
    while rs.has_next():
        data = rs.next()
        writer.writerow([data[0], data[1], data[2]])

blob.upload_from_filename('/tmp/archived_iot_data.csv')