Related Links

    Integrate ClickHouse DBMS with IDrive® e2

    ClickHouse is a high-performance, open-source column-oriented database management system built for real-time online analytical processing (OLAP) and large-scale analytics. It supports fast query execution, distributed processing, and efficient data compression for massive datasets. Integrating ClickHouse with IDrive® e2 allows you to enhance these capabilities with scalable, secure, and cost-effective cloud object storage.

    Prerequisites

    Before you begin, ensure the following:

    1. An active IDrive® e2 account. Sign up here if you do not have one.
    2. A bucket in IDrive® e2. See how to create a bucket.
    3. Valid Access Key ID and Secret Access Key. Learn how to create an access key.
    4. ClickHouse Server ver.23.0 or later should be installed" and ClickHouse Backup tool (Recommended)

    Configure ClickHouse DBMS with IDrive® e2

    1. Install ClickHouse on your Ubuntu Linux Server.
      2. sudo apt-key adv --keyserver keyserver.ubuntu.com --recv E0C56BD4 
      echo "deb https://packages.clickhouse.com/deb stable main" | sudo tee /etc/apt/sources.list.d/clickhouse.list
      sudo apt update
      sudo apt install clickhouse-server clickhouse-client -y
      sudo systemctl start clickhouse-server

    2. Configure S3 Disk in ClickHouse Using Your IDrive® e2 Keys

      2a. Create a file

      /etc/clickhouse-server/config.d/backup_disk.xml
      <clickhouse>
      <storage_configuration>
      <disks>
      <s3_backup>
      <type>s3</type>
      <endpoint>https://idrivee2-endpoint/bucket-name/</endpoint> (Idrive e2 Endpoint : https://v1e8.da.idrivee2-17.com & iDrive e2 Bucket)
      <access_key_id>YOUR_ACCESS_KEY</access_key_id>
      <secret_access_key>YOUR_SECRET_KEY</secret_access_key>
      <s3_max_single_part_upload_size>10485760</s3_max_single_part_upload_size>
      <upload_part_size>67108864</upload_part_size>
      <s3_max_put_rps>25</s3_max_put_rps>
      <metadata_path>/var/lib/clickhouse/disks/s3_backup/metadata</metadata_path>
      </s3_backup>
      </disks>
      </storage_configuration>
      </clickhouse>

      2b. Restart ClickHouse

       sudo systemctl restart clickhouse-server

      2c. Check log

      /var/log/clickhouse-server/clickhouse-server.log for confirmation

    3. Install ClickHouse Backup (Optional)

      This tool simplifies backup/export and upload to S3:

      wget
      https://github.com/AlexAkulov/clickhouse-backup/releases/download/v2.5.3/clickhouse-backup
      chmod +x clickhouse-backup
      sudo mv clickhouse-backup /usr/local/bin/

    4. Configure the ClickHouse Backup tool

      Edit /etc/ClickHouse-backup/config.yml
      general:
      remote_storage: s3
      disable_progress_bar: true
      s3:
      bucket: clickhouse-backup
      endpoint: v1e8.da.idrivee2-17.com
       access_key: YOUR_ACCESS_KEY
      secret_key: YOUR_SECRET_KEY
      use_path_style: true
      compression_format: tar
      compression_level: 1

    5. Run Backup & Restore Operations

      Backup a specific database:

      clickhouse-backup create my_backup
      clickhouse-backup upload my_backup

      Restore a specific database:

      clickhouse-backup download my_backup
      clickhouse-backup restore my_backup

      Once the backup is complete, open the IDrive® e2 console, navigate to your bucket, and verify that the backup folder/files exist.

    Note: Data restoration is handled by your specific backup solution provider and is affected by multiple variables that are unique to your environment. For application-related enquiries/support, it is strongly recommended you seek guidance from the technical team of your backup solution provider.