1 of 4

Dataset API

In the following sections, you will find more details about the Dataset API.

Create

Use this API to create new datasets for client onboarding, specifying the dataset's structure, settings, and initial data. This API is particularly useful for automating the setup of datasets during the onboarding process, allowing for easy integration with client-specific data requirements.

For example, use this endpoint to define the columns, types, and other properties for a new dataset tailored to a client's needs.

Create a Dataset

To create a new dataset, follow these steps:

To create a new dataset, make a POST request to /dataset-create with the required parameters in the request body.
To specify the dataset structure, include an array of column definitions with each column's name, alias, type, subtype, and optional format.

Type can be:
- measure
- dimension
Subtype can be:
- text
- numeric
- date
- datetime

To customize the dataset further, optional parameters such as CSV settings and a description can be included in the request.

The response will include key details about the created dataset, including the dataset code, table name, and the number of columns.

Example Usage

Creating a New Dataset

For example, making a POST request to the following URL with the provided JSON body would result in the response below: Request

POST /dataset-create
Authorization: Bearer YOUR-TENANT-TOKEN

{
  "user-code": "0f02b4d4f9ae",
  "columns": [
    {
      "name": "InvoiceNo",
      "alias": "InvoiceNo",
      "type": "dimension",
      "subtype": "text"
    },
    {
      "name": "StockCode",
      "alias": "StockCode",
      "type": "dimension",
      "subtype": "text"
    },
    {
      "name": "Description",
      "alias": "Description",
      "type": "dimension",
      "subtype": "text"
    },
    {
      "name": "Quantity",
      "alias": "Quantity",
      "type": "measure",
      "subtype": "numeric",
      "format": "#,###.##"
    },
    {
      "name": "InvoiceDate",
      "alias": "InvoiceDate",
      "type": "dimension",
      "subtype": "datetime",
      "format": "Y-d-m H:i:s"
    },
    {
      "name": "UnitPrice",
      "alias": "UnitPrice",
      "type": "measure",
      "subtype": "numeric",
      "format": "#,###.##"
    },
    {
      "name": "CustomerID",
      "alias": "CustomerID",
      "type": "measure",
      "subtype": "numeric",
      "format": "#,###.##"
    }
  ],
  "name": "Client onboarding dataset creation"
}

Response

{
  "data": {
    "dataset-code": "eca2ad3940e3",
    "table-name": "dataset_csv_eca2ad3940e3",
    "columns": 7
  }
}

This request creates a dataset with the specified columns, each having unique names, types, and formats tailored to client onboarding requirements.

Fill

Use this API to populate an existing dataset with new data. This API allows you to insert or append rows of data into a pre-defined dataset, making it useful for updating client data during onboarding or other data integration tasks.

For example, use this endpoint to fill a dataset with transactional data, customer information, or any other structured data relevant to your client's needs.

Fill a Dataset

To populate a dataset, follow these steps:

Make a POST request to /dataset-fill with the required parameters in the request body.
Include the user-code and dataset-code to identify the dataset and the user making the request.
Define the structure of the data by specifying the columns parameter, which includes details like column names, aliases, types, subtypes, and optional formats.
Provide the data to be inserted via the insert-data parameter.

If compressed is false, the data should be formatted as a JSON-escaped string.
If compressed is true, the data should be base64-encoded after being gzipped.

Example of insert-data with JSON (when compressed: false):

{
  "insert-data": "[[\"536365\",\"85123A\",\"WHITE HANGING HEART T-LIGHT HOLDER\",6,\"2010-12-01 08:26:00\",2.55,17850], [\"536365\",\"71053\",\"WHITE METAL LANTERN\",6,\"2010-12-01 08:26:00\",3.39,17850]]",
  "compressed": false
}

Optionally, use the append parameter to indicate whether to append the data to the existing dataset (true) or truncate the dataset before inserting the new data (false).
Optionally, use the compressed parameter to specify if the data is gzip compressed (true) or not (false).

Example Usage

Filling a Dataset with Base64-encoded Data

For example, making a POST request to the following URL with the provided JSON body would result in the response below:

POST /dataset-fill
Authorization: Bearer YOUR-TENANT-TOKEN

{
  "user-code": "0f02b4d4f9ae",
  "dataset-code": "eca2ad3940e3",
  "columns": [
    {
      "name": "InvoiceNo",
      "alias": "InvoiceNo",
      "type": "dimension",
      "subtype": "text"
    },
    {
      "name": "StockCode",
      "alias": "StockCode",
      "type": "dimension",
      "subtype": "text"
    },
    {
      "name": "Description",
      "alias": "Description",
      "type": "dimension",
      "subtype": "text"
    },
    {
      "name": "Quantity",
      "alias": "Quantity",
      "type": "measure",
      "subtype": "numeric",
      "format": "#,###.##"
    },
    {
      "name": "InvoiceDate",
      "alias": "InvoiceDate",
      "type": "dimension",
      "subtype": "datetime",
      "format": "Y-d-m H:i:s"
    },
    {
      "name": "UnitPrice",
      "alias": "UnitPrice",
      "type": "measure",
      "subtype": "numeric",
      "format": "#,###.##"
    },
    {
      "name": "CustomerID",
      "alias": "CustomerID",
      "type": "measure",
      "subtype": "numeric",
      "format": "#,###.##"
    }
  ],
  "insert-data": "H4sIAAAAAAAACtT9W4/...NDtGHlBlNUP4lMXrO9OfaUk2XMReE/t///68rEPhOT+AC"
  "compressed": true
}

Response

{
  "data": {
    "status": "success",
    "details": {
      "dataset-code": "eca2ad3940e3",
      "rows-count": 518125
    }
  }
}

The response will confirm the status of the operation, including the dataset code and the number of rows inserted. Sample Python Code: Convert CSV Data to a Base64 Encoded String

This Python script reads data from a CSV file, converts it to JSON, compresses it using gzip, and encodes it as a Base64 string, ready to be sent via API requests.

import csv
import json
import gzip
import base64

# Read data from CSV file
def read_csv(file_path):
    rows = []
    with open(file_path, newline='') as csvfile:
        reader = csv.reader(csvfile)
        # Skip the header row
        next(reader, None)
        for row in reader:
            rows.append(row)
    return rows

# Convert data to JSON, compress using gzip, and encode to base64
def prepare_data_for_request(data):
    # Encode data to JSON
    json_encoded = json.dumps(data)

    # Compress JSON data using gzip
    gzipped_insert_data = gzip.compress(json_encoded.encode('utf-8'))

    # Encode gzipped data to base64
    base64_encoded_insert_data = base64.b64encode(gzipped_insert_data).decode('utf-8')

    return base64_encoded_insert_data

# Write base64 encoded data to a file
def write_encoded_data_to_file(encoded_data, file_path):
    with open(file_path, 'w') as file:
        file.write(encoded_data)
    print(f"Base64 encoded data has been written to {file_path}")

# Main function
def main():
    csv_file_path = 'data.csv'  # Replace with your CSV file path
    data = read_csv(csv_file_path)

    base64_encoded_data = prepare_data_for_request(data)

    # Write base64 encoded data to a file
    encoded_data_file = 'encoded_data.txt'  # Replace with your desired file path
    write_encoded_data_to_file(base64_encoded_data, encoded_data_file)

if __name__ == '__main__':
    main()

Use the encoded string saved in the file for your JSON request.

Complete

Description

The /dataset-complete endpoint is designed to signal the end of the dataset insertion process. Once all batches have been inserted via the /dataset-fill endpoint, this method should be called to trigger the final dataset shape calculation and any other necessary post-processing steps.

Parameters

user-code (string): Unique code identifying the user.
dataset-code (string): A unique code for the dataset, if pre-defined.

Indicate end of dataset insertion

To populate a dataset, follow these steps:

Make a POST request to /dataset-complete with the required parameters in the request body.
Include the user-code and dataset-code to identify the dataset and the user making the request.

Example Usage

For example, making a POST with following header included into request with the provided JSON body would result in the response below:

Header

POST /dataset-fill
Authorization: Bearer YOUR-TENANT-TOKEN

Request

{ 
"user-code": "0f02b4d4f9ae", 
"dataset-code": " a49932c0f135"
 }

Response

{
    "data": {
        "status": "success",
        "details": {
            "dataset-code": "a49932c0f135",
            "rows-count": 542289
        }
    }
}

Fill

For example, use this endpoint to fill a dataset with transactional data, customer information, or any other structured data relevant to your client's needs.

Fill a Dataset

To populate a dataset, follow these steps:

Make a POST request to /dataset-fill with the required parameters in the request body.
Include the user-code and dataset-code to identify the dataset and the user making the request.
Define the structure of the data by specifying the columns parameter, which includes details like column names, aliases, types, subtypes, and optional formats.
Provide the data to be inserted via the insert-data parameter.

If compressed is false, the data should be formatted as a JSON-escaped string.
If compressed is true, the data should be base64-encoded after being gzipped.

Example of insert-data with JSON (when compressed: false):

{
  "insert-data": "[[\"536365\",\"85123A\",\"WHITE HANGING HEART T-LIGHT HOLDER\",6,\"2010-12-01 08:26:00\",2.55,17850], [\"536365\",\"71053\",\"WHITE METAL LANTERN\",6,\"2010-12-01 08:26:00\",3.39,17850]]",
  "compressed": false
}

Optionally, use the append parameter to indicate whether to append the data to the existing dataset (true) or truncate the dataset before inserting the new data (false).
Optionally, use the compressed parameter to specify if the data is gzip compressed (true) or not (false).

Example Usage

Filling a Dataset with Base64-encoded Data

For example, making a POST request to the following URL with the provided JSON body would result in the response below:

POST /dataset-fill
Authorization: Bearer YOUR-TENANT-TOKEN

{
  "user-code": "0f02b4d4f9ae",
  "dataset-code": "eca2ad3940e3",
  "columns": [
    {
      "name": "InvoiceNo",
      "alias": "InvoiceNo",
      "type": "dimension",
      "subtype": "text"
    },
    {
      "name": "StockCode",
      "alias": "StockCode",
      "type": "dimension",
      "subtype": "text"
    },
    {
      "name": "Description",
      "alias": "Description",
      "type": "dimension",
      "subtype": "text"
    },
    {
      "name": "Quantity",
      "alias": "Quantity",
      "type": "measure",
      "subtype": "numeric",
      "format": "#,###.##"
    },
    {
      "name": "InvoiceDate",
      "alias": "InvoiceDate",
      "type": "dimension",
      "subtype": "datetime",
      "format": "Y-d-m H:i:s"
    },
    {
      "name": "UnitPrice",
      "alias": "UnitPrice",
      "type": "measure",
      "subtype": "numeric",
      "format": "#,###.##"
    },
    {
      "name": "CustomerID",
      "alias": "CustomerID",
      "type": "measure",
      "subtype": "numeric",
      "format": "#,###.##"
    }
  ],
  "insert-data": "H4sIAAAAAAAACtT9W4/...NDtGHlBlNUP4lMXrO9OfaUk2XMReE/t///68rEPhOT+AC"
  "compressed": true
}

Response

{
  "data": {
    "status": "success",
    "details": {
      "dataset-code": "eca2ad3940e3",
      "rows-count": 518125
    }
  }
}

The response will confirm the status of the operation, including the dataset code and the number of rows inserted. Sample Python Code: Convert CSV Data to a Base64 Encoded String

This Python script reads data from a CSV file, converts it to JSON, compresses it using gzip, and encodes it as a Base64 string, ready to be sent via API requests.

import csv
import json
import gzip
import base64

# Read data from CSV file
def read_csv(file_path):
    rows = []
    with open(file_path, newline='') as csvfile:
        reader = csv.reader(csvfile)
        # Skip the header row
        next(reader, None)
        for row in reader:
            rows.append(row)
    return rows

# Convert data to JSON, compress using gzip, and encode to base64
def prepare_data_for_request(data):
    # Encode data to JSON
    json_encoded = json.dumps(data)

    # Compress JSON data using gzip
    gzipped_insert_data = gzip.compress(json_encoded.encode('utf-8'))

    # Encode gzipped data to base64
    base64_encoded_insert_data = base64.b64encode(gzipped_insert_data).decode('utf-8')

    return base64_encoded_insert_data

# Write base64 encoded data to a file
def write_encoded_data_to_file(encoded_data, file_path):
    with open(file_path, 'w') as file:
        file.write(encoded_data)
    print(f"Base64 encoded data has been written to {file_path}")

# Main function
def main():
    csv_file_path = 'data.csv'  # Replace with your CSV file path
    data = read_csv(csv_file_path)

    base64_encoded_data = prepare_data_for_request(data)

    # Write base64 encoded data to a file
    encoded_data_file = 'encoded_data.txt'  # Replace with your desired file path
    write_encoded_data_to_file(base64_encoded_data, encoded_data_file)

if __name__ == '__main__':
    main()

Use the encoded string saved in the file for your JSON request.

Create

For example, use this endpoint to define the columns, types, and other properties for a new dataset tailored to a client's needs.

Create a Dataset

To create a new dataset, follow these steps:

To create a new dataset, make a POST request to /dataset-create with the required parameters in the request body.
To specify the dataset structure, include an array of column definitions with each column's name, alias, type, subtype, and optional format.

Type can be:
- measure
- dimension
Subtype can be:
- text
- numeric
- date
- datetime

To customize the dataset further, optional parameters such as CSV settings and a description can be included in the request.

The response will include key details about the created dataset, including the dataset code, table name, and the number of columns.

Example Usage

Creating a New Dataset

For example, making a POST request to the following URL with the provided JSON body would result in the response below: Request

POST /dataset-create
Authorization: Bearer YOUR-TENANT-TOKEN

{
  "user-code": "0f02b4d4f9ae",
  "columns": [
    {
      "name": "InvoiceNo",
      "alias": "InvoiceNo",
      "type": "dimension",
      "subtype": "text"
    },
    {
      "name": "StockCode",
      "alias": "StockCode",
      "type": "dimension",
      "subtype": "text"
    },
    {
      "name": "Description",
      "alias": "Description",
      "type": "dimension",
      "subtype": "text"
    },
    {
      "name": "Quantity",
      "alias": "Quantity",
      "type": "measure",
      "subtype": "numeric",
      "format": "#,###.##"
    },
    {
      "name": "InvoiceDate",
      "alias": "InvoiceDate",
      "type": "dimension",
      "subtype": "datetime",
      "format": "Y-d-m H:i:s"
    },
    {
      "name": "UnitPrice",
      "alias": "UnitPrice",
      "type": "measure",
      "subtype": "numeric",
      "format": "#,###.##"
    },
    {
      "name": "CustomerID",
      "alias": "CustomerID",
      "type": "measure",
      "subtype": "numeric",
      "format": "#,###.##"
    }
  ],
  "name": "Client onboarding dataset creation"
}

Response

{
  "data": {
    "dataset-code": "eca2ad3940e3",
    "table-name": "dataset_csv_eca2ad3940e3",
    "columns": 7
  }
}

This request creates a dataset with the specified columns, each having unique names, types, and formats tailored to client onboarding requirements.