Databricks

Overview of the Databricks integration.

The following documentation is for Databricks v1.1.0.

Overview

Databricks is a unified analytics platform for data engineering, data science, and machine learning. This integration enables automated data pipeline management, notebook execution, and analytics workflows within your Databricks workspace.

Setup

The following integrations need to be connected to your Kognitos workspace:

• Databricks

Steps

Follow these steps to connect the integration in Kognitos:

1

Navigate

Using the left navigation menu, go to Integrations → Explore Integrations.

2

Find

Search for the integration and click on it.

3

Connect

Click on Connect to add a connection to the integration.

4

Configure

Add a name for the connection. You'll be prompted for authentication details if needed. Then, click on Connect.

Authentication

Use one of the following authentication methods to connect this integration in Kognitos. Each method has its own configuration requirements.

Connect using Cluster URL and Access Token

Connects to Databricks using the provided access token.

Label	Description	Type
Cluster URL	The Databricks cluster URL	text
Access Token	The access token to be used for connecting	sensitive

Actions

The following actions are available in the Databricks integration:

1. Append a file to a table

Append data from a file to an existing table.

2. Append a table to a databricks table

Append data from a Kognitos table to an existing Databricks table.

3. Create a databricks table from a table

Create a table in Databricks from a Kognitos table.

4. Create databricks table from a file

Create a table from an uploaded file.

5. Get a catalog

Get a catalog by its name.

6. Get file at a path

Get file metadata for a file at the given path.

7. Get some catalog's schema

Get a schema from a catalog by its name.

8. Get some warehouses

List all warehouses in the workspace.

9. Upload a file to databricks

Upload a file to a catalog volume.

Concepts

Databricks warehouse

Warehouse in Databricks.

Field Name	Description	Type
id	The unique identifier of the warehouse	text
name	The name of the warehouse	text

Databricks file

File in Databricks.

Field Name	Description	Type
schema	The schema containing the file	json
path	The path to the file	text
file_type	The MIME type of the file	text
file_format	The file format extension (e.g., csv, json, parquet)	text

Databricks schema

Schema in Databricks.

Field Name	Description	Type
name	The name of the schema	text
catalog	The catalog containing this schema	json

Databricks catalog

Catalog in Databricks.

Field Name	Description	Type
name	The name of the catalog	text