Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
62 changes: 62 additions & 0 deletions docs/ecosystem/data-integration/bladepipe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
---
title: BladePipe
---

# BladePipe

[BladePipe](https://www.bladepipe.com/) is a real-time data integration and CDC platform. It helps you build data pipelines in a no-code way between different databases, data warehouses and data lakes, including Apache Cloudberry.

BladePipe supports data movement from PostgreSQL/Greenplum to Apache Cloudberry, including:

- Schema migration.
- Full data migration.
- Incremental data synchronization.
- Data verification and correction.
- DDL synchronization for common schema changes.

This document will show how to synchronize data from PostgreSQL to Apache Cloudberry.

## Prerequisites

- Make sure the PostgreSQL account has the [required permissions](https://www.bladepipe.com/docs/dataMigrationAndSync/datasource_func/PostgreSQL/privs_for_pg/).
- Apache Cloudberry is deployed and running.
- [BladePipe (v1.6.0 or higher)](https://www.bladepipe.com/docs/productOP/onPremise/installation/install_all_in_one_docker/) is installed and accessible.

## Steps

### Step 1: Add DataSources

1. Log in to BladePipe. Go to **DataSource** > **Add DataSource**.
![](../../media/bladepipe1.png)

2. Add Apache Cloudberry and PostgreSQL separately as a DataSource:
- **Deployment**: Select **Self-managed**.
- **Type**: Select **Cloudberry** / **PostgreSQL**.
- **Host**: Enter the database host and port.
- **Account & Password**: Enter the username and password of Cloudberry / PostgreSQL.

3. Click **Test Connection**. Then click **Add DataSource**.
![](../../media/bladepipe2.png)

### Step 2: Create a DataJob

1. Go to **DataJob** > **Create DataJob**.
![](../../media/bladepipe4.png)

2. Select PostgreSQL as the source DataSource and Cloudberry as the target DataSource.
![](../../media/bladepipe5.png)

3. Choose the DataJob type. For synchronization tasks, select **Incremental** with **Initial Load**. The initial load migrates existing data. Incremental sync keeps Cloudberry updated after the initial load finishes.
![](../../media/bladepipe6.png)

4. Select which tables to sync. You can include everything or pick specific ones.
![](../../media/bladepipe7.png)

5. Select the columns to sync. Here you can process data, like filtering and transformation.
![](../../media/bladepipe8.png)

6. Click **Create DataJob** to start the DataJob.
![](../../media/bladepipe9.png)

7. You can check the pipeline status and progress on the DataJob page.
![](../../media/bladepipe10.png)
60 changes: 60 additions & 0 deletions docs/ecosystem/sql-clients/clouddm.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
title: CloudDM
---

# CloudDM

CloudDM is an open-source database management tool for teams. It provides a web console for SQL queries, database management, access control, SQL auditing, approval workflows, and database CI/CD. It supports multiple databases, including MySQL, Oracle, StarRocks, and Apache Cloudberry.

This document shows how to connect CloudDM to Apache Cloudberry and run SQL queries.


## Prerequisites

- Apache Cloudberry is deployed and running.
- Database access permissions are properly configured in `pg_hba.conf`.
- [CloudDM (v4.1.0 or higher)](https://github.com/ClouGence/open-cdm) is installed and accessible.

## Steps

1. Log in to the CloudDM console, and click **Back to Workspace**.

2. In the left panel, click **Instances**, and then click **New** to add a database connection.
![](../../media/clouddm1.png)

3. Select Cloudberry and click **Submit**.
![](../../media/clouddm2.png)

4. Enter the Cloudberry connection information:
- **Host**: Enter the hostname or IP address of your Cloudberry instance.
- **Port**: Enter the port.
- **Authentication**: Select an authentication method and enter the required information (for example, username and password)

![](../../media/clouddm3.png)

5. Test the connection. Once Cloudberry is connected to CloudDM, click **Add DataSource**.

If the test fails, check the Cloudberry address, port, database user, password, network access, and `pg_hba.conf` rules.

6. Go to **SQL Query**.
![](../../media/clouddm4.png)

7. The added Cloudberry instance is shown in the left panel. Now you can start running a query.
![](../../media/clouddm5.png)

## Other features

### Work orders

CloudDM supports work orders for production database changes. Teams can submit, approve, and execute changes through a controlled workflow.
![](../../media/clouddm6.png)

### CI/CD

CloudDM can trigger database CI/CD workflows through Git Push, Web Hook, and HttpCall. Teams can include SQL checks, approvals, and execution in application release processes, and manage database change scripts in one place.
![](../../media/clouddm7.png)

### User access control

CloudDM supports role-based access control (RBAC) for team collaboration. Teams can grant access by role and control which data sources and operations each user can use.
![](../../media/clouddm8.png)
Binary file added docs/media/bladepipe1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/bladepipe10.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/bladepipe2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/bladepipe3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/bladepipe4.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/bladepipe5.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/bladepipe6.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/bladepipe7.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/bladepipe8.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/bladepipe9.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/clouddm1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/clouddm2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/clouddm3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/clouddm4.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/clouddm5.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/clouddm6.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/clouddm7.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added docs/media/clouddm8.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
30 changes: 0 additions & 30 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions sidebars.ts
Original file line number Diff line number Diff line change
Expand Up @@ -308,12 +308,12 @@ const sidebars: SidebarsConfig = {
{
type: 'category',
label: 'SQL Clients',
items: ['ecosystem/sql-clients/dbeaver']
items: ['ecosystem/sql-clients/dbeaver', 'ecosystem/sql-clients/clouddm']
},
{
type: 'category',
label: 'Data Integration',
items: ['ecosystem/data-integration/seatunnel']
items: ['ecosystem/data-integration/seatunnel', 'ecosystem/data-integration/bladepipe']
},
{
type: 'category',
Expand Down
62 changes: 62 additions & 0 deletions versioned_docs/version-2.x/ecosystem/data-integration/bladepipe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
---
title: BladePipe
---

# BladePipe

[BladePipe](https://www.bladepipe.com/) is a real-time data integration and CDC platform. It helps you build data pipelines in a no-code way between different databases, data warehouses and data lakes, including Apache Cloudberry.

BladePipe supports data movement from PostgreSQL/Greenplum to Apache Cloudberry, including:

- Schema migration.
- Full data migration.
- Incremental data synchronization.
- Data verification and correction.
- DDL synchronization for common schema changes.

This document will show how to synchronize data from PostgreSQL to Apache Cloudberry.

## Prerequisites

- Make sure the PostgreSQL account has the [required permissions](https://www.bladepipe.com/docs/dataMigrationAndSync/datasource_func/PostgreSQL/privs_for_pg/).
- Apache Cloudberry is deployed and running.
- [BladePipe (v1.6.0 or higher)](https://www.bladepipe.com/docs/productOP/onPremise/installation/install_all_in_one_docker/) is installed and accessible.

## Steps

### Step 1: Add DataSources

1. Log in to BladePipe. Go to **DataSource** > **Add DataSource**.
![](../../media/bladepipe1.png)

2. Add Apache Cloudberry and PostgreSQL separately as a DataSource:
- **Deployment**: Select **Self-managed**.
- **Type**: Select **Cloudberry** / **PostgreSQL**.
- **Host**: Enter the database host and port.
- **Account & Password**: Enter the username and password of Cloudberry / PostgreSQL.

3. Click **Test Connection**. Then click **Add DataSource**.
![](../../media/bladepipe2.png)

### Step 2: Create a DataJob

1. Go to **DataJob** > **Create DataJob**.
![](../../media/bladepipe4.png)

2. Select PostgreSQL as the source DataSource and Cloudberry as the target DataSource.
![](../../media/bladepipe5.png)

3. Choose the DataJob type. For synchronization tasks, select **Incremental** with **Initial Load**. The initial load migrates existing data. Incremental sync keeps Cloudberry updated after the initial load finishes.
![](../../media/bladepipe6.png)

4. Select which tables to sync. You can include everything or pick specific ones.
![](../../media/bladepipe7.png)

5. Select the columns to sync. Here you can process data, like filtering and transformation.
![](../../media/bladepipe8.png)

6. Click **Create DataJob** to start the DataJob.
![](../../media/bladepipe9.png)

7. You can check the pipeline status and progress on the DataJob page.
![](../../media/bladepipe10.png)
60 changes: 60 additions & 0 deletions versioned_docs/version-2.x/ecosystem/sql-clients/clouddm.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
---
title: CloudDM
---

# CloudDM

CloudDM is an open-source database management tool for teams. It provides a web console for SQL queries, database management, access control, SQL auditing, approval workflows, and database CI/CD. It supports multiple databases, including MySQL, Oracle, StarRocks, and Apache Cloudberry.

This document shows how to connect CloudDM to Apache Cloudberry and run SQL queries.


## Prerequisites

- Apache Cloudberry is deployed and running.
- Database access permissions are properly configured in `pg_hba.conf`.
- [CloudDM (v4.1.0 or higher)](https://github.com/ClouGence/open-cdm) is installed and accessible.

## Steps

1. Log in to the CloudDM console, and click **Back to Workspace**.

2. In the left panel, click **Instances**, and then click **New** to add a database connection.
![](../../media/clouddm1.png)

3. Select Cloudberry and click **Submit**.
![](../../media/clouddm2.png)

4. Enter the Cloudberry connection information:
- **Host**: Enter the hostname or IP address of your Cloudberry instance.
- **Port**: Enter the port.
- **Authentication**: Select an authentication method and enter the required information (for example, username and password)

![](../../media/clouddm3.png)

5. Test the connection. Once Cloudberry is connected to CloudDM, click **Add DataSource**.

If the test fails, check the Cloudberry address, port, database user, password, network access, and `pg_hba.conf` rules.

6. Go to **SQL Query**.
![](../../media/clouddm4.png)

7. The added Cloudberry instance is shown in the left panel. Now you can start running a query.
![](../../media/clouddm5.png)

## Other features

### Work orders

CloudDM supports work orders for production database changes. Teams can submit, approve, and execute changes through a controlled workflow.
![](../../media/clouddm6.png)

### CI/CD

CloudDM can trigger database CI/CD workflows through Git Push, Web Hook, and HttpCall. Teams can include SQL checks, approvals, and execution in application release processes, and manage database change scripts in one place.
![](../../media/clouddm7.png)

### User access control

CloudDM supports role-based access control (RBAC) for team collaboration. Teams can grant access by role and control which data sources and operations each user can use.
![](../../media/clouddm8.png)
Binary file added versioned_docs/version-2.x/media/bladepipe1.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added versioned_docs/version-2.x/media/bladepipe2.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added versioned_docs/version-2.x/media/bladepipe3.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added versioned_docs/version-2.x/media/bladepipe4.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added versioned_docs/version-2.x/media/bladepipe5.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added versioned_docs/version-2.x/media/bladepipe6.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added versioned_docs/version-2.x/media/bladepipe8.png
Binary file added versioned_docs/version-2.x/media/bladepipe9.png
Binary file added versioned_docs/version-2.x/media/clouddm1.png
Binary file added versioned_docs/version-2.x/media/clouddm2.png
Binary file added versioned_docs/version-2.x/media/clouddm3.png
Binary file added versioned_docs/version-2.x/media/clouddm4.png
Binary file added versioned_docs/version-2.x/media/clouddm5.png
Binary file added versioned_docs/version-2.x/media/clouddm6.png
Binary file added versioned_docs/version-2.x/media/clouddm7.png
Binary file added versioned_docs/version-2.x/media/clouddm8.png
8 changes: 5 additions & 3 deletions versioned_sidebars/version-2.x-sidebars.json
Original file line number Diff line number Diff line change
Expand Up @@ -372,14 +372,16 @@
"type": "category",
"label": "SQL Clients",
"items": [
"ecosystem/sql-clients/dbeaver"
"ecosystem/sql-clients/dbeaver",
"ecosystem/sql-clients/clouddm"
]
},
{
"type": "category",
"label": "Data Integration",
"items": [
"ecosystem/data-integration/seatunnel"
"ecosystem/data-integration/seatunnel",
"ecosystem/data-integration/bladepipe"
]
},
{
Expand Down Expand Up @@ -819,4 +821,4 @@
"href": "/pxf/"
}
]
}
}
Loading