-
Notifications
You must be signed in to change notification settings - Fork 120
Add seatunnel integration doc #464
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
zhupitertop
wants to merge
23
commits into
apache:master
Choose a base branch
from
hugegraph:add-seatunnel-integration-doc
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
23 commits
Select commit
Hold shift + click to select a range
62baf2a
chore: update release workflow and .gitignore entries
imbajin 5bb8ee0
refactor: unify release validation script and add local path support
imbajin e4e9e08
refactor: revamp release validation script with enhanced checks V2
imbajin 1ab58f1
refactor: enhance release validation for multi-arch and license check…
imbajin ba74766
refactor: improve validation script error context and reporting V4
imbajin af87f8a
fix: add JSON to CATEGORY_X license validation
imbajin 512b51c
introduce new version of validation
Pengzna c4107b6
Update GPG username and regex in workflow
imbajin c773933
feat: add 1.7.0 release-notes (#429)
Pengzna dff6ba1
chore: add docker using / add load data methods (#423)
Ethereal-O ea90243
chore: minor test (#430)
Ethereal-O e4660a6
docs: delete licsence/add link of twitter (#431)
Ethereal-O 34bfab5
chore: add deepwiki notes in docs (#432)
Ethereal-O 13b6d1f
Merge pull request #24 from apache/master
imbajin 5840b8b
feat: add graphspace related doc (#425)
Pengzna 966a459
Merge branch 'apache:master' into master
imbajin b55fe2b
feat: 1.7.0 download-links (#434)
Pengzna d533820
docs: refactor docs of loader & client for new version(1.7.0) (#415)
sadwitdastreetz 90f21e5
Merge pull request #29 from apache/master
imbajin 51e334f
Merge pull request #30 from apache/master
imbajin 4ae78ff
@
spr1lng 68a004a
@
spr1lng 7851a25
@
spr1lng File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+1.22 MB
content/cn/docs/images/seatunnel/hugegraph-seatunnel-architecture.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
146 changes: 146 additions & 0 deletions
146
content/cn/docs/quickstart/toolchain/hugegraph-seatunnel-connector.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,146 @@ | ||
| --- | ||
| title: "HugeGraph-SeaTunnel Connector Quick Start" | ||
| linkTitle: "使用 SeaTunnel Connector 同步数据" | ||
| weight: 5 | ||
| --- | ||
|
|
||
| ### 1 HugeGraph-SeaTunnel Connector 概述 | ||
|
|
||
| [Apache SeaTunnel](https://seatunnel.apache.org/) 是一个高性能、分布式的数据集成平台,支持海量数据的实时同步与批处理。 | ||
| HugeGraph 社区已向 Apache SeaTunnel 贡献了 Connector-V2 支持,用户可以通过 SeaTunnel 将 HugeGraph 与外部数据系统进行数据同步。 | ||
|
|
||
|  | ||
|
|
||
| ### 2 功能特性 | ||
|
|
||
| HugeGraph-SeaTunnel Connector 提供以下能力: | ||
|
|
||
| - **HugeGraph Sink Connector**(已发布):将外部数据写入 HugeGraph,支持顶点和边的批量写入、更新与删除 | ||
| - **HugeGraph Source Connector**(开发中):从 HugeGraph 读取图数据,支持顶点和边的批量读取 | ||
| - **顶点同步**:支持全量或增量同步顶点数据,支持多种 ID 策略 | ||
| - **边同步**:支持全量或增量同步边数据,自动关联源顶点与目标顶点 | ||
| - **Schema 自动管理**:支持自动创建 PropertyKey、VertexLabel、EdgeLabel(`CREATE_SCHEMA_WHEN_NOT_EXIST`) | ||
| - **数据迁移**:支持在不同 HugeGraph 实例之间迁移数据,或从其他数据源导入图数据 | ||
|
|
||
|  | ||
|
|
||
| ### 3 环境要求 | ||
|
|
||
| | 组件 | 版本要求 | 说明 | | ||
| |------|---------|------| | ||
| | Java | 8+ | SeaTunnel 运行环境要求 | | ||
| | Apache SeaTunnel | 2.3.12+ | Sink Connector 自此版本发布 | | ||
| | HugeGraph Server | 1.0.0+ | 建议使用最新稳定版 | | ||
|
|
||
| > **Source Connector 说明**:HugeGraph Source Connector 目前仅在 SeaTunnel 开发分支(Next/master)中可用, | ||
| > 尚未包含在正式 Release 中。如需使用 Source 功能,请从 SeaTunnel 源码编译或等待下一版本发布。 | ||
|
|
||
| ### 4 快速开始 | ||
|
|
||
| #### 4.1 安装 SeaTunnel | ||
|
|
||
| 请参考 [Apache SeaTunnel 部署指南](https://seatunnel.apache.org/docs/getting-started/) 完成 SeaTunnel 的安装部署。 | ||
|
|
||
| #### 4.2 使用 HugeGraph Sink Connector(写入数据) | ||
|
|
||
| 以下示例展示如何通过 SeaTunnel 将数据写入 HugeGraph: | ||
|
|
||
| **写入顶点数据:** | ||
|
|
||
| ```hocon | ||
| env { | ||
| job.mode = "BATCH" | ||
| } | ||
|
|
||
| source { | ||
| FakeSource { | ||
| plugin_output = "fake" | ||
| schema = { | ||
| fields = { | ||
| name = "string" | ||
| age = "int" | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| sink { | ||
| HugeGraph { | ||
| host = "127.0.0.1" | ||
| port = 8080 | ||
| graph_name = "hugegraph" | ||
| mappings = [ | ||
| { | ||
| type = "VERTEX" | ||
| label = "person" | ||
| idStrategy = "PRIMARY_KEY" | ||
| idFields = ["name"] | ||
| properties = ["name", "age"] | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| **写入边数据:** | ||
|
|
||
| ```hocon | ||
| sink { | ||
| HugeGraph { | ||
| host = "127.0.0.1" | ||
| port = 8080 | ||
| graph_name = "hugegraph" | ||
| mappings = [ | ||
| { | ||
| type = "EDGE" | ||
| label = "knows" | ||
| sourceConfig = { | ||
| label = "person" | ||
| idFields = ["person1_name"] | ||
| } | ||
| targetConfig = { | ||
| label = "person" | ||
| idFields = ["person2_name"] | ||
| } | ||
| properties = ["since"] | ||
| fieldMapping = { | ||
| person1_name = "name" | ||
| person2_name = "name" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| > **注意**:HugeGraph 的 Schema(PropertyKey、VertexLabel、EdgeLabel)需要在执行写入任务前预先创建, | ||
| > 或使用 `schema_save_mode = CREATE_SCHEMA_WHEN_NOT_EXIST`(默认行为)由 Connector 自动创建。 | ||
|
|
||
| #### 4.3 核心配置参数 | ||
|
|
||
| | 参数 | 类型 | 必填 | 默认值 | 说明 | | ||
| |------|------|------|--------|------| | ||
| | `host` | String | 是 | - | HugeGraph Server 地址 | | ||
| | `port` | Integer | 是 | - | HugeGraph Server 端口 | | ||
| | `graph_name` | String | 是 | - | 图名称 | | ||
| | `graph_space` | String | 否 | DEFAULT | 图空间名称 | | ||
| | `username` | String | 否 | - | 认证用户名 | | ||
| | `password` | String | 否 | - | 认证密码 | | ||
| | `protocol` | String | 否 | http | 协议,可选 `http` 或 `https` | | ||
| | `batch_size` | Integer | 否 | 500 | 每批写入的记录数 | | ||
| | `batch_interval_ms` | Integer | 否 | 5000 | 批次最大等待时间(毫秒) | | ||
| | `max_retries` | Integer | 否 | 3 | 写入失败最大重试次数 | | ||
| | `retry_backoff_ms` | Integer | 否 | 5000 | 重试间隔(毫秒) | | ||
|
|
||
| > 完整参数列表和详细说明请参考 [Apache SeaTunnel HugeGraph Connector 官方文档](https://seatunnel.apache.org/docs/connector-v2/sink/HugeGraph/)。 | ||
|
|
||
| ### 5 文档与资源 | ||
|
|
||
| - [Apache SeaTunnel 官方网站](https://seatunnel.apache.org/) | ||
| - [Apache SeaTunnel Connector-V2 文档](https://seatunnel.apache.org/docs/connector-v2/) | ||
| - [Apache SeaTunnel GitHub](https://github.com/apache/seatunnel) | ||
| - [HugeGraph GitHub](https://github.com/apache/hugegraph) | ||
|
|
||
| ### 6 许可证 | ||
|
|
||
| 与 HugeGraph 一样,HugeGraph-SeaTunnel Connector 采用 Apache 2.0 许可证。 |
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added
BIN
+1.22 MB
content/en/docs/images/seatunnel/hugegraph-seatunnel-architecture.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
149 changes: 149 additions & 0 deletions
149
content/en/docs/quickstart/toolchain/hugegraph-seatunnel-connector.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,149 @@ | ||
| --- | ||
| title: "HugeGraph-SeaTunnel Connector Quick Start" | ||
| linkTitle: "Sync Data with SeaTunnel Connector" | ||
| weight: 5 | ||
| --- | ||
|
|
||
| ### 1 HugeGraph-SeaTunnel Connector Overview | ||
|
|
||
| [Apache SeaTunnel](https://seatunnel.apache.org/) is a high-performance, distributed data integration platform | ||
| that supports real-time synchronization and batch processing of massive data. The HugeGraph community has contributed | ||
| Connector-V2 support to Apache SeaTunnel, enabling users to synchronize data between HugeGraph and external data systems. | ||
|
|
||
|  | ||
|
|
||
| ### 2 Features | ||
|
|
||
| HugeGraph-SeaTunnel Connector provides the following capabilities: | ||
|
|
||
| - **HugeGraph Sink Connector** (Released): Write external data to HugeGraph, supporting batch write, update, and delete of vertices and edges | ||
| - **HugeGraph Source Connector** (In Development): Read graph data from HugeGraph, supporting batch reading of vertices and edges | ||
| - **Vertex Synchronization**: Support full or incremental vertex data synchronization with multiple ID strategies | ||
| - **Edge Synchronization**: Support full or incremental edge data synchronization, auto-resolving source and target vertices | ||
| - **Automatic Schema Management**: Support auto-creating PropertyKey, VertexLabel, EdgeLabel (`CREATE_SCHEMA_WHEN_NOT_EXIST`) | ||
| - **Data Migration**: Support data migration between different HugeGraph instances, or importing graph data from other data sources | ||
|
|
||
|  | ||
|
|
||
| ### 3 Environment Requirements | ||
|
|
||
| | Component | Version | Notes | | ||
| |-----------|---------|-------| | ||
| | Java | 8+ | SeaTunnel runtime requirement | | ||
| | Apache SeaTunnel | 2.3.12+ | Sink Connector released since this version | | ||
| | HugeGraph Server | 1.0.0+ | Latest stable version recommended | | ||
|
|
||
| > **Source Connector Note**: The HugeGraph Source Connector is currently only available in the SeaTunnel development | ||
| > branch (Next/master) and has not been included in an official release. If you need the Source feature, please build | ||
| > SeaTunnel from source or wait for the next release. | ||
|
|
||
| ### 4 Quick Start | ||
|
|
||
| #### 4.1 Install SeaTunnel | ||
|
|
||
| Please refer to the [Apache SeaTunnel Deployment Guide](https://seatunnel.apache.org/docs/getting-started/) to install and deploy SeaTunnel. | ||
|
|
||
| #### 4.2 Use HugeGraph Sink Connector (Write Data) | ||
|
|
||
| The following examples demonstrate how to write data to HugeGraph via SeaTunnel: | ||
|
|
||
| **Write vertices:** | ||
|
|
||
| ```hocon | ||
| env { | ||
| job.mode = "BATCH" | ||
| } | ||
|
|
||
| source { | ||
| FakeSource { | ||
| plugin_output = "fake" | ||
| schema = { | ||
| fields = { | ||
| name = "string" | ||
| age = "int" | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| sink { | ||
| HugeGraph { | ||
| host = "127.0.0.1" | ||
| port = 8080 | ||
| graph_name = "hugegraph" | ||
| mappings = [ | ||
| { | ||
| type = "VERTEX" | ||
| label = "person" | ||
| idStrategy = "PRIMARY_KEY" | ||
| idFields = ["name"] | ||
| properties = ["name", "age"] | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| **Write edges:** | ||
|
|
||
| ```hocon | ||
| sink { | ||
| HugeGraph { | ||
| host = "127.0.0.1" | ||
| port = 8080 | ||
| graph_name = "hugegraph" | ||
| mappings = [ | ||
| { | ||
| type = "EDGE" | ||
| label = "knows" | ||
| sourceConfig = { | ||
| label = "person" | ||
| idFields = ["person1_name"] | ||
| } | ||
| targetConfig = { | ||
| label = "person" | ||
| idFields = ["person2_name"] | ||
| } | ||
| properties = ["since"] | ||
| fieldMapping = { | ||
| person1_name = "name" | ||
| person2_name = "name" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| > **Note**: The HugeGraph schema (PropertyKey, VertexLabel, EdgeLabel) must be created before executing write tasks, | ||
| > or you can use `schema_save_mode = CREATE_SCHEMA_WHEN_NOT_EXIST` (default behavior) for the Connector to auto-create it. | ||
|
|
||
| #### 4.3 Core Configuration Parameters | ||
|
|
||
| | Parameter | Type | Required | Default | Description | | ||
| |-----------|------|----------|---------|-------------| | ||
| | `host` | String | Yes | - | HugeGraph Server host | | ||
| | `port` | Integer | Yes | - | HugeGraph Server port | | ||
| | `graph_name` | String | Yes | - | Graph name | | ||
| | `graph_space` | String | No | DEFAULT | Graph space name | | ||
| | `username` | String | No | - | Authentication username | | ||
| | `password` | String | No | - | Authentication password | | ||
| | `protocol` | String | No | http | Protocol, `http` or `https` | | ||
| | `batch_size` | Integer | No | 500 | Records per batch write | | ||
| | `batch_interval_ms` | Integer | No | 5000 | Max wait time per batch (ms) | | ||
| | `max_retries` | Integer | No | 3 | Max retries on write failure | | ||
| | `retry_backoff_ms` | Integer | No | 5000 | Retry backoff interval (ms) | | ||
|
|
||
| > For the complete parameter list and detailed documentation, please refer to the | ||
| > [Apache SeaTunnel HugeGraph Connector Official Docs](https://seatunnel.apache.org/docs/connector-v2/sink/HugeGraph/). | ||
|
|
||
| ### 5 Documentation & Resources | ||
|
|
||
| - [Apache SeaTunnel Official Website](https://seatunnel.apache.org/) | ||
| - [Apache SeaTunnel Connector-V2 Documentation](https://seatunnel.apache.org/docs/connector-v2/) | ||
| - [Apache SeaTunnel GitHub](https://github.com/apache/seatunnel) | ||
| - [HugeGraph GitHub](https://github.com/apache/hugegraph) | ||
|
|
||
| ### 6 License | ||
|
|
||
| Same as HugeGraph, HugeGraph-SeaTunnel Connector is licensed under Apache 2.0. |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
content/cnandcontent/endocumentation. Add the corresponding English page andcontent/en/docs/introduction/_index.mdentry so the integration is discoverable on both sites.