[Feature] why partitioned mast be in table filelds?cat not cast create_time as Date

[zsl461975543]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Paimon version
1.4.2, master

Compute Engine
Flink 1.20, paimon-kafka-connect

Minimal reproduce step:
step1: Use Debezium MySQL Connector to write customer Debezium JSON records to Kafka.

step2: then user paimon-kafka-connect to write json message to paimon.

I attempted to use create_time (formatted as yyyy-MM-dd HH:mm:ss) as a partitioning column, but that failed. Subsequently, I used PARTITIONED BY (to_date(create_time FROM 1 FOR 10)), which resulted in an error stating that the expression must be based on table columns. Is it possible to support partitioning rules like those in Doris (AUTO PARTITION BY RANGE (date_trunc(create_time, 'day'))) ?

Why do we use Kafka instead of Flink CDC? Because other business services need to reuse these JSON messages.

JSON Records to Kafka.eg:

{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"order_id"},{"type":"string","optional":true,"default":"","field":"out_order_id"},{"type":"string","optional":true,"default":"","field":"pay_order_id"},{"type":"int16","optional":true,"field":"order_status"},{"type":"string","optional":true,"default":"","field":"result_detail"},{"type":"int32","optional":false,"default":0,"field":"game_id"},{"type":"string","optional":true,"field":"game_name"},{"type":"int32","optional":true,"default":0,"field":"game_type"},{"type":"int64","optional":false,"field":"user_id"},{"type":"string","optional":true,"field":"user_account"},{"type":"int64","optional":true,"field":"account_code"},{"type":"string","optional":true,"default":"","field":"device_code"},{"type":"int16","optional":true,"default":0,"field":"platform"},{"type":"int64","optional":false,"default":0,"field":"payable_amount"},{"type":"int64","optional":false,"default":0,"field":"receipts_amount"},{"type":"int64","optional":false,"default":0,"field":"discount_amount"},{"type":"int64","optional":true,"default":0,"field":"recovery_amount"},{"type":"string","optional":true,"default":"","field":"coupon_type"},{"type":"int64","optional":true,"default":0,"field":"excise_tax"},{"type":"string","optional":true,"default":"","field":"currency"},{"type":"int64","optional":true,"field":"coupon_id"},{"type":"string","optional":true,"default":"","field":"bet_detail"},{"type":"string","optional":true,"field":"description"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"cancel_time"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"create_time"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"update_time"},{"type":"int64","optional":true,"field":"agent_code"},{"type":"string","optional":true,"field":"agent_account"},{"type":"string","optional":true,"field":"play_method"},{"type":"string","optional":true,"field":"number_detail"},{"type":"int64","optional":true,"field":"develop_id"},{"type":"string","optional":true,"default":"","field":"bill_date"},{"type":"int32","optional":true,"default":0,"field":"cancel_cash_times"},{"type":"string","optional":true,"default":"0","field":"issue_num"},{"type":"double","optional":true,"default":0.0,"field":"min_odds"},{"type":"string","optional":true,"default":"","field":"register_from"},{"type":"string","optional":true,"default":"","field":"login_channel"},{"type":"int64","optional":true,"default":0,"field":"main_amount"},{"type":"int64","optional":true,"default":0,"field":"redpackets_amount"},{"type":"int64","optional":true,"default":0,"field":"activity_amount"},{"type":"int64","optional":true,"default":0,"field":"activity_id"},{"type":"string","optional":true,"default":"0","field":"redpackets_detail"},{"type":"string","optional":true,"default":"0","field":"activity_type"},{"type":"int16","optional":true,"default":0,"field":"settle_status"},{"type":"int64","optional":true,"default":0,"field":"settle_amount"},{"type":"int64","optional":true,"default":0,"field":"wh_tax"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"settle_time"},{"type":"string","optional":false,"field":"db_name"},{"type":"string","optional":false,"field":"table_name"},{"type":"int32","optional":false,"field":"dml_type"}],"optional":false,"name":"ng_g_game_deal_order.Value"},"payload":{"order_id":"8773639782452027392102012759","out_order_id":"t8773639782411239424","pay_order_id":"1020127598773639782501179392","order_status":1,"result_detail":"","game_id":60015,"game_name":"Gold Hunter","game_type":2,"user_id":102012759,"user_account":"1730000255","account_code":10201275902416582,"device_code":"3bd7151f-0bf3-404f-bf02-119c749645ec","platform":3,"payable_amount":5000000,"receipts_amount":5000000,"discount_amount":0,"recovery_amount":0,"coupon_type":"","excise_tax":0,"currency":"NGN","coupon_id":null,"bet_detail":"","description":null,"cancel_time":null,"create_time":1786008944000,"update_time":null,"agent_code":null,"agent_account":null,"play_method":null,"number_detail":"1;1;5000000;|","develop_id":10,"bill_date":"20260806","cancel_cash_times":0,"issue_num":"0","min_odds":0.0,"register_from":"ilot","login_channel":"3","main_amount":5000000,"redpackets_amount":0,"activity_amount":0,"activity_id":0,"redpackets_detail":"0","activity_type":"0","settle_status":0,"settle_amount":0,"wh_tax":0,"settle_time":null,"db_name":"ilot_trade1","table_name":"g_game_deal_order_0","dml_type":1}}
{"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"order_id"},{"type":"string","optional":true,"default":"","field":"out_order_id"},{"type":"string","optional":true,"default":"","field":"pay_order_id"},{"type":"int16","optional":true,"field":"order_status"},{"type":"string","optional":true,"default":"","field":"result_detail"},{"type":"int32","optional":false,"default":0,"field":"game_id"},{"type":"string","optional":true,"field":"game_name"},{"type":"int32","optional":true,"default":0,"field":"game_type"},{"type":"int64","optional":false,"field":"user_id"},{"type":"string","optional":true,"field":"user_account"},{"type":"int64","optional":true,"field":"account_code"},{"type":"string","optional":true,"default":"","field":"device_code"},{"type":"int16","optional":true,"default":0,"field":"platform"},{"type":"int64","optional":false,"default":0,"field":"payable_amount"},{"type":"int64","optional":false,"default":0,"field":"receipts_amount"},{"type":"int64","optional":false,"default":0,"field":"discount_amount"},{"type":"int64","optional":true,"default":0,"field":"recovery_amount"},{"type":"string","optional":true,"default":"","field":"coupon_type"},{"type":"int64","optional":true,"default":0,"field":"excise_tax"},{"type":"string","optional":true,"default":"","field":"currency"},{"type":"int64","optional":true,"field":"coupon_id"},{"type":"string","optional":true,"default":"","field":"bet_detail"},{"type":"string","optional":true,"field":"description"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"cancel_time"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"create_time"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"update_time"},{"type":"int64","optional":true,"field":"agent_code"},{"type":"string","optional":true,"field":"agent_account"},{"type":"string","optional":true,"field":"play_method"},{"type":"string","optional":true,"field":"number_detail"},{"type":"int64","optional":true,"field":"develop_id"},{"type":"string","optional":true,"default":"","field":"bill_date"},{"type":"int32","optional":true,"default":0,"field":"cancel_cash_times"},{"type":"string","optional":true,"default":"0","field":"issue_num"},{"type":"double","optional":true,"default":0.0,"field":"min_odds"},{"type":"string","optional":true,"default":"","field":"register_from"},{"type":"string","optional":true,"default":"","field":"login_channel"},{"type":"int64","optional":true,"default":0,"field":"main_amount"},{"type":"int64","optional":true,"default":0,"field":"redpackets_amount"},{"type":"int64","optional":true,"default":0,"field":"activity_amount"},{"type":"int64","optional":true,"default":0,"field":"activity_id"},{"type":"string","optional":true,"default":"0","field":"redpackets_detail"},{"type":"string","optional":true,"default":"0","field":"activity_type"},{"type":"int16","optional":true,"default":0,"field":"settle_status"},{"type":"int64","optional":true,"default":0,"field":"settle_amount"},{"type":"int64","optional":true,"default":0,"field":"wh_tax"},{"type":"int64","optional":true,"name":"io.debezium.time.Timestamp","version":1,"field":"settle_time"},{"type":"string","optional":false,"field":"db_name"},{"type":"string","optional":false,"field":"table_name"},{"type":"int32","optional":false,"field":"dml_type"}],"optional":false,"name":"ng_g_game_deal_order.Value"},"payload":{"order_id":"8773639782452027392102012759","out_order_id":"t8773639782411239424","pay_order_id":"1020127598773639782501179392","order_status":1,"result_detail":"","game_id":60015,"game_name":"Gold Hunter","game_type":2,"user_id":102012759,"user_account":"1730000255","account_code":10201275902416582,"device_code":"3bd7151f-0bf3-404f-bf02-119c749645ec","platform":3,"payable_amount":5000000,"receipts_amount":5000000,"discount_amount":0,"recovery_amount":0,"coupon_type":"","excise_tax":0,"currency":"NGN","coupon_id":null,"bet_detail":"","description":null,"cancel_time":null,"create_time":1786008944000,"update_time":1786008956000,"agent_code":null,"agent_account":null,"play_method":null,"number_detail":"1;1;5000000;|","develop_id":10,"bill_date":"20260806","cancel_cash_times":0,"issue_num":"0","min_odds":0.0,"register_from":"ilot","login_channel":"3","main_amount":5000000,"redpackets_amount":0,"activity_amount":0,"activity_id":0,"redpackets_detail":"0","activity_type":"0","settle_status":3,"settle_amount":7500000,"wh_tax":0,"settle_time":1786008956000,"db_name":"ilot_trade1","table_name":"g_game_deal_order_0","dml_type":2}}

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!