Skip to content

[fix](fe) Reject path partition keys for http_stream - #67409

Open
0AyanamiRei wants to merge 1 commit into
apache:masterfrom
0AyanamiRei:fix/doris-25117-reject-http-stream-path-partitions
Open

[fix](fe) Reject path partition keys for http_stream#67409
0AyanamiRei wants to merge 1 commit into
apache:masterfrom
0AyanamiRei:fix/doris-25117-reject-http-stream-path-partitions

Conversation

@0AyanamiRei

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: http_stream accepted path_partition_keys even though streamed request bodies have no file path from which to derive partition values. The invalid property reached backend execution and failed after planning. Reject it while constructing the table-valued function so clients receive a clear analysis error before a transaction is created.

Release note

http_stream now rejects the unsupported path_partition_keys property during analysis.

Check List (For Author)

  • Test: FE unit test, regression test, and manual HTTP stream test
    • FE unit test: HttpStreamTableValuedFunctionTest
    • Regression test: test_http_stream_path_partition_keys
    • Manual test: control load succeeded with 10 rows; unsupported property returned ANALYSIS_ERROR
  • Behavior changed: Yes. An unsupported property is now rejected before execution.
  • Does this need documentation: No

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: http_stream accepted path_partition_keys even though streamed request bodies have no file path from which to derive partition values. The invalid property reached backend execution and failed after planning. Reject it while constructing the table-valued function so clients receive a clear analysis error before a transaction is created.

### Release note

http_stream now rejects the unsupported path_partition_keys property during analysis.

### Check List (For Author)

- Test: FE unit test, regression test, and manual HTTP stream test
    - FE unit test: HttpStreamTableValuedFunctionTest
    - Regression test: test_http_stream_path_partition_keys
    - Manual test: control load succeeded with 10 rows; unsupported property returned ANALYSIS_ERROR
- Behavior changed: Yes. An unsupported property is now rejected before execution.
- Does this need documentation: No
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@0AyanamiRei

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 16834 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6e0303170a75d3e13062dbc0890a0921ac863392, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17581	3061	3054	3054
q2	2150	260	226	226
q3	10181	874	516	516
q4	4672	251	208	208
q5	7671	579	384	384
q6	136	117	97	97
q7	526	499	390	390
q8	9243	820	853	820
q9	3454	2385	2386	2385
q10	6503	857	735	735
q11	395	198	178	178
q12	615	283	203	203
q13	18095	1533	1156	1156
q14	162	156	142	142
q15	q16	426	402	370	370
q17	1404	923	807	807
q18	3134	2252	2268	2252
q19	1265	879	741	741
q20	381	293	204	204
q21	5592	1735	1841	1735
q22	338	271	231	231
Total cold run time: 93924 ms
Total hot run time: 16834 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3388	3348	3295	3295
q2	504	392	372	372
q3	2220	2278	2127	2127
q4	1195	1192	899	899
q5	2186	2113	2109	2109
q6	168	122	88	88
q7	1036	950	870	870
q8	1577	1383	1405	1383
q9	3136	3084	3096	3084
q10	1858	1803	1631	1631
q11	356	271	260	260
q12	456	422	350	350
q13	1461	1546	1158	1158
q14	172	165	168	165
q15	q16	392	399	356	356
q17	3580	3326	3213	3213
q18	4823	4406	4745	4406
q19	838	859	862	859
q20	1019	956	832	832
q21	3798	3096	3285	3096
q22	387	353	313	313
Total cold run time: 34550 ms
Total hot run time: 30866 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 82329 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6e0303170a75d3e13062dbc0890a0921ac863392, data reload: false

query5	4260	407	326	326
query6	401	139	123	123
query7	4954	434	237	237
query8	312	124	129	124
query9	8693	2882	2889	2882
query10	402	219	183	183
query11	5380	1022	903	903
query12	113	67	68	67
query13	1187	440	339	339
query14	6082	2221	2113	2113
query14_1	1999	1994	1996	1994
query15	177	121	113	113
query16	922	374	355	355
query17	806	460	377	377
query18	2343	332	255	255
query19	168	141	109	109
query20	75	70	73	70
query21	208	103	88	88
query22	5411	5402	5406	5402
query23	6782	6193	6002	6002
query23_1	6100	6089	6045	6045
query24	7237	1103	785	785
query24_1	813	778	782	778
query25	413	292	254	254
query26	1231	226	125	125
query27	2788	377	263	263
query28	4704	1494	1484	1484
query29	911	424	340	340
query30	253	154	131	131
query31	809	408	334	334
query32	127	74	79	74
query33	450	218	176	176
query34	1004	819	482	482
query35	410	399	347	347
query36	570	587	530	530
query37	121	82	70	70
query38	1002	867	852	852
query39	486	484	482	482
query39_1	453	460	495	460
query40	204	91	79	79
query41	53	55	51	51
query42	70	71	71	71
query43	240	244	216	216
query44	1016	544	553	544
query45	103	103	97	97
query46	776	858	528	528
query47	769	779	721	721
query48	307	321	227	227
query49	537	240	180	180
query50	787	264	195	195
query51	8127	8049	8109	8049
query52	70	73	60	60
query53	203	196	151	151
query54	232	192	169	169
query55	82	60	56	56
query56	211	189	277	189
query57	683	630	646	630
query58	212	174	163	163
query59	1243	1252	1103	1103
query60	271	193	194	193
query61	144	134	143	134
query62	354	207	181	181
query63	170	149	144	144
query64	2784	707	607	607
query65	1632	1613	1572	1572
query66	1882	262	219	219
query67	10323	9595	9652	9595
query68	2986	1122	745	745
query69	364	222	211	211
query70	656	631	620	620
query71	258	176	170	170
query72	2358	1764	1590	1590
query73	656	610	352	352
query74	1996	1229	1134	1134
query75	1190	1104	970	970
query76	2385	734	555	555
query77	262	263	217	217
query78	3953	3676	3131	3131
query79	2837	817	593	593
query80	1608	335	289	289
query81	513	157	132	132
query82	635	124	96	96
query83	292	208	185	185
query84	304	112	91	91
query85	812	390	288	288
query86	472	178	169	169
query87	1007	971	886	886
query88	2828	2117	2093	2093
query89	278	197	177	177
query90	2074	129	130	129
query91	142	121	98	98
query92	93	72	72	72
query93	1519	1073	723	723
query94	640	251	224	224
query95	526	255	235	235
query96	800	571	289	289
query97	1019	1065	1030	1030
query98	175	142	135	135
query99	423	344	319	319
Total cold run time: 179518 ms
Total hot run time: 82329 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 14.54 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6e0303170a75d3e13062dbc0890a0921ac863392, data reload: false

query1	0.01	0.00	0.01
query2	0.09	0.04	0.04
query3	0.25	0.12	0.10
query4	1.61	0.11	0.09
query5	0.18	0.15	0.16
query6	1.21	0.67	0.70
query7	0.04	0.01	0.01
query8	0.04	0.03	0.02
query9	0.28	0.21	0.21
query10	0.34	0.34	0.37
query11	0.17	0.12	0.11
query12	0.14	0.12	0.12
query13	0.31	0.30	0.31
query14	0.45	0.45	0.45
query15	0.37	0.35	0.34
query16	0.24	0.23	0.24
query17	0.68	0.64	0.63
query18	0.18	0.15	0.15
query19	1.15	1.11	1.18
query20	0.04	0.01	0.01
query21	15.44	0.16	0.12
query22	5.08	0.04	0.05
query23	16.14	0.26	0.10
query24	3.00	0.33	0.27
query25	0.11	0.04	0.03
query26	0.74	0.17	0.12
query27	0.05	0.02	0.03
query28	3.65	0.58	0.29
query29	12.42	3.16	2.53
query30	0.25	0.12	0.13
query31	2.75	0.38	0.18
query32	3.50	0.33	0.23
query33	1.44	1.50	1.40
query34	15.37	2.19	1.76
query35	1.74	1.74	1.71
query36	0.46	0.29	0.29
query37	0.06	0.04	0.04
query38	0.04	0.03	0.03
query39	0.03	0.02	0.02
query40	0.11	0.08	0.07
query41	0.08	0.03	0.03
query42	0.03	0.03	0.03
query43	0.03	0.03	0.03
Total cold run time: 90.3 s
Total hot run time: 14.54 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants