Skip to content

[Fix](user-property) Preserve legacy user properties during desrialization#65781

Open
linrrzqqq wants to merge 1 commit into
apache:masterfrom
linrrzqqq:fix-user-property
Open

[Fix](user-property) Preserve legacy user properties during desrialization#65781
linrrzqqq wants to merge 1 commit into
apache:masterfrom
linrrzqqq:fix-user-property

Conversation

@linrrzqqq

Copy link
Copy Markdown
Collaborator

Related PR: #37243

Problem Summary:

(Before doris-3.0)Older persisted CommonUserProperties JSON uses long field names such as resourceTags and sqlBlockRules. The current serialized names are rt and sbr, while the compatibility aliases only covered the singular forms resourceTag and sqlBlockRule.

As a result, legacy user properties may be silently ignored during metadata
deserialization, causing resource tags or SQL block rules to be reset to their
default values.

This change adds compatibility aliases for the legacy plural field names:

  • resourceTags
  • sqlBlockRules

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

/review

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review result: changes requested.

The production compatibility fix is correct and focused: the added plural aliases exactly match the historical image fields, the nested Gson post-process rebuilds sqlBlockRulesSplit, canonical writes remain on the short names, and the edit-log path is unaffected. One required validation issue remains in the added test.

Blocking finding:

  • The legacy JSON fixture fails the mandatory FE Checkstyle gate with 13 OperatorWrap violations; see the inline comment.

Critical checkpoint conclusions:

  • Goal/correctness and proof: the aliases restore both legacy resource tags and SQL block rules through the production image-read mechanism. The unit assertions cover canonical, singular alternate, plural legacy, and SQL-rule split reconstruction, but the test source must first pass style validation.
  • Scope and reuse: the production change is minimal, clear, and uses Gson's existing compatibility mechanism; no broader refactor is needed.
  • Concurrency and lifecycle: no new concurrency, locks, shared mutable state, or non-intuitive lifecycle is introduced. The existing image-load lifecycle and nested post-process adapter remain unchanged.
  • Configuration, compatibility, and parallel paths: no configuration or FE-BE protocol is changed. Historical plural and current short/singular names are accepted; canonical serialization is unchanged. Image loading is covered, while property edit-log replay remains an unaffected property-pair path.
  • Conditions and error handling: no new conditional or error-handling path is introduced.
  • Tests and results: the new JUnit 4 tests semantically target the changed behavior and no regression .out result is involved. Current live Checkstyle fails on the new concatenation formatting, so validation is not green.
  • Observability: no new runtime branch or distributed operation requires logs, metrics, or tracing.
  • Persistence, transactions, and data writes: this is FE metadata image compatibility only; it changes neither transaction semantics nor edit-log payloads and introduces no data-write path.
  • Performance: alias lookup is confined to deserialization and adds no meaningful hot-path cost or redundant work.
  • Other issues: no additional substantiated correctness, compatibility, lifecycle, or coverage defect remains.

User focus: no additional focus points were supplied; the full PR and persistence boundary were reviewed.

Validation note: per the review-runner contract, no local build or source modification was attempted. The live Checkstyle log for the reviewed head is the authoritative failing evidence.

@linrrzqqq
linrrzqqq force-pushed the fix-user-property branch from e5128f0 to 24d1268 Compare July 18, 2026 06:56
@linrrzqqq

Copy link
Copy Markdown
Collaborator Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29241 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 24d1268559ebe6a9cb857ff198f9456c15ec2a56, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17633	3991	4033	3991
q2	2013	322	197	197
q3	10271	1383	795	795
q4	4717	467	336	336
q5	7595	837	572	572
q6	207	165	132	132
q7	758	814	600	600
q8	10122	1620	1517	1517
q9	6145	4292	4335	4292
q10	6688	1725	1459	1459
q11	512	337	328	328
q12	734	569	457	457
q13	18086	3320	2731	2731
q14	263	262	243	243
q15	q16	781	771	706	706
q17	999	982	985	982
q18	6799	5683	5659	5659
q19	1402	1267	1054	1054
q20	855	683	594	594
q21	5899	2634	2304	2304
q22	414	353	292	292
Total cold run time: 102893 ms
Total hot run time: 29241 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4372	4264	4273	4264
q2	289	334	210	210
q3	4545	4944	4460	4460
q4	2063	2144	1338	1338
q5	4359	4243	4271	4243
q6	228	171	126	126
q7	1688	1926	1637	1637
q8	2526	2169	2124	2124
q9	7713	7745	7788	7745
q10	4674	4611	4194	4194
q11	566	428	400	400
q12	745	771	552	552
q13	3438	3600	2971	2971
q14	321	305	292	292
q15	q16	707	753	641	641
q17	1365	1342	1337	1337
q18	8033	7279	7081	7081
q19	1143	1108	1086	1086
q20	2197	2214	1947	1947
q21	5180	4519	4436	4436
q22	513	438	390	390
Total cold run time: 56665 ms
Total hot run time: 51474 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 177592 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 24d1268559ebe6a9cb857ff198f9456c15ec2a56, data reload: false

query5	4320	613	466	466
query6	475	219	207	207
query7	4873	568	355	355
query8	329	182	172	172
query9	8768	4029	4060	4029
query10	459	359	289	289
query11	5906	2318	2141	2141
query12	156	104	100	100
query13	1245	578	422	422
query14	6255	5635	4873	4873
query14_1	4289	4258	4267	4258
query15	220	208	186	186
query16	998	470	449	449
query17	1141	732	576	576
query18	2431	460	353	353
query19	208	195	158	158
query20	113	108	108	108
query21	240	163	139	139
query22	13630	13557	13339	13339
query23	17509	16528	16199	16199
query23_1	16273	16128	16434	16128
query24	7466	1773	1302	1302
query24_1	1289	1278	1250	1250
query25	562	473	392	392
query26	1324	362	214	214
query27	2599	626	401	401
query28	4491	2000	2002	2000
query29	1080	637	492	492
query30	350	265	227	227
query31	1119	1093	975	975
query32	109	62	62	62
query33	533	330	256	256
query34	1169	1136	663	663
query35	766	779	684	684
query36	1192	1160	1012	1012
query37	153	108	95	95
query38	1884	1694	1693	1693
query39	892	872	865	865
query39_1	842	834	834	834
query40	248	168	155	155
query41	113	63	63	63
query42	95	88	87	87
query43	315	322	272	272
query44	1410	777	740	740
query45	193	183	173	173
query46	1023	1169	711	711
query47	2143	2115	2007	2007
query48	408	430	294	294
query49	584	421	309	309
query50	1118	426	320	320
query51	10677	10862	10916	10862
query52	85	84	74	74
query53	253	269	198	198
query54	278	235	225	225
query55	72	70	65	65
query56	297	305	304	304
query57	1352	1300	1202	1202
query58	287	258	264	258
query59	1568	1664	1466	1466
query60	319	272	253	253
query61	150	146	151	146
query62	547	497	436	436
query63	237	204	196	196
query64	2865	1027	866	866
query65	4711	4628	4617	4617
query66	1839	505	383	383
query67	29284	29203	29129	29129
query68	3173	1634	906	906
query69	403	300	282	282
query70	1039	953	956	953
query71	367	345	312	312
query72	3059	2700	2400	2400
query73	849	789	445	445
query74	5087	4948	4741	4741
query75	2549	2519	2152	2152
query76	2312	1158	775	775
query77	342	403	301	301
query78	11865	11966	11207	11207
query79	1404	1171	765	765
query80	1283	556	474	474
query81	533	336	297	297
query82	604	155	123	123
query83	372	329	305	305
query84	275	157	123	123
query85	980	658	518	518
query86	421	284	278	278
query87	1824	1821	1725	1725
query88	3685	2771	2783	2771
query89	437	391	325	325
query90	1923	195	191	191
query91	205	186	168	168
query92	61	57	56	56
query93	1701	1536	975	975
query94	730	365	320	320
query95	805	504	568	504
query96	999	777	344	344
query97	2640	2658	2500	2500
query98	210	204	207	204
query99	1094	1119	976	976
Total cold run time: 263744 ms
Total hot run time: 177592 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.98 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 24d1268559ebe6a9cb857ff198f9456c15ec2a56, data reload: false

query1	0.00	0.00	0.01
query2	0.09	0.05	0.05
query3	0.26	0.14	0.14
query4	1.61	0.14	0.14
query5	0.23	0.23	0.22
query6	1.23	1.05	1.08
query7	0.04	0.01	0.01
query8	0.05	0.03	0.04
query9	0.38	0.32	0.31
query10	0.55	0.54	0.53
query11	0.20	0.14	0.13
query12	0.18	0.14	0.13
query13	0.48	0.47	0.49
query14	1.02	1.01	0.99
query15	0.61	0.60	0.59
query16	0.33	0.32	0.31
query17	1.10	1.12	1.08
query18	0.23	0.20	0.20
query19	2.08	1.94	2.02
query20	0.01	0.02	0.02
query21	15.43	0.24	0.14
query22	4.80	0.06	0.05
query23	16.12	0.31	0.13
query24	3.14	0.44	0.32
query25	0.11	0.05	0.06
query26	0.72	0.21	0.16
query27	0.04	0.03	0.04
query28	3.56	0.96	0.58
query29	12.54	4.17	3.28
query30	0.27	0.15	0.16
query31	2.77	0.62	0.31
query32	3.23	0.60	0.48
query33	3.15	3.10	3.19
query34	15.58	4.27	3.54
query35	3.52	3.54	3.56
query36	0.55	0.44	0.41
query37	0.09	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.04
query40	0.18	0.17	0.15
query41	0.08	0.03	0.03
query42	0.03	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 96.72 s
Total hot run time: 24.98 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants