MAX_COLUMN_BYTES is applied only by default only if no annotation is …#59
Conversation
jakubZielAdform
force-pushed
the
get-rid-of-max-column-bytes-default-limit-for-string-fields-in-generated-encoders
branch
2 times, most recently
from
5742e55 to
d941d85
Compare
…applied on a field
jakubZielAdform
force-pushed
the
get-rid-of-max-column-bytes-default-limit-for-string-fields-in-generated-encoders
branch
from
d941d85 to
91a07a1
Compare
| Column(q"false", q"${fl.length}", q"pw.writeFixedString(r, ${fl.length}, ${fl.truncate})") | ||
| case ml: MaxLength => | ||
| Column(q"false", q"-1", q"pw.writeVarString(r, ${ml.length}, ${ml.truncate})") | ||
| }.get |
There was a problem hiding this comment.
If we get a non matching field in annotations this get will throw exception, can we add getOrElse abort so that we may get a clear msg instead of None.get
| case fl: FixedLength => | ||
| Column(q"false", q"${fl.length}", q"pw.writeFixedString(r, ${fl.length}, ${fl.truncate})") | ||
| case ml: MaxLength => | ||
| Column(q"false", q"-1", q"pw.writeVarString(r, ${ml.length}, ${ml.truncate})") |
There was a problem hiding this comment.
What would be the maximum allowed length of the field ?
There was a problem hiding this comment.
imo, there should no be any max length when user is adding an annotation.
Case class that is processed here is supposed to be a mapping of db table and be an ultimate src of truth.
Tbh I think that silent truncating is also a bad option, i suppose that is probably done because one 'too big' record would fail entire file loading.
I think that would be nice to consider reading a schema at the beginning and treating it as a src of truth and based on that removing 'too big' records from a batch. Of course it would have to be done in runtime, not in preprocessing.
wdyt?
There was a problem hiding this comment.
can we create a tag from this PR and test with some test table in vertica ?
As per vertica documentation, 65,000 is the physical limit for VARCHAR/VARBINARY. While for string we may use LONGVARCHAR, Vertica arrays do not support long data types
There was a problem hiding this comment.
table for which this is done is
create table fraud.crawler_output_raw
(
_topic varchar(128) not null,
_partition int not null,
_offset int not null,
_watermark timestamp not null,
log_time timestamp not null,
url varchar(256),
content long varchar(32000000),
redirection_urls varchar(65000),
redirection_codes varchar(65000),
error_code varchar(1000),
error_message varchar(65000)
);
so it looks like you are right, this change is caused by a problem with loading ads.txt files, which sometiems are bigger than 65k characters.
docs
https://docs.vertica.com/24.2.x/en/sql-reference/data-types/long-data-types/
There was a problem hiding this comment.
I've already tested it (loaded sources as a dependency) and it worked fine.
jakubZielAdform
force-pushed
the
get-rid-of-max-column-bytes-default-limit-for-string-fields-in-generated-encoders
branch
2 times, most recently
from
f4513fb to
76bd37a
Compare
jakubZielAdform
force-pushed
the
get-rid-of-max-column-bytes-default-limit-for-string-fields-in-generated-encoders
branch
2 times, most recently
from
813c9bd to
f53b5ac
Compare
| s"KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://${dockerNetwork.ip}:$kafkaPort", | ||
| s"KAFKA_CONTROLLER_QUORUM_VOTERS=1@127.0.0.1:$kafkaControllerPort", | ||
| s"KAFKA_LOG_RETENTION_HOURS=${Int.MaxValue}", | ||
| s"KAFKA_MESSAGE_MAX_BYTES=${32_000_000}", |
There was a problem hiding this comment.
allow for bigger msg sizes on a kafka broker (will be usable when test classes are exposed via #60)
jakubZielAdform
force-pushed
the
get-rid-of-max-column-bytes-default-limit-for-string-fields-in-generated-encoders
branch
from
f53b5ac to
58108ec
Compare
2 participants
…applied on a field