[Bug]: Schema widening int -> string does not promote values, so Floe encoding fails on sort/join/group-by

[jenky]

What happened?

My code worked fine on 0.41, but the issue started happening after upgrading to 0.42. Nothing changed on my end. It seems the issue comes from the new "Floe" encoder. Here are Claude's findings:

When a column contains both integers and strings, IntegerDefinition::merge() widens the column definition to StringDefinition, but the row values are left untouched — the rows still hold IntegerEntry with a PHP int.

PhpFloeEncoder builds one encoder per column from the schema alone:

// src/core/etl/src/Flow/Floe/PhpFloeEncoder.php::buildEncoders()
foreach ($this->schema->definitions() as $name => $definition) {
    $encoders[$name] = $valueEncoder->encoderFor($definition->type());
}

and then applies that encoder to whatever the row holds, without any per-value check. So the widened column gets a StringEncoder, which is handed an int:

// src/core/etl/src/Flow/Floe/Encoding/StringEncoder.php
final class StringEncoder implements ValueEncoder
{
    public function encode(mixed $value): string
    {
        /** @var string $value */
        return pack('V', strlen($value)) . $value;   // strlen(int) under strict_types -> TypeError
    }
}

The schema says string, the value is an int, and the file declares strict_types=1, so strlen() throws.

Why this shows up everywhere in 0.42+: since 0.42 the default execution paths spill through Floe — EXTERNAL_SORT became the default sort algorithm (#2526), and group-by and hash join were rebuilt on the bucketing engine (#2543, #2536). So this fires on ordinary pipelines that never mention Floe. I confirmed all three fail: sortBy(), groupBy()->aggregate(), and join().

Expected: either the widening promotes the values losslessly (int 1000 -> string "1000") so the schema and values stay in agreement, or int/string columns merge into a union that the encoder resolves per value, or the encoder rejects the mismatch with a clear Flow-level error instead of a raw TypeError from strlen().

Regression range:

version	sortBy() on a mixed int/string column
0.41.1	OK
0.42.0	TypeError
0.43.0	TypeError

0.43.0 fixed several adjacent merge issues (#2589 Definition::merge() no longer throws on incompatible types, union columns build the correct entry per value, CommonType, UnionDefinition::memberFor()), but IntegerDefinition::merge() still returns StringDefinition unchanged and StringEncoder is untouched, so this specific case still fails on the latest release. The new union handling only engages when the other side is already a UnionDefinition.

How to reproduce?

<?php

declare(strict_types=1);

require __DIR__ . '/vendor/autoload.php';

use function Flow\ETL\DSL\{data_frame, from_array, ref};

// A column that holds both ints and strings - e.g. product codes from a spreadsheet
$rows = [];

for ($i = 0; $i < 50; $i++) {
    $rows[] = ['code' => $i % 2 === 0 ? 1000 + $i : 'AB-' . $i];
}

data_frame()
    ->read(from_array($rows))
    ->sortBy(ref('code'))
    ->run();


`groupBy()->aggregate(sum(ref('amount')))` and `join(...)` on the same data fail identically.

To see the root cause without any Floe involvement:


<?php

declare(strict_types=1);

require __DIR__ . '/vendor/autoload.php';

use function Flow\ETL\DSL\{int_entry, string_entry, row, rows, ref};

$rows = rows(row(int_entry('code', 1000)), row(string_entry('code', 'AB-1')));
$def  = $rows->schema()->get('code');

echo "schema definition : ", $def::class, "\n";              // StringDefinition
echo "schema type       : ", $def->type()->toString(), "\n"; // string

foreach ($rows as $i => $r) {
    echo "row $i value type  : ", get_debug_type($r->get(ref('code'))->value()), "\n";
}
// row 0 value type  : int      <-- schema says string
// row 1 value type  : string

Playground snippet

No response

Data required to reproduce bug locally

None — the snippets above generate their own data. Any column mixing PHP int and string values triggers it. In our case it came from a Google Sheets extract where product codes are numeric on some rows and alphanumeric on others.

Version

0.42+

Relevant error output

PHP Fatal error:  Uncaught TypeError: strlen(): Argument #1 ($string) must be of type string, int given in vendor/flow-php/etl/src/Flow/Floe/Encoding/StringEncoder.php:15
Stack trace:
#0 vendor/flow-php/etl/src/Flow/Floe/PhpFloeEncoder.php(127): Flow\Floe\Encoding\StringEncoder->encode(1000)
#1 vendor/flow-php/etl/src/Flow/Floe/AdaptiveFloeEncoder.php(34): Flow\Floe\PhpFloeEncoder->encode(Array)
#2 vendor/flow-php/etl/src/Flow/Floe/FloeStreamWriter.php(270): Flow\Floe\AdaptiveFloeEncoder->encode(Array)
#3 vendor/flow-php/etl/src/Flow/Floe/FloeStreamWriter.php(176): Flow\Floe\FloeStreamWriter->emitBatch(Array)
#4 vendor/flow-php/etl/src/Flow/Floe/FloeWriter.php(110): Flow\Floe\FloeStreamWriter->write(Object(Flow\ETL\Rows))
#5 vendor/flow-php/etl/src/Flow/ETL/Bucketing/Storage/FilesystemBuckets.php(104): Flow\Floe\FloeWriter->write(Object(Flow\ETL\Rows))
#6 vendor/flow-php/etl/src/Flow/ETL/Bucketing/SortedRunBucketing.php(61): Flow\ETL\Bucketing\Storage\FilesystemBuckets->set('sort-c7ab3671fc...', Object(Flow\ETL\Rows))
#7 vendor/flow-php/etl/src/Flow/ETL/Bucketing/SortedRunBucketing.php(54): Flow\ETL\Bucketing\SortedRunBucketing->spill(Object(Flow\ETL\Rows), Object(Flow\ETL\Bucketing\Storage\FilesystemBuckets), 'c7ab3671fc9b889...', 0)
#8 vendor/flow-php/etl/src/Flow/ETL/Processor/BucketingProcessor.php(31): Flow\ETL\Bucketing\SortedRunBucketing->bucketize(Object(Generator), Object(Flow\ETL\Bucketing\Storage\FilesystemBuckets))
#9 [internal function]: Flow\ETL\Processor\BucketingProcessor->process(Object(Generator), Object(Flow\ETL\FlowContext))
#10 vendor/flow-php/etl/src/Flow/ETL/Pipeline/Segment.php(70): Generator->valid()
#11 vendor/flow-php/etl/src/Flow/ETL/Processor/MergeSortProcessor.php(53): Flow\ETL\Pipeline\Segment->execute(Object(Generator), Object(Flow\ETL\FlowContext))
#12 [internal function]: Flow\ETL\Processor\MergeSortProcessor->process(Object(Generator), Object(Flow\ETL\FlowContext))
#13 vendor/flow-php/etl/src/Flow/ETL/Pipeline/Segment.php(70): Generator->valid()
#14 vendor/flow-php/etl/src/Flow/ETL/Pipeline.php(67): Flow\ETL\Pipeline\Segment->execute(Object(Generator), Object(Flow\ETL\FlowContext))
#15 vendor/flow-php/etl/src/Flow/ETL/DataFrame.php(783): Flow\ETL\Pipeline->process(Object(Flow\ETL\FlowContext))
#16 minimal.php(19): Flow\ETL\DataFrame->run()
#17 {main}

[norberttech]

Thanks for reporting @jenky
I'm going take a look at this today!

[norberttech]

Hmm making Floe a default serialization engine is the reason behind this issue, and the reason why it worked in 0.41.1 but stopped in 0.42.0 was a fix implemented in the Floe format.

Floe is a binary, self descriptive data format optimized for saving Rows.
This means that Floe requires schema to be delivered upfront, otherwise it will try to infer (guess it) based on the first batch of rows.

If rows are coming in batches like this:

rows 0: int, string, string 
rows 1: int, int, string

the second write of rows will cause the type error as int !== string.

In your case @jenky the quick walkaround is to tell DataFrame upfront what are the column types. You can do it either by casting each field separately or by passing schema to Extractor (most extractors exposes ->withSchema()) method.

Obviously it's not a solution, I need to figure out a better mechanism but for now casting / providing schema should od the trick for you