[BUG] Duplicate fields returned when disable_objects set

[tronboto]

What is the bug?

Fields are duplicated in search results when objects have disable_objects set to true.

How can one reproduce the bug?

1. Create the index with disable_objects: true, index a document and query using the PPL API

PUT test-disable-objects
{
  "mappings": {
    "properties": {
        "attributes": {
          "disable_objects": true,
          "type": "object"
        }
      }
  }
}

POST test-disable-objects/_doc
{
  "attributes": {
    "log.file.path": "/var/log/app.log"
  }
}

POST _plugins/_ppl
{
  "query" : "source=test-disable-objects"
}

2. This returns both an attributes struct field and a log.file.path string field:

{
  "schema": [
    {
      "name": "attributes",
      "type": "struct"
    },
    {
      "name": "attributes.log.file.path",
      "type": "string"
    }
  ],
  "datarows": [
    [
      {
        "log": {
          "file": {
            "path": "/var/log/app.log"
          }
        }
      },
      "/var/log/app.log"
    ]
  ],
  "total": 1,
  "size": 1
}

What is the expected behavior?

Fields should not be duplicated

What is your host/environment?

Opensearch 3.8.0

Do you have any screenshots?
If applicable, add screenshots to help explain your problem.

Do you have any additional context?

This causes fields to be duplicated in the Opensearch Dashboards Explore Logs app:

Interestingly, there is no attributes.logtag in that screenshot, which suggests the bug only presents itself when there are multiple layers of nesting.

[penghuo]

@tronboto Thanks for reporting the issue.

Reproduced on 3.9.0-SNAPSHOT. Root cause is the field dedup check,

With disable_objects: true, OpenSearch doesn't expand the dotted key into intermediate object mappers — the mapping holds one leaf literally named log.file.path. So flattening yields {attributes, attributes.log.file.path} with no attributes.log / attributes.log.file in between.

tryToRemoveNestedFields (CalciteRelNodeVisitor.java:665) drops a field only when the prefix up to the last dot is also a field. attributes.log.file isn't, so the leaf survives next to the struct. attributes.logtag is dropped correctly — hence only multi-dot keys duplicate.