Skip to content

Naming discrepancy between env in Processing and environment in Estimator #6214

Description

@mikeweltevrede

Describe the feature you'd like

There is a discrepancy on passing environment variables in Processing and Estimator. The parameter is called env in Processing and environment in Estimator.

I would like these to be aligned. For backwards compatibility sake, this should probably be manifested through a environment_variables parameter, but any solution would work for me.

The problem

The problem is that we cannot have a unified interface to these entities using **kwargs to pass arguments without manually parsing a parameter ourselves.

Current situation van be something like this if we use env for both situations:

def data_processing(environment: Literal["dev", "preprod", "prod"], **kwargs):
    initialize_environment(environment)

    return Processing(**kwargs)

def model_training(environment: Literal["dev", "preprod", "prod"], **kwargs):
    initialize_environment(environment)

    env_vars = kwargs.pop("env")
    if env_vars:
        kwargs["environment"] = env_vars

    return Estimator(**kwargs)

data_processing(environment="dev", env={"MY_VAR": 42})
model_training(environment="dev", env={"MY_VAR": 67})

Or with a more compatible interface

def data_processing(environment: Literal["dev", "preprod", "prod"], **kwargs):
    initialize_environment(environment)

    env_vars = kwargs.pop("environment_variables")
    if env_vars:
        kwargs["env"] = env_vars

    return Processing(**kwargs)

def model_training(environment: Literal["dev", "preprod", "prod"], **kwargs):
    initialize_environment(environment)

    env_vars = kwargs.pop("environment_variables")
    if env_vars:
        kwargs["environment"] = env_vars

    return Estimator(**kwargs)

data_processing(environment="dev", environment_variables={"MY_VAR": 42})
model_training(environment="dev", environment_variables={"MY_VAR": 67})

Ideally, we would want it to look like this because both classes accept a environment_variables parameter:

def data_processing(environment: Literal["dev", "preprod", "prod"], **kwargs):
    initialize_environment(environment)

    return Processing(**kwargs)

def model_training(environment: Literal["dev", "preprod", "prod"], **kwargs):
    initialize_environment(environment)

    return Estimator(**kwargs)

data_processing(environment="dev", environment_variables={"MY_VAR": 42})
model_training(environment="dev", environment_variables={"MY_VAR": 67})

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions