Describe the solution you'd like
Automation of loading files from Databricks Volumes which serves as a landing layer for the project ETL logic. Currently pipeline has hardcoded path to single file inside a Volume.
Task is to investigate possible options for automation of file ingestion into the volume. One of the solutions is to use autoloader.
Describe the solution you'd like
Automation of loading files from Databricks Volumes which serves as a landing layer for the project ETL logic. Currently pipeline has hardcoded path to single file inside a Volume.
Task is to investigate possible options for automation of file ingestion into the volume. One of the solutions is to use autoloader.