[runtime] Resolve relative file: URLs when loading classpath skills (#966) - #1033
[runtime] Resolve relative file: URLs when loading classpath skills (#966)#1033Olawoyin007 wants to merge 2 commits into
Conversation
…pache#966) A Flink deployment can add user-code JARs relative to the TaskManager working directory, so a classloader may expose a skill resource through a relative file: URL such as file:../../flink/usrlib/job.jar. That URL parses to an opaque URI, which new File(URI) / Paths.get(URI) reject with "URI is not hierarchical", failing skill initialization. Centralize local file: URL resolution in LocalUrls.toLocalFile, which resolves an opaque relative URI against the process working directory and keeps existing behavior for absolute hierarchical file URLs. Reuse it in the three affected sites: direct file resource materialization, JAR entry extraction, and the URLClassLoader fallback scan. Non-file URLs are rejected explicitly. Add a LocalUrls unit test and an end-to-end ClasspathSkillRepository test that loads skills through a relative jar URL.
6143698 to
991ff6b
Compare
wenjin272
left a comment
There was a problem hiding this comment.
Thanks for working on this fix! The overall approach looks good. I left two inline comments about an exception-handling edge case and regression-test coverage.
| // directory, which is exactly how a relative File is interpreted. | ||
| return new File(uri.getSchemeSpecificPart()); | ||
| } | ||
| return new File(uri); |
There was a problem hiding this comment.
Could we also wrap the IllegalArgumentException thrown by new File(uri)? For example, file://host/share/job.jar is hierarchical, but new File(uri) throws IllegalArgumentException: URI has an authority component. Since the callers only catch IOException, this escapes the fallback scan instead of being skipped and also changes the previous copyJarEntries() exception contract. Shall we keep the conversion inside the try and wrap both URISyntaxException and IllegalArgumentException as IOException?
There was a problem hiding this comment.
Good catch, fixed in 05b4a46. new File(uri) is now inside the try, and the catch handles both URISyntaxException and IllegalArgumentException, wrapping them as IOException. So a hierarchical-but-non-representable URL like file://host/share/job.jar now surfaces as IOException and is skipped by the fallback scan, restoring the previous copyJarEntries contract instead of escaping unchecked. Added a LocalUrlsTest case asserting the authority-component URL comes back as IOException.
| } | ||
|
|
||
| @Test | ||
| void loadFromRelativeJarUrl(@TempDir Path tempDir) throws IOException { |
There was a problem hiding this comment.
This test covers the URLClassLoader fallback scan and JAR extraction paths, but the PR also changes direct resource materialization. Could we add regression cases where getResources() directly returns a relative jar:file: URL and a relative file: directory URL? That would exercise all three affected call paths described in the PR.
There was a problem hiding this comment.
Agreed, added in 05b4a46. Two new regression tests feed the URLs through getResources() directly (via a non-URLClassLoader loader, so the fallback scan is bypassed and each path is exercised in isolation): loadFromDirectRelativeJarUrl returns a relative jar:file: URL to drive SkillMaterializer.copyJarEntries, and loadFromDirectRelativeDirectoryUrl returns a relative file: directory URL to drive ClasspathSkillRepository.materializeFileUrl. Together with the existing fallback-scan test, all three affected call paths are now covered.
…apache#966) Address review feedback on the relative file: URL fix. LocalUrls.toLocalFile only wrapped URISyntaxException, so a hierarchical but non-representable file: URL (for example file://host/share/job.jar, which new File(URI) rejects with "URI has an authority component") escaped as an unchecked IllegalArgumentException. Callers only catch IOException, so this broke the fallback scan's graceful skip and changed the previous copyJarEntries exception contract. Move the conversion inside the try and wrap both URISyntaxException and IllegalArgumentException as IOException. Add regression tests for the two directly-affected call paths not yet covered: getResources() returning a relative jar:file: URL (drives SkillMaterializer.copyJarEntries) and a relative file: directory URL (drives ClasspathSkillRepository.materializeFileUrl), plus a LocalUrls test that the authority-component URL surfaces as IOException.
What is the purpose of the change
Fixes #966.
A Flink deployment can add user-code JARs relative to the TaskManager working directory, so a classloader may expose a skill resource through a relative
file:URL such asfile:../../flink/usrlib/job.jar(orjar:file:../../flink/usrlib/job.jar!/diagnosis-skills). Such a URL parses to an opaqueURI, whichnew File(URI)/Paths.get(URI)reject withIllegalArgumentException: URI is not hierarchical, soClasspathSkillRepositoryinitialization fails and skills never load.Brief change log
LocalUrls.toLocalFile(URL), a single place that converts a localfile:URL to aFile:file:URLs keep their existingnew File(uri)behavior;file:URLs are resolved against the process working directory via the decoded scheme-specific part;fileURLs are rejected explicitly.ClasspathSkillRepository.materializeFileUrl(direct file/dir/zip resource),SkillMaterializer.copyJarEntries(JAR entry extraction),ClasspathSkillRepository.findAllMatches(URLClassLoaderfallback scan).Verifying this change
LocalUrlsTestcovers absolute resolution, relative/opaque resolution against the working directory, and non-filerejection.ClasspathSkillRepositoryTest#loadFromRelativeJarUrlloads skills end-to-end through a relative jar URL (exercises both the fallback scan and jar extraction). Verified that this test fails against the pre-fix code with the reportedClasspath resource not found/URI is not hierarchicalerror.runtimemodule test suite passes (668 tests, 0 failures).Does this pull request potentially affect one of the following parts
Documentation