CFTL-589 GitHub OAuth as a third git authentication method

README.md

@@ -5,6 +5,8 @@
   - [Configuration](#configuration)
     - [Git configuration](#git-configuration)
     - [SSH configuration](#ssh-configuration)
+    - [GitHub OAuth configuration](#github-oauth-configuration)
+      - [Choose "Only select repositories"](#choose-only-select-repositories)
     - [Example: Running code saved in custom repository + template 🧩](#example-running-code-saved-in-custom-repository--template-)
     - [Example: Listing preinstalled packages](#example-listing-preinstalled-packages)
     - [Example: Accessing custom configuration parameters](#example-accessing-custom-configuration-parameters)
@@ -79,13 +81,16 @@ configured and executed directly in Keboola. This eliminates the need to build a
 
 The git configuration object supports the following parameters:
 
-- `url`: Repository URL – supports both HTTPS and SSH formats.
+- `url`: Repository URL – supports both HTTPS and SSH formats. Type or paste it, or with `"auth": "oauth"`
+  load the repositories the GitHub App may read and pick one from the list.
 - `branch`: Branch name to checkout – UI provides branch selection.
 - `filename`: Python script filename to execute – UI lists available files.
 - `auth`: Repository visibility & authentication method.
   - `none`: Public repository, no authentication (default).
   - `pat`: Private repository, Personal Access Token.
   - `ssh`: Private repository, SSH key.
+  - `oauth`: Private GitHub repository authorized via the **Authorization** section of the configuration.
+    Requires the Keboola GitHub App to be installed first – see [GitHub OAuth configuration](#github-oauth-configuration).
 - `#token`: Personal Access Token (`"auth": "pat"` only). This value will be encrypted in Keboola Storage.
   The same token also authenticates private git dependencies declared in `[tool.uv.sources]` in your `pyproject.toml`,
   so there is no need to embed tokens directly in the source file.
@@ -99,6 +104,58 @@ The git configuration object supports the following parameters:
   - `#private`: Private key used for authentication. This value will be encrypted in Keboola Storage.
 
 
+### GitHub OAuth configuration
+
+With `"auth": "oauth"` no credential is entered into the configuration at all – the access token is issued by
+the Keboola OAuth broker. Only `https://github.com` URLs are supported.
+
+The Keboola GitHub App has the client ID `Iv23liWeMeCpr1xBOVsj`. You can review the access you granted it, and
+revoke it, at
+[github.com/settings/connections/applications/Iv23liWeMeCpr1xBOVsj](https://github.com/settings/connections/applications/Iv23liWeMeCpr1xBOVsj).
+
+Setting this up takes three steps, **in this order**. Only the first one happens on GitHub:
+
+1. **Install** the app on your account or organisation
+   (`https://github.com/apps/keboola-custom-python/installations/new`) and choose which repositories it may read.
+   Repository selection happens here and nowhere else.
+2. **Authorize** the component in the **Authorization** section of the configuration in Keboola.
+3. Pick the repository in **Repository URL** – **List Repositories** loads what the installation makes available.
+
+Installing and authorizing are independent. Authorizing does not install the app, and the authorization
+screen offers no repository selection at all – so if you authorize without installing first, you receive a
+valid token that can see no repositories and **List Repositories** reports that none are available. To
+change which repositories are available later, reconfigure the installation on GitHub; re-authorizing in
+Keboola will not change it.
+
+Some organisations require an owner to approve the installation before it takes effect.
+
+
+#### Choose "Only select repositories"
+
+The installation dialog offers **All repositories** or **Only select repositories**. Choose the second one and
+list only the repositories this component needs.
+
+That dialog is the only place where the reach of the access token is decided, and narrowing it is the entire
+reason to use OAuth rather than a personal access token. **All repositories** grants `Contents: Read-only`
+across every repository in the account or organisation, including ones created later, and the token that
+results does not expire – which is the same over-scoped, long-lived credential that a personal access token
+was criticised for. Keboola cannot narrow this from its side; only the installation can.
+
+On a large organisation, **All repositories** also makes the **Repository** dropdown slow to load or unable to
+load at all. A narrow selection avoids that.
+
+The token is always the intersection of what the app may read and what you can read yourself, so it never
+reaches anything you could not already reach. The app requests `Contents: Read-only` and `Metadata:
+Read-only`, and nothing else.
+
+Private git dependencies declared in `[tool.uv.sources]` authenticate with the same token and need no
+credentials of their own. The repositories they live in have to be part of the installation's repository
+selection as well, not just the repository holding the code.
+
+To change the selection later, reconfigure the installation on GitHub. Re-authorizing in Keboola does not
+change it.
+
+
 ### Example: Running code saved in custom repository + template 🧩
 
 As this might become a preferred way of running custom Python code in Keboola for many, we prepared a [simple example project](https://github.com/keboola/component-custom-python-example-repo-1), which help you with your first steps (and can also server you as a template for any of your future projects).

component_config/configSchema.json

@@ -109,15 +109,10 @@
         }
       },
       "required": [
-        "url",
-        "auth"
+        "auth",
+        "url"
       ],
       "properties": {
-        "url": {
-          "type": "string",
-          "title": "Repository URL",
-          "propertyOrder": 70
-        },
         "auth": {
           "type": "string",
           "format": "radio",
@@ -126,13 +121,16 @@
           "enum": [
             "none",
             "pat",
-            "ssh"
+            "ssh",
+            "oauth"
           ],
           "options": {
+            "tooltip": "The **GitHub (OAuth)** option is authorized in the **Authorization** section of this configuration. The repositories the component may read are selected on GitHub when the Keboola GitHub App is installed.",
             "enum_titles": [
               "Public – None",
               "Private – Personal Access Token",
-              "Private – SSH Key"
+              "Private – SSH Key",
+              "Private – GitHub (OAuth)"
             ]
           },
           "default": "none"
@@ -158,6 +156,23 @@
             }
           }
         },
+        "url": {
+          "type": "string",
+          "enum": [],
+          "format": "select",
+          "title": "Repository URL",
+          "minLength": 1,
+          "propertyOrder": 105,
+          "options": {
+            "creatable": true,
+            "tooltip": "Type or paste the clone URL. With GitHub authorization you can also load the repositories the Keboola GitHub App is allowed to read; to make more of them available, change the repository selection of the installation on GitHub.",
+            "async": {
+              "label": "List Repositories",
+              "action": "listRepositories",
+              "cache": false
+            }
+          }
+        },
         "branch": {
           "type": "string",
           "enum": [],

src/component.py

@@ -15,6 +15,7 @@
 from keboola.component.exceptions import UserException
 
 from configuration import AuthEnum, Configuration, SourceEnum, VenvEnum, encrypted_keys
+from github_api import GitHubApi
 from package_installer import PackageInstaller
 from source_file import FileHandler
 from source_git import GitHandler
@@ -64,13 +65,29 @@ def __init__(self):
                 f"in the configuration. Detail: {err}"
             ) from err
 
+        self.oauth_token = self._get_oauth_token()
+
+    def _get_oauth_token(self) -> str | None:
+        """Access token issued by the OAuth broker, delivered outside "parameters" in the authorization
+        section. Returns None for configurations that do not use OAuth."""
+        try:
+            credentials = self.configuration.oauth_credentials
+        except json.JSONDecodeError as err:
+            # unreadable broker credentials are a configuration problem, not an internal one
+            raise UserException(
+                "The stored GitHub authorization could not be read. Please authorize the component again "
+                "in the Authorization section of the configuration."
+            ) from err
+
+        return credentials.data.get("access_token") if credentials else None
+
     def run(self):
         if self.parameters.source == SourceEnum.CODE:
             base_path = Path(self.data_folder_path)
             script_filename = FileHandler.prepare_script_file(self.data_folder_path, self.parameters.code)
         else:
             base_path = Path(GitHandler.REPO_PATH).absolute()
-            git_handler = GitHandler(self.parameters.git)
+            git_handler = GitHandler(self.parameters.git, self.oauth_token)
             script_filename = git_handler.clone_repository()
 
         if self.parameters.venv == VenvEnum.BASE:
@@ -87,7 +104,7 @@ def run(self):
                 self.parameters.packages.insert(0, "keboola.component")
             PackageInstaller.install_packages(self.parameters.packages)
         else:
-            PackageInstaller.install_packages_for_repository(base_path)
+            PackageInstaller.install_packages_for_repository(base_path, git_handler.subprocess_env())
 
         self._merge_user_parameters()
 
@@ -147,18 +164,49 @@ def _merge_user_parameters(self):
         # remove code
         config_data = self.configuration.config_data.copy()
 
+        # the authorization section carries the decrypted OAuth access token and the shared application
+        # secret, neither of which may reach the executed user script
+        config_data.pop("authorization", None)
+
         # build config data and overwrite for the user script
         config_data["parameters"] = self.parameters.user_properties
         with open(Path(self.data_folder_path) / "config.json", "w+") as inp:
             json.dump(config_data, inp)
 
+    @sync_action("listRepositories")
+    def get_oauth_repositories(self):
+        """
+        Returns the repositories the Keboola GitHub App is allowed to read.
+        This method is used to populate the repository dropdown in the UI.
+        """
+        if self.parameters.git.auth != AuthEnum.OAUTH:
+            # the button belongs to the shared repository field, so every authentication method is
+            # offered it even though only GitHub authorization can answer it
+            raise UserException("Supported only for OAuth. Please insert the URL manually.")
+
+        if not self.oauth_token:
+            raise UserException(
+                "GitHub authorization is missing. Please authorize the component in the Authorization "
+                "section of the configuration."
+            )
+
+        repositories = GitHubApi(self.oauth_token).list_installation_repositories()
+        if not repositories:
+            # authorizing does not install the app, so this is the expected state after authorizing alone
+            raise UserException(
+                "No repositories are available to the Keboola GitHub App. Install the app on the account "
+                "owning the repository and include that repository in the app's repository selection."
+            )
+
+        return repositories
+
     @sync_action("listBranches")
     def get_repository_branches(self):
         """
         Returns a list of branches in the git repository.
         This method is used to populate the branches dropdown in the UI.
         """
-        git_handler = GitHandler(self.parameters.git)
+        git_handler = GitHandler(self.parameters.git, self.oauth_token)
         return git_handler.get_repository_branches()
 
     @sync_action("listFiles")
@@ -167,7 +215,7 @@ def get_repository_files(self):
         Returns a list of branches in the git repository.
         This method is used to populate the branches dropdown in the UI.
         """
-        git_handler = GitHandler(self.parameters.git)
+        git_handler = GitHandler(self.parameters.git, self.oauth_token)
         return git_handler.get_repository_files()
 
 

src/configuration.py

@@ -25,6 +25,7 @@ class AuthEnum(Enum):
     NONE = "none"
     PAT = "pat"
     SSH = "ssh"
+    OAUTH = "oauth"
 
 
 # the ssh_keys.keys.[#private,public] structure is based on Keboola's standard SSH keys UI element output structure

src/github_api.py

@@ -0,0 +1,86 @@
+import json
+import urllib.error
+import urllib.request
+
+from keboola.component.exceptions import UserException
+
+API_BASE_URL = "https://api.github.com"
+API_VERSION = "2022-11-28"
+USER_AGENT = "keboola-custom-python-component"
+PAGE_SIZE = 100
+REQUEST_TIMEOUT = 30
+# a user access token never reaches anywhere near this many pages, it is a runaway guard only
+MAX_PAGES = 50
+
+
+class GitHubApi:
+    """Read-only client for the endpoints needed to list the repositories the app may read."""
+
+    def __init__(self, token: str, base_url: str = API_BASE_URL):
+        self.token = token
+        self.base_url = base_url.rstrip("/")
+
+    def list_installation_repositories(self) -> list[dict]:
+        """Repositories of every app installation visible to the authorizing user, as dropdown options.
+
+        The value is the clone URL, so that selecting a repository fills in the same thing the other
+        authentication methods expect to be typed in by hand.
+        """
+        options = []
+        for installation in self._get_all("/user/installations", "installations"):
+            path = f"/user/installations/{installation['id']}/repositories"
+            for repository in self._get_all(path, "repositories"):
+                # value and label are deliberately the same string: the options are only loaded when
+                # the user runs the action, so a saved configuration reopened without running it again
+                # would show the bare value under a label the form cannot resolve
+                clone_url = repository["clone_url"]
+                options.append({"value": clone_url, "label": clone_url})
+
+        return options
+
+    def _get_all(self, path: str, items_key: str) -> list[dict]:
+        items: list[dict] = []
+        for page in range(1, MAX_PAGES + 1):
+            payload = self._get(f"{path}?per_page={PAGE_SIZE}&page={page}")
+            page_items = payload.get(items_key, [])
+            items.extend(page_items)
+            if not page_items or len(items) >= payload.get("total_count", 0):
+                break
+
+        return items
+
+    def _get(self, path: str) -> dict:
+        request = urllib.request.Request(
+            f"{self.base_url}{path}",
+            headers={
+                "Authorization": f"Bearer {self.token}",
+                "Accept": "application/vnd.github+json",
+                "X-GitHub-Api-Version": API_VERSION,
+                # GitHub rejects requests without a User-Agent
+                "User-Agent": USER_AGENT,
+            },
+        )
+
+        try:
+            with urllib.request.urlopen(request, timeout=REQUEST_TIMEOUT) as response:
+                return json.load(response)
+        except urllib.error.HTTPError as err:
+            raise UserException(self._explain_http_error(err.code, err.reason)) from err
+        except urllib.error.URLError as err:
+            raise UserException(f"Could not reach the GitHub API: {err.reason}") from err
+        except OSError as err:
+            # a read timeout arrives as a bare TimeoutError: an OSError, but not a URLError, and
+            # with no "reason" attribute to report
+            raise UserException(f"Could not reach the GitHub API: {err}") from err
+
+    @staticmethod
+    def _explain_http_error(code: int, reason: str) -> str:
+        if code == 401:
+            return (
+                "The GitHub authorization is no longer valid. Please authorize the component again in the "
+                "Authorization section of the configuration."
+            )
+        if code == 403:
+            return "The GitHub API refused the request. The authorization may not have the required permissions."
+
+        return f"The GitHub API returned an unexpected error: HTTP {code} {reason}"

src/package_installer.py

@@ -17,14 +17,16 @@ def install_packages(packages: list[str]):
             SubprocessRunner.run(args, MSG_OK, MSG_ERR)
 
     @staticmethod
-    def install_packages_for_repository(repository_path: Path):
+    def install_packages_for_repository(repository_path: Path, env: dict[str, str] | None = None):
         """
         Install packages based on the given repository path.
         - If there is a pyproject.toml and a uv.lock file, run uv sync.
         - If there is a requirements.txt file, install packages from it using uv.
 
         Args:
             repository_path (str): Path to the repository containing requirements.txt.
+            env: Environment for the installation. Carries the git credentials, without which
+                private git dependencies cannot be fetched.
         """
         pyproject_file = repository_path / "pyproject.toml"
         uv_lock_file = repository_path / "uv.lock"
@@ -46,4 +48,4 @@ def install_packages_for_repository(repository_path: Path):
             logging.info("No dependencies file found")
             return
 
-        SubprocessRunner.run(args, MSG_OK, MSG_ERR)
+        SubprocessRunner.run(args, MSG_OK, MSG_ERR, env)