feat: add aws-uc-external-location-file-events module - #237
Open
dgokeeffe wants to merge 15 commits into
Open
Conversation
Add Terraform module for deploying Databricks clusters pre-configured with Claude Code CLI for AI-assisted development. The module includes: - Unity Catalog Volume for secure init script storage - Automatic Claude Code CLI installation via init scripts - MLflow tracing integration for Claude sessions - Helper bash functions for cluster users - Support for single-node and autoscaling clusters Follows Databricks best practices using Unity Catalog Volumes for init script storage (DBR 13.3+).
Add Terraform module for deploying Databricks clusters with Claude Code CLI in air-gapped or restricted network environments. The module includes: - Offline installation scripts for Claude Code CLI - Dependency downloader for preparing offline packages - Unity Catalog Volume for secure script storage - MLflow tracing integration - Support for DBFS and workspace file storage Designed for environments without internet access or strict firewall policies.
Add complete deployment example demonstrating usage of the adb-coding-assistants-cluster module. The example includes: - Azure Databricks workspace integration - Provider configuration with unified authentication - Complete variable definitions with examples - Post-deployment usage instructions - Troubleshooting guide Uses Databricks unified authentication supporting Azure CLI, environment variables, and configuration profiles.
- Add adb-coding-assistants-cluster module to modules table - Add adb-coding-assistants-cluster-offline module to modules table - Add adb-coding-assistants-cluster example to examples table - Add *.plan pattern to .gitignore to prevent committing Terraform plan files
- Incorporate detailed usage instructions from temporary guide - Add steps for SSH setup, IDE connection, and port forwarding - Include tips for persistent sessions with tmux - Add instructions for finding Python interpreter path
Add automatic token refresh functionality and VS Code/Cursor Remote SSH setup helpers to both online and offline installation scripts. Token refresh features: - Automatic token refresh on shell login when token changes - Hourly cron job for periodic token refresh - Token hash tracking to detect changes efficiently - Helper commands: claude-token-status, claude-setup-token-refresh VS Code/Cursor Remote SSH features: - Helper functions to find Python virtual environment path - Setup guide and verification commands - VS Code settings.json generation - Standalone vscode-setup.sh script for independent use - Documentation updates with step-by-step instructions All helpers are automatically available after cluster initialization and support both VS Code and Cursor IDEs.
Add external data source to fetch Azure subscription ID from Azure CLI or environment variable when using profile-based Databricks authentication. This fixes the error where Azure provider could not determine subscription ID when databricks_resource_id is not provided. - Add external provider to required_providers - Add data source to get subscription ID from Azure CLI or ARM_SUBSCRIPTION_ID - Update provider configuration to use subscription ID from multiple sources
Remove automatic MLflow tracing enable and cron setup that could cause script failures. Keep all helper functions available for manual use. - Remove auto-enable MLflow tracing on login - Remove automatic cron job setup in main() - Keep all helper functions (claude-tracing-enable, claude-setup-token-refresh, etc.) - Users can manually enable features if desired
Document that Unity Catalog metastore must have a root storage credential configured before volumes can be created. This is required for the module to work properly.
Add support for Databricks CLI profile-based authentication as an alternative to Azure resource ID. This provides a simpler, cloud-agnostic authentication method. - Add databricks_profile variable to example and module - Update variable descriptions to clarify authentication options - Update terraform.tfvars.example with both authentication methods - Add validation to ensure at least one auth method is provided
Add CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1 environment variable to both init scripts (install-claude.sh and install-claude-minimal.sh) to disable experimental beta features when running Claude Code on Databricks clusters. The variable is exported in bashrc and set in the environment to ensure it persists across shell sessions and is available when the cluster starts. Co-authored-by: Cursor <cursoragent@cursor.com>
Add check-network-deps.sh to verify connectivity to all required domains before running the Claude Code installer. This helps diagnose network and firewall issues in restricted environments. Checked domains (10 total): - claude.ai - CLI installer script - api.anthropic.com - Claude CLI binary download - deb.nodesource.com - Node.js repository - archive.ubuntu.com - APT packages (x86_64) - ports.ubuntu.com - APT packages (ARM64) - registry.npmjs.org - NPM packages - pypi.org - Python package index - files.pythonhosted.org - Python package downloads - raw.githubusercontent.com - Databricks skills - storage.googleapis.com - Binary downloads Features: - Color-coded output (green OK / red FAIL) - DNS resolution check before HTTP check - 5-second connect timeout - --verbose flag for detailed HTTP status codes - Exit code 0 for success, 1 for failures - Troubleshooting tips when failures occur Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove api.anthropic.com (not used during installation) - Clarify storage.googleapis.com hosts Claude CLI binaries - The Claude installer downloads binaries from GCS bucket: storage.googleapis.com/claude-code-dist-*/claude-code-releases Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add an AWS module (and example) that creates a Unity Catalog storage credential and external location(s) with automatic managed SQS file events. This is the AWS counterpart to the existing Azure adb-uc-external-location-file-events module. The module encodes the two non-obvious requirements for this pattern: - a self-assuming IAM role (UCMasterRole trust gated by the Databricks account external ID, plus an AllowSelfAssume sts:AssumeRole permission on the role itself); and - the csms-*-scoped sns/sqs/s3 bucket-notification permissions required for automatic managed file events, without which file events silently fail to provision. Includes a time_sleep to let IAM propagate before UC validation, optional UC grants, and a paired example under examples/.
tanmay-db
requested review from
alexott
and removed request for
alexott and
tanmay-db
July 20, 2026 15:39
|
Hi @alexott, could you help review this? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AWS counterpart to adb-uc-external-location-file-events: UC storage
credential (self-assuming IAM role) + external location(s) with automatic managed
SQS file events, incl. the csms-* SNS/SQS/bucket-notification IAM permissions and
IAM-propagation wait. Adds a paired example under examples/.