diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..145f819 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,22 @@ +# devservices + +Python CLI that manages Docker Compose and supervisor dependencies for Sentry services. User docs live in [README.md](README.md). + +## Commands + +- Set up: `direnv allow`. [.envrc](.envrc) runs `uv sync` and installs the pre-commit hooks. +- Test: `pytest -n4`. Tests marked `integration` call real GitHub. +- Lint, format, and type check: `pre-commit run --all-files` (ruff, mypy, shellcheck). + +## Invariants + +- `scripts/bump-version.sh` rewrites the version in `pyproject.toml` and the `devservices==` line in [README.md](README.md). Keep that README line. + +## Where things live + +- CLI entry point and subcommand registration: [devservices/main.py](devservices/main.py) +- One module per subcommand: [devservices/commands/](devservices/commands/) +- Config parsing and validation: [devservices/configs/service_config.py](devservices/configs/service_config.py) +- Paths, versions, and timeouts: [devservices/constants.py](devservices/constants.py) +- Release: the `Release` workflow in [.github/workflows/release.yml](.github/workflows/release.yml) runs craft with [.craft.yml](.craft.yml). [.github/workflows/build.yml](.github/workflows/build.yml) builds the PyOxidizer binaries. +- Changes per version: [CHANGELOG.md](CHANGELOG.md) diff --git a/README.md b/README.md index 922165c..7143a0e 100644 --- a/README.md +++ b/README.md @@ -1,56 +1,60 @@ # devservices -A standalone cli tool used to manage dependencies for services. It simplifies the process of managing services for development purposes and bringing services up/down. +`devservices` is a CLI that brings a service's development dependencies up and down. It reads them from the `devservices/config.yml` file in your repository. -## Overview +## Requirements -`devservices` reads configuration files located in the `devservices` directory of your repository. These configurations define services, their dependencies, and various modes of operation. +- Docker must be running. If Docker Compose is missing or older than 2.29.7, `devservices` downloads a newer Compose into your Docker CLI plugins directory. +- Commands that take a service name, and `devservices list-services`, look for repositories in the devenv `coderoot` set in `~/.config/sentry-devenv/config.ini`. -## Usage +## Install -devservices provides several commands to manage your services: +1. Add `devservices` to your repository's `requirements.txt` and install it in your virtualenv: -### Commands + ``` + devservices==1.2.4 + ``` -NOTE: service-name is an optional parameter. If not provided, devservices will attempt to automatically find a devservices configuration in the current directory in order to proceed. +2. Add a `devservices/config.yml` file to your repository. See [Configuration](#configuration). +3. Run `devservices up`. -- `devservices up `: Bring up a service and its dependencies. -- `devservices down `: Bring down service including its dependencies. -- `devservices status `: Display the current status of all services, including their dependencies and ports. -- `devservices logs `: View logs for a specific service. -- `devservices list-services`: List all available Sentry services. -- `devservices list-dependencies `: List all dependencies for a service and whether they are enabled/disabled. -- `devservices update` Update devservices to the latest version. -- `devservices purge`: Purge the local devservices cache. -- `devservices toggle `: Toggle the runtime for a service between containerized and local. -- `devservices serve`: Run a service's devserver. -- `devservices foreground `: Foreground a process that is currently running in the background. This enables interactive debugging. +## Commands -## Installation +If you omit ``, `devservices` uses the `devservices/config.yml` file in the current directory. Run `devservices --help` for every flag. -### 1. Add devservices to your requirements.txt +- `devservices up `: Bring up a service and its dependencies. Use `--mode ` to pick a mode other than `default`. Use `--offline` to use cached dependencies and images. +- `devservices down `: Bring down a service and its dependencies. +- `devservices status `: Show the status and ports of a running service and its dependencies. +- `devservices logs `: Show logs for a running service. +- `devservices list-services` (alias `ls`): List running services. Use `--all` to include stopped services in your coderoot. +- `devservices list-dependencies `: List a service's dependencies. +- `devservices toggle [local|containerized]`: Switch a service's runtime between containerized and local. +- `devservices serve`: Run the `devserver` program from the current directory's `x-programs` block. Arguments after `serve` go to the devserver. +- `devservices foreground `: Stop a background program and run it in the foreground for interactive debugging. +- `devservices reset `: Remove the volumes of a Docker Compose service. +- `devservices purge `: Remove all devservices state, cached dependencies, containers, and volumes. If you pass a service name, `purge` asks for confirmation and removes only that service. +- `devservices update`: Update devservices to the latest version. -The recommended way to install devservices is through a virtualenv in the requirements.txt. Once that is installed and a devservices config file is added, you should be able to run `devservices up` to begin local development. +The global `-c ` flag (`--config`) loads a config file from another path. You cannot combine `-c` with a service name. -``` -devservices==1.2.4 -``` +## Environment variables -### 2. Add devservices config files +- `GITHUB_TOKEN` or `GH_TOKEN`: Token for downloading remote dependencies from GitHub. Without one, `devservices` uses `gh auth token`, then falls back to rate-limited unauthenticated downloads. +- `DEVSERVICES_DISABLE_SENTRY=1`: Stop sending errors and traces to Sentry. -Each repo should have a `devservices` directory with a `config.yml` file. This file is used to define services, dependencies, and modes. Other files and subdirectories in the `devservices` directory are optional and can be most commonly used for volume mounts. +## Configuration -The configuration file is a yaml file that looks like this: +`devservices/config.yml` is a Docker Compose file with an `x-sentry-service-config` block and an optional `x-programs` block. Other files in the `devservices` directory are optional. You usually use them for volume mounts. ```yaml -# This is a yaml block that holds devservices specific configuration settings. This is comprised of a few main sections: -# - version: The version of the devservices config file. This is used to ensure compatibility between devservices and the config file. -# - service_name: The name of the service. This is used to identify the service in the config file. -# - dependencies: A list of dependencies for the service. Each dependency is a yaml block that holds the dependency configuration. There are two types of dependencies: -# - local: A dependency that is defined in the config file. These dependencies do not have a remote field and must correspond to either a service defined in the 'services' section or a program defined in the 'x-programs' section. -# - remote: A dependency that is defined in the devservices directory in a remote repository. These configs are automatically fetched from the remote repository and installed. Any dependency with a remote field will be treated as a remote dependency. Example: https://github.com/getsentry/snuba/blob/59a5258ccbb502827ebc1d3b1bf80c607a3301bf/devservices/config.yml#L8 -# - modes: A list of modes for the service. Each mode includes a list of dependencies that are used in that mode. -# - healthcheck_timeout: Optional. The number of seconds to wait for all containers to become healthy before failing. Defaults to 180. +# devservices settings: +# - version: Config file version. The only valid value is 0.1. +# - service_name: The name you pass to commands such as `devservices up `. +# - dependencies: Each dependency accepts only `description` and `remote`. +# - A dependency without `remote` must match a service in `services` or a program in `x-programs`. +# - A dependency with `remote` is fetched from the `devservices` directory of that repository. Example: https://github.com/getsentry/snuba/blob/59a5258ccbb502827ebc1d3b1bf80c607a3301bf/devservices/config.yml#L8 +# - modes: Each mode lists the dependencies it runs. A `default` mode is required. +# - healthcheck_timeout: Optional. Seconds to wait for all containers to become healthy. Defaults to 180. x-sentry-service-config: version: 0.1 service_name: example-service @@ -65,49 +69,29 @@ x-sentry-service-config: repo_name: example-repository branch: main repo_link: https://github.com/getsentry/example-repository.git - mode: default # Optional field, mode to run remote dependency in that defaults to `default` + mode: default # Optional. The remote dependency's mode. Defaults to `default`. modes: default: [example-dependency-1, example-remote-dependency] custom-mode: [example-dependency-1, example-dependency-2, example-remote-dependency] -# This block defines supervisor programs that can be managed by devservices. -# Programs defined here are managed using Python's supervisor package and can be: -# - Started and stopped along with docker services when running `devservices up/down` -# - Run in the foreground for interactive debugging using `devservices foreground ` -# - Used as development server with `devservices serve` (requires a program named "devserver") -# -# These are particularly useful for: -# - Defining a devserver to run locally -# - Background workers or task processors -# - Any process that needs to be managed alongside your docker services +# Programs that devservices runs with supervisor. List a program in `dependencies` to start and stop it with `devservices up` and `devservices down`. +# - `devservices foreground ` runs a program in the foreground. +# - `devservices serve` runs the program named `devserver`. x-programs: - # Example devserver devserver: - # Required: The command to execute + # Required. command: "python manage.py run_server" - # Optional: You can also specify any of the supervisor program settings defined here: https://supervisord.org/configuration.html#program-x-section-settings + # Optional: any supervisor program setting from https://supervisord.org/configuration.html#program-x-section-settings - # Example background worker example-worker: command: "python manage.py worker" -# This will be a standard block used by docker compose to define dependencies. -# -# The following fields are important to all dependencies: -# - image: The docker image to use for the dependency. -# - ports: The ports to expose for the dependency. Please only expose ports to localhost(127.0.0.1) -# - healthcheck: The docker healthcheck to use for the dependency. -# - environment: The environment variables to set for the dependency. -# - extra_hosts: The extra hosts to add to the dependency. -# - networks: The networks to add to the dependency. In order for devservices to work properly, the dependency must be on the `devservices` network. -# - labels: The labels to add to the dependency. The `orchestrator=devservices` label is required for devservices to determine a container is managed by devservices. -# - restart: The restart policy to use for the dependency. -# -# These fields are optional: -# - ulimits: The ulimits to set for the dependency. This is useful for setting resource constraints for the dependency. -# - volumes: The volumes to mount for the dependency. This is useful for mounting data volumes for the dependency if data should be persisted between runs. It can also be useful to use a bind mount to mount a local directory into a container. Example of bind mounting clickhouse configs from a local directory https://github.com/getsentry/snuba/blob/59a5258ccbb502827ebc1d3b1bf80c607a3301bf/devservices/config.yml#L44 -# - command: The command to run for the dependency. This can override the default command for the docker image. -# For more information on the docker compose file services block, see the docker compose file reference: https://docs.docker.com/reference/compose-file/services/ +# Standard Docker Compose services: https://docs.docker.com/reference/compose-file/services/ +# - Publish ports on 127.0.0.1 only. +# - Put every service on the `devservices` network. +# - Add the `orchestrator=devservices` label. `devservices purge` and `devservices reset` find containers by this label. +# - Add a healthcheck. `devservices up` waits for containers to become healthy. +# - Use `volumes` to keep data between runs or to bind mount local files. Example: https://github.com/getsentry/snuba/blob/59a5258ccbb502827ebc1d3b1bf80c607a3301bf/devservices/config.yml#L44 services: example-dependency-1: image: ghcr.io/getsentry/example-dependency-1:1.0.0 @@ -125,7 +109,7 @@ services: volumes: - example-dependency-1-data:/var/lib/example-dependency-1 restart: unless-stopped - # Everything below this line is required for devservices to work properly. + # devservices requires the fields below. ports: - 127.0.0.1:1234:1234 extra_hosts: @@ -146,7 +130,7 @@ services: environment: EXAMPLE_ENV_VAR: example-value restart: unless-stopped - # Everything below this line is required for devservices to work properly. + # devservices requires the fields below. ports: - 127.0.0.1:2345:2345 extra_hosts: @@ -156,23 +140,22 @@ services: labels: - orchestrator=devservices -# This is a standard block used by docker compose to define volumes. -# For more information, see the docker compose file reference: https://docs.docker.com/reference/compose-file/volumes/ +# Standard Docker Compose volumes: https://docs.docker.com/reference/compose-file/volumes/ volumes: example-dependency-1-data: -# This is a standard block used by docker compose to define networks. Defining the devservices network is required for devservices to work properly. By default, devservices will create an external network called `devservices` that is used to connect all dependencies. -# For more information, see the docker compose file reference: https://docs.docker.com/reference/compose-file/networks/ +# The external `devservices` network is required. `devservices up` creates it if it does not exist. +# Docker Compose networks reference: https://docs.docker.com/reference/compose-file/networks/ networks: devservices: name: devservices external: true ``` -## Dev +## Get help -```sh -uv sync -direnv allow -pytest -n4 -``` +Open an issue in [getsentry/devservices](https://github.com/getsentry/devservices/issues). + +## License + +See [LICENSE.md](LICENSE.md). diff --git a/devservices/commands/foreground.py b/devservices/commands/foreground.py index 95a9279..cb62a02 100644 --- a/devservices/commands/foreground.py +++ b/devservices/commands/foreground.py @@ -109,13 +109,11 @@ def foreground(args: Namespace) -> None: return try: - # Stop the supervisor process before running in foreground console.info(f"Stopping {program_name} in supervisor") manager.stop_process(program_name) console.info(f"Starting {program_name} in foreground") argv = shlex.split(program_command) - # Run the process in foreground pty.spawn(argv) except SupervisorProcessError as e: capture_exception(e) diff --git a/devservices/commands/list_services.py b/devservices/commands/list_services.py index 67ef1c2..9db183b 100644 --- a/devservices/commands/list_services.py +++ b/devservices/commands/list_services.py @@ -30,7 +30,6 @@ def add_parser(subparsers: _SubParsersAction[ArgumentParser]) -> None: def list_services(args: Namespace) -> None: """List the services installed locally.""" console = Console() - # Get all of the services installed locally coderoot = get_coderoot() try: services = get_local_services(coderoot) diff --git a/devservices/commands/logs.py b/devservices/commands/logs.py index 75d61b1..7a40f58 100644 --- a/devservices/commands/logs.py +++ b/devservices/commands/logs.py @@ -118,7 +118,6 @@ def logs(args: Namespace) -> None: if log_stdout is not None: console.info(log_stdout) - # Get supervisor program logs supervisor_programs = [ dep for dep in mode_dependencies diff --git a/devservices/commands/status.py b/devservices/commands/status.py index e0da7be..5c60a8c 100644 --- a/devservices/commands/status.py +++ b/devservices/commands/status.py @@ -333,7 +333,6 @@ def generate_service_status_details( ) -> str: output = [f"{indentation}{Color.BOLD}{dependency.name}{Color.RESET}:"] - # Handle supervisor dependencies if dependency.dependency_type == DependencyType.SUPERVISOR: return generate_supervisor_status_details( dependency, process_statuses, indentation diff --git a/devservices/commands/up.py b/devservices/commands/up.py index 91e4e46..54bbf32 100644 --- a/devservices/commands/up.py +++ b/devservices/commands/up.py @@ -174,7 +174,6 @@ def up(args: Namespace, existing_status: Status | None = None) -> None: service, mode, status, offline ) _create_devservices_network() - # Add the service to the starting services table state.update_service_entry(service.name, mode, StateTables.STARTING_SERVICES) mode_dependencies = modes[mode] for service_with_local_runtime in services_with_local_runtime: diff --git a/devservices/main.py b/devservices/main.py index 9f99788..d7bbac7 100644 --- a/devservices/main.py +++ b/devservices/main.py @@ -158,7 +158,6 @@ def main() -> None: subparsers = parser.add_subparsers(dest="command", title="commands", metavar="") - # Add subparsers for each command up.add_parser(subparsers) down.add_parser(subparsers) list_dependencies.add_parser(subparsers) @@ -174,13 +173,11 @@ def main() -> None: args = parser.parse_args() - # If the command has a debug flag, set the logger to debug if "debug" in args and args.debug: logger = logging.getLogger(LOGGER_NAME) logger.setLevel(logging.DEBUG) if args.command: - # Call the appropriate function based on the command with start_transaction(op="command", name=args.command): args.func(args) else: diff --git a/devservices/utils/docker.py b/devservices/utils/docker.py index 7eb08c4..f14fe32 100644 --- a/devservices/utils/docker.py +++ b/devservices/utils/docker.py @@ -71,7 +71,6 @@ def wait_for_healthy( ) start = time.time() while time.time() - start < timeout: - # Run docker inspect to get the container's health status try: # For containers with no healthchecks, the output will be "unknown" result = subprocess.check_output( @@ -102,7 +101,6 @@ def wait_for_healthy( ) return - # If not healthy, wait and try again time.sleep(HEALTHCHECK_INTERVAL) raise ContainerHealthcheckFailedError(container.short_name, timeout) diff --git a/devservices/utils/docker_compose.py b/devservices/utils/docker_compose.py index 82138d6..c1f77de 100644 --- a/devservices/utils/docker_compose.py +++ b/devservices/utils/docker_compose.py @@ -42,7 +42,6 @@ class DockerComposeCommand(NamedTuple): def install_docker_compose() -> None: console = Console() - # Determine the platform system = platform.system() machine = platform.machine() @@ -61,7 +60,6 @@ def install_docker_compose() -> None: binary_name = "docker-compose" - # Determine the download URL based on the platform if system == "Linux": binary_name_with_extension = f"docker-compose-linux-{arch}" elif system == "Darwin": @@ -87,7 +85,6 @@ def install_docker_compose() -> None: f"Docker Compose {DOCKER_COMPOSE_INSTALL_VERSION} installed successfully to {destination}" ) - # Verify the installation try: version = get_docker_compose_version() except DockerComposeError as e: @@ -135,7 +132,6 @@ def get_container_names_for_project( def check_docker_compose_version() -> None: console = Console() - # Throw an error if docker daemon isn't running check_docker_daemon_running() try: version_output = get_docker_compose_version() diff --git a/devservices/utils/supervisor.py b/devservices/utils/supervisor.py index dd80d6f..578bf88 100644 --- a/devservices/utils/supervisor.py +++ b/devservices/utils/supervisor.py @@ -141,7 +141,6 @@ def __init__( DEVSERVICES_SUPERVISOR_CONFIG_DIR, f"{service_name}.sock" ) - # Load service config and extract x-programs data if os.path.exists(service_config_path): with open(service_config_path, "r", encoding="utf-8") as stream: config = yaml.safe_load(stream) @@ -155,18 +154,15 @@ def __init__( self.has_programs = len(programs_data.keys()) > 0 - # Generate supervisor config file from x-programs data self.config_file_path = self._generate_config_from_programs_data(programs_data) def _generate_config_from_programs_data(self, programs_data: ProgramData) -> str: config = configparser.ConfigParser() - # Add program sections for program_name, program_config in programs_data.items(): section_name = f"program:{program_name}" config[section_name] = {} - # Apply defaults for any missing configuration values program_config_with_defaults = { **SUPERVISOR_PROGRAM_DEFAULTS, **program_config, @@ -180,17 +176,14 @@ def _generate_config_from_programs_data(self, programs_data: ProgramData) -> str os.makedirs(DEVSERVICES_SUPERVISOR_CONFIG_DIR, exist_ok=True) - # Set unix http server to use the socket path config["unix_http_server"] = {"file": self.socket_path} - # Set generated pidfile to use service name config["supervisord"] = { "pidfile": os.path.join( DEVSERVICES_SUPERVISOR_CONFIG_DIR, f"{self.service_name}.pid" ) } - # Set supervisorctl to use the socket path config["supervisorctl"] = {"serverurl": f"unix://{self.socket_path}"} # Required by supervisor to work properly @@ -384,7 +377,6 @@ def tail_program_logs(self, program_name: str) -> None: return try: - # Use supervisorctl tail -f command to follow logs subprocess.run( [ "supervisorctl", @@ -404,7 +396,6 @@ def tail_program_logs(self, program_name: str) -> None: def get_all_process_info(self) -> dict[str, ProcessInfo]: """Get status information for all supervisor programs.""" - # Check if supervisor client is up first, return empty list if down if not self.has_programs: return {} @@ -434,7 +425,6 @@ def get_all_process_info(self) -> dict[str, ProcessInfo]: if not isinstance(process_info, dict): continue - # Extract basic fields with defaults name = process_info.get("name", "") state = process_info.get("state", SupervisorProcessState.UNKNOWN) state_name = SupervisorProcessState(state).name