Remove Flink, Arroyo, asap-summary-ingest, and legacy Kafka-ingest infra from asap-tools #709

Summary

Remove Flink, Arroyo, asap-summary-ingest, and the legacy Kafka-ingest path (streaming_engine=flink|arroyo, use_kafka_ingest) from asap-tools and the repo root. precompute (HTTP remote write) is the only supported streaming engine going forward.

Directories to delete entirely

• asap-summary-ingest/ (repo root — the Arroyo pipeline project)
• asap-tools/installation/arroyo/, asap-tools/installation/flink/, asap-tools/installation/prometheus_kafka_adapter/
• asap-tools/execution-utilities/asap_query_latency/
• asap-tools/execution-utilities/asap_benchmark_pipeline/
• asap-tools/execution-utilities/high-throughput-kafka-producer/, high-throughput-kafka-producer-single-threaded/
• asap-tools/execution-utilities/kafka-consumer/, kafka_toy_data_generator/
• asap-tools/execution-utilities/clickhouse-benchmark-pipeline/
• asap-common/dependencies/py/promql_utilities/promql_utilities/streaming_config/ (MetricConfig, SQLTableConfig, StreamingAggregationConfig) — its only consumer is asap-summary-ingest/run_arroyosketch.py and that package's own tests; dead once asap-summary-ingest/ is deleted
• asap-common/dependencies/py/promql_utilities/promql_utilities/query_logics/ (logics.py, parsing.py, enums.py) — already has zero importers anywhere in the Python codebase today (confirmed by audit), independent of the Arroyo removal. Note: the Rust crate's query_logics module (asap-common/dependencies/rs/promql_utilities/src/query_logics/) is heavily used by asap-planner-rs/asap-query-engine/asap_types and is out of scope — only the Python port is dead.

Files to delete entirely

• asap-tools/docker/arroyo-cli-compose.yml
• asap-tools/experiments/experiment_utils/services/flink.py
• asap-tools/experiments/experiment_utils/services/arroyo.py
• asap-tools/experiments/experiment_utils/services/arroyo_throughput_monitor.py
• asap-tools/experiments/experiment_utils/services/prometheus_kafka_adapter.py
• asap-tools/experiments/experiment_utils/services/kafka.py
• asap-tools/experiments/generate_arroyosketch_compose.py
• asap-tools/experiments/dumb_kafka_consumer_arroyo.py
• asap-tools/experiments/arroyo_throughput_monitor.py
• asap-tools/experiments/config/overrides/arroyo.yaml
• asap-tools/installation/recompile_flink.sh
• asap-tools/installation/empty_flink_logs.sh

Files needing edits (not full deletes)

• asap-tools/components.conf — drop asap-summary-ingest line
• asap-tools/installation/install_external_components.sh — remove "flink", "arroyo" from PREDEFINED_COMPONENTS
• asap-tools/docker/asap-docker-compose.yml — drop the arroyo-cli-compose.yml / arroyosketch-compose.yml includes
• asap-tools/experiments/constants.py — drop FLINK_INPUT_TOPIC, FLINK_OUTPUT_TOPIC, KAFKA_BROKER, ARROYO_IMAGE, ARROYO_THROUGHPUT_POLLING_INTERVAL, AVOID_RUN_ARROYOSKETCH_LONG_SSH
• asap-tools/experiments/experiment_utils/services/__init__.py — remove Kafka/Flink/Arroyo/PrometheusKafkaAdapter/DumbKafkaConsumer imports
• asap-tools/experiments/experiment_utils/services/query_engine.py — drop streaming_engine="arroyo"/kafka-ingest branch and kafka_broker/flink_output_format params, keep only precompute
• asap-tools/experiments/experiment_run_e2e.py, experiment_teardown_everything.py, experiment_only_ingest_path.py, experiment_run_grafana_demo.py, experiment_run_clickhouse.py — strip Kafka/Flink/Arroyo service setup, teardown, and use_kafka_ingest/streaming_engine=flink|arroyo branches
• asap-tools/experiments/generate_controller_compose.py — drop "flink"/"arroyo" from streaming_engine choices
• asap-tools/experiments/experiment_utils/config.py — remove flink/arroyo/kafka config validation branches
• asap-tools/experiments/experiment_utils/services/misc.py — remove DumbKafkaConsumerService
• asap-tools/experiments/experiment_utils/providers/local.py — remove flink/arroyo-specific provider hooks
• asap-tools/experiments/config/config.yaml — drop use_kafka_ingest
• asap-tools/experiments/config/experiment_type/ingest_path_v2.yaml — check for kafka/flink knobs
• asap-tools/README.md, asap-tools/docs/architecture.md, configuration.md, deployment.md, troubleshooting.md, experiments/CONFIG_PARAMETERS_REFERENCE.md, experiments/HYDRA_CONFIG_USAGE.md — rewrite; these document Flink/Arroyo as first-class streaming engines and the Kafka ingest data flow
• Top-level docs/02-components/README.md, docs/01-getting-started/architecture.md, docs/README.md, docs/03-how-to-guides/** — also reference asap-summary-ingest

Out of scope, needs a separate decision

• asap-query-engine/src/precompute_engine/worker.rs has a real parsing path, deserialize_from_json_arroyo, plus tests asserting Arroyo-output equivalence. Removing Arroyo tooling doesn't automatically mean dropping this format support — decide separately.

🤖 Generated with Claude Code