docs: restructure how-to and tutorial pages

docs/how-to/add-plans-and-devices.md

@@ -25,6 +25,8 @@ To add plans, you would add the following into your configuration file:
 :language: yaml
 ```
 
+For BlueAPI to detect plans from repositories and be registered, they must be added to the `__init__.py` file with `__all__` = [<plan_name>, ...].
+
 
 Devices are added similarly, using `deviceManager` as the `kind`, like so:
 ```{literalinclude} ../../tests/unit_tests/valid_example_config/plans_and_devices.yaml

docs/how-to/integrate-with-numtracker.md

@@ -6,7 +6,7 @@
 
 You will need to [get numtracker itself configured for your instrument](https://github.com/DiamondLightSource/numtracker/wiki/new_beamline). 
 
-Blueapi needs [valid authentication configured](./authenticate.md) to communicate with numtracker. It will propogate its auth token so both blueapi and numtracker should be aware of who the user is and that they have permission to be on the instrument sessions (visits) that are intended for use.
+Blueapi needs [valid authentication configured](../how-to/authenticate.md) to communicate with numtracker. It will propogate its auth token so blueapi is aware of who the user is and that they have permission to be on the instrument sessions (visits) that are intended for use. Numtracker will not be aware of the user. 
 
 ## Configuration
 

docs/tutorials/local-auth.md → docs/how-to/local-auth.md

@@ -1,6 +1,6 @@
-# Run Auth Locally
+# Run BlueAPI and connect to services locally
 
-BlueAPI can be secured using OIDC authenticaiton. For development it can be useful to run a containerised version of the OIDC stack, to serve a local instance of BlueAPI.
+For development purposes, it can be useful to run BlueAPI and adjacent services (Numtracker, Tiled, OPA etc.) locally. To do this, a containerised version of the stack should be run. This includes launching an instance of NumTracker, RabbitMQ, Keycloak, Tiled, OPA and a number of IOCs, in detached mode. Running BlueAPI and connected services locally as opposed to connecting to the cluster can be useful for learning more about the stack, running system tests, checking if changes during development propagate as expected etc. 
 
 To run the stack:
 
@@ -18,6 +18,8 @@ To run the stack:
 > alias docker=podman
 > ```
 
+There are also docs in the [Developer Guide](https://dev-guide.diamond.ac.uk/epics-containers/reference/setup/#docker-compose) to set up docker-compose on DLS workstations.
+
 3. Run `source tests/system_tests/.env` which will set required EPICS environmental variables
 4. Run `blueapi -c tests/system_tests/config.yaml serve` to launch BlueAPI configured to use the launched stack. This may take a while, as BlueAPI will attempt to connect to a number of devices via Channel Access
 

docs/how-to/run-container.md

@@ -19,3 +19,12 @@ podman run ghcr.io/diamondlightsource/blueapi:latest
 ```
 
 To get a released version, use a numbered release instead of `latest`.
+
+## Starting the message bus
+
+The worker requires a running instance of the message bus, which in this case is RabbitMQ. The easiest way to start it is
+ to `compose` the services in `tests/system_tests/compose.yaml`
+
+```sh
+docker compose -f tests/system_tests/compose.yaml run rabbitmq --detach
+```

docs/tutorials/run-with-dev.md → docs/how-to/run-with-dev.md

@@ -10,7 +10,7 @@ Ensure you are inside your virtual environment:
 source venv/bin/activate
 ```
 
-You will need to follow the instructions for setting up RabbitMQ as in [instructions for setting up RabbitMQ](run-bus.md).
+You will need to follow the instructions for setting up RabbitMQ as in [instructions for setting up RabbitMQ](../tutorials/5.%20config-add-rabbitmq-and-run-plan-cli.md).
 
 The worker will be available from the command line (`blueapi serve`), but can be started from vscode with additional
 debugging capabilities.

docs/how-to/use-persistent-volumes.md

@@ -5,7 +5,7 @@ BlueAPI can use persistent volumes as a scratch area. This allows the user to re
 ## Configuration
 
 :::{seealso}
-[Configure the Application](./configure-app.md)
+[Configure the Application](../how-to/configure-app.md)
 :::
 
 The relevant configuration is below:

docs/how-to/write-devices.md

@@ -23,5 +23,5 @@ The return type annotation `-> MyTypeOfDetector` is required as blueapi uses it
 The device is created via a function rather than a global to preserve side-effect-free imports. Each device must have its own factory function.
 
 :::{seealso}
-[Numtracker integration](./integrate-with-numtracker.md) for how to configure where detectors write files.
+[Numtracker integration](../how-to/integrate-with-numtracker.md) for how to configure where detectors write files.
 :::

docs/resources/enabled_plugins

@@ -0,0 +1 @@
+[rabbitmq_stomp].

docs/tutorials/installation.md → docs/tutorials/1. installation.md

@@ -1,4 +1,4 @@
-# Installation
+# 1. Installation
 
 ## Check your version of python
 
@@ -8,6 +8,10 @@ typing into a terminal:
 python3 --version
 ```
 
+## Clone the blueapi repository locally 
+
+Clone the blueapi repository to your local scratch directory and `cd` into it.
+
 ## Create a virtual environment
 
 It is recommended that you install into a “virtual environment” so this

docs/tutorials/2. quickstart.md

@@ -0,0 +1,39 @@
+# 2. Quickstart guide
+
+Blueapi acts as a worker that can run Bluesky plans against devices for a specific laboratory setup. It can control devices to collect data and export events to tell downstream services about the data it has collected.
+
+You should be in the blueapi directory that you have cloned. From here, if you open VSCode, you will probably be prompted to 'Reopen in Container' which you can do and then run the following commands to start the worker. 
+
+If you not, you can still run the following from the venv created in page 1 but you may need to run the command below to have the dependencies required for this tutorial.
+
+```
+pip install .[demo]
+```
+
+## Start Worker
+
+To start the worker:
+```
+blueapi serve
+```
+This will use all the defaults in `src/blueapi/config.py`
+
+## Basic Introspection
+
+Blueapi comes with a minimal CLI client for basic control/debugging. We can use this to see which plans and devices are available. In a new terminal window, run the following commands. 
+
+If working in a venv, you may need to reactivate it. To do this, assuming you are still in the blueapi repo in your terminal, use:
+```
+source .venv/bin/activate
+```
+
+Now, run the following to see what plans and devices are available:
+
+``` 
+blueapi controller plans
+blueapi controller devices
+```
+
+We should now have an empty context.
+
+See also [Full CLI reference](../reference/cli.md)

docs/tutorials/3. config-add-devices.md

@@ -0,0 +1,28 @@
+# 3. Config add devices
+
+Blueapi is highlighly configurable. The config details aspects that impact how Blueapi works and it is customisable. 
+
+As a first step, create a `config.yaml` file. This would be best placed in the `docs/resources` folder. 
+
+Intially, add devices using the following code:
+
+```yaml
+env:
+  metadata:
+    instrument: demo
+  sources:
+    - kind: deviceManager
+      module: blueapi.tutorial.devices
+```
+
+The `env` section of the `config.yaml` file lets you create and set environment variables. The `demo` points to optional dependencies in the [pyproject.toml file](../../pyproject.toml). The module above points to the devices you will use for this tutorial which can be found [here](../../src/blueapi/tutorial/devices.py).
+
+Save the file and restart the worker to use this new `config.yaml` file you have created. To do this, navigate back to the terminal window where the worker is running, CTRL+C to close it then run the following command: 
+``` 
+blueapi --config docs/resources/config.yaml serve 
+```
+
+The above command asuumes you created the `config.yaml` file in `docs/resources`. Else, replace `docs/resources/config.yaml` with whichever file path your `config.yaml` has.
+
+Now run 
+```blueapi controller devices``` in the same terminal window as you did when previously running this command and you should see `det` and `stage`.

docs/tutorials/4. config-add-plans.md

@@ -0,0 +1,16 @@
+# 4. Config add plans
+
+As well as adding devices to the `config.yaml`, you can add plans. Under the code you already have, add the following:
+
+``` yaml
+    - kind: planFunctions
+      module: dodal.plans
+    - kind: planFunctions
+      module: dodal.plan_stubs.wrapped
+```
+These plans that will be used in this tutorial can be found in the [dodal repository](https://github.com/DiamondLightSource/dodal/tree/b4b38f1e0cf480faeb29d67695440c45a148f169). Specifically, in `dodal/src/dodal/plans/wrapped.py` and `dodal/src/dodal/plan_stubs/wrapped.py`.
+
+Save the changes you've made to the `config.yaml` file. Navigate back to the terminal where the blueapi worker is running. Restart the server to use this updated `config.yaml` file.
+
+Now run 
+```blueapi controller plans``` in the same terminal window used previously for this command and you should see a list of plans that can be run. For example, `count`, `list_grid_scan`, `num_scan` etc.

docs/tutorials/5. config-add-rabbitmq-and-run-plan-cli.md

@@ -0,0 +1,92 @@
+# 5. Run a Plan
+
+Now that we have a running worker and we have added plans and devices to the `config.yaml` file, we should be able to run a plan. 
+
+## Example Plans
+Run the following command in the terminal. (Same terminal where you ran `blueapi controller devices/plans`).
+
+Run a Sleep plan
+```
+blueapi -c config.yaml controller run -i cm12345-1 sleep '{"time": 5}'
+```
+You may get this error in the terminal logs: 
+```
+Error: Stomp configuration required to run plans is missing or disabled
+```
+
+We need to enable stomp in our `config.yaml` file. Add the following lines to your `config.yaml`. This should not be within the `env` section, but should rather be its own section.
+
+```yaml
+stomp:
+  enabled: true
+  url: tcp://localhost:61613/
+```
+
+Your overall config.yaml should now look like:
+
+```yaml
+env:
+  metadata:
+    instrument: demo
+  sources:
+    - kind: deviceManager
+      module: blueapi.tutorial.devices
+    - kind: planFunctions
+      module: dodal.plans
+    - kind: planFunctions
+      module: dodal.plan_stubs.wrapped
+stomp:
+  enabled: true
+  url: tcp://localhost:61613/
+```
+
+If you had difficulties creating your `config.yaml` file, you can use the pre-made one in the `docs/resources` folder. 
+
+Try running the Sleep plan again. You may come across this error:
+
+```
+WARNING stomp.py could not connect to host localhost, port 61613: [Errno 111] Connection refused
+```
+To run a plan in blocking mode, we need the message bus to be started. Blueapi can publish updates to this message bus asynchronously and then the CLI can view these updates and display them to the user. Run the following command in a new terminal window and restart the blueapi server.
+
+```sh
+echo "[rabbitmq_stomp].">enabled_plugins && podman run -it --rm --name rabbitmq-docs -v $(readlink -f enabled_plugins):/etc/rabbitmq/enabled_plugins:z -p 5672:5672 -p 61613:61613 rabbitmq:latest
+``` 
+
+Now, try running the Sleep plan. You should 'Plan succeeded' in the terminal logs of the terminal window where the blueapi server is running.
+
+Below are some other plans you can try. In the terminal window where the Sleep plan was run, you should see 
+```
+INFO root Stomp client subscribing to name='public.worker.event'
+```
+
+Press CTLR+C to disconnect the Stomp client then run a new plan.
+
+Run a Scan
+```
+blueapi -c config.yaml controller run count '{"detectors": ["det"], "num":5}' -i cm1234-5
+
+```
+Run a Step Scan
+```
+blueapi -c config.yaml controller run step_scan '{"detectors":["det"], "params":[["stage.x",[0,5,1]]]}' -i cm1234-5
+```
+
+## Adding extra components to the configuration
+
+Having added the minimal configuration required to run a plan, we can add optional components. For example: 
+
+The CLI uses REST requests talk to the worker via a message broker on `tcp://localhost:61613`,
+but you can customize this via the `config.yaml` file.
+
+```{literalinclude} ../../tests/unit_tests/valid_example_config/client.yaml
+:language: yaml
+```
+
+Then run
+
+```
+blueapi -c custom-address.yaml controller plans
+```
+
+Other components that can be added to the config are detailed in the the [How-to-Guide section](../how-to/).

docs/tutorials/6. run-plan-docs.md

@@ -0,0 +1,75 @@
+# 6. Run a Plan from Docs page
+
+As well as running plans through the CLI, it is possible to run plans using the docs page. This is an easy way to run one plan but if you need to run multiple plans, it may be easier to use the CLI.
+
+In the terminal window where the worker is running, scroll to this line in the logs:
+
+```
+INFO uvicorn.error Uvicorn running on http://localhost:8000 (Press CTRL+C to quit)
+```
+Open the link (ctrl+click) and this should take you to the BlueAPI Control docs page. It should look like this:
+
+
+![BlueAPI docs page](docs_homepage.png)
+
+Scrolling down will show you endpoints grouped. The different groups are described below. 
+
+## Definitions:
+- Plan: a set of instructions for one aspect of experiment orchestration. More details can be found [here](https://blueskyproject.io/bluesky/v1.13.1rc1/plans.html)
+- Task: one individual instance of the plan being run. 
+- Device: hardware defined via ophyd protocols. For more information visit [dodal](https://github.com/DiamondLightSource/dodal) and its [Glossary page](https://diamondlightsource.github.io/dodal/main/reference/glossary.html).
+- Environment - definition already provided under the endpoint. 
+- Meta - definition already provided under the endpoint.
+
+## Steps for running a plan
+1. Find available devices. 
+
+The first recommended step is to find out what devices are available on the instrument. Scroll down to the 'Get Devices' endpoint (/api/v1/devices) and press the downwards arrow which should expand it to show the 'Try it out' button. Click the 'Try it out' button to enable further interaction. 
+
+![Find devices endpoint](find_devices.png)
+
+Next, press the 'Execute' button and scroll down to 'Responses' where you should see available devices (e.g. 'det' in the example below). Take note of the device name for later use.
+
+![Show devices](show_devices.png)
+
+2. Create a task using one of the available devices.
+
+Scroll back up to the 'Submit Task' endpoint (/api/v1/tasks). The default setting that should appear in the request body is the example of a 'count' task using detector 'x' and instrument session 'cm12345-1'. 
+
+![Submit task default](submit_task.png)
+
+Press the 'Try it out' button and replace the placeholder 'x' in the request body with the device you took note of in step 1 ('det' in this example). The instrument session can be left as 'cm12345-1' for the purposes of this tutorial.
+
+![Submit task with details](add_detector.png)
+
+Press 'Execute' and you should receive a '201' response that contains a 'task_id'. At the end of the response body box, press the clipboard symbol to copy this 'task_id'. 
+
+![Execute task](submit_task.png)
+
+3. Set created task in the previous step to be the active task 
+
+After creating a task in Step 2, it still needs to be set to be the active task. Scroll down to the 'Set Active Task' endpoint (/api/v1/worker/task). The default for 'task_id' should be 'string'.
+
+![Set active task default](create_task_default.png)
+
+Press 'Try it out' and paste the copied 'task_id' from earlier in the 'Request' body.
+
+![Set active task](set_active_task.png)
+
+Press the 'Execute' button and scroll down to 'Responses' and check if you got a '200 Successful Response'. The BlueAPI docs page only provides feedback for submitting the task to be the active task. To check if the plan actually ran successfully, check the logs (e.g. through ArgoCD, see below).
+
+4. Check the logs in the terminal window where the blueapi server is running
+
+To confirm if the plan ran successfully check the terminal logs. You should see something like: 
+
+```
+INFO blueapi.worker.task_worker Task ran successfully - returned: None
+```
+
+## Troubleshooting
+- 401 Response/cannot Execute plan - reload whole BlueAPI docs web page and log-in again using Keycloak. 
+- 2XX Task was created but could not be run successfully, no response - check logs for correct instrument session, have motor limits been exceeded?
+- New plan changes have been pushed to the repo but the plan isn't showing up 
+    - May need to pull through PlanDev and reload the environment
+    - Make sure the blueapi pod is pointing to the correct commit in ArgoCD (check values.yaml file in pxx-depolyment repo which is accessed through [epics-containers] (https://github.com/epics-containers) ). 
+        - Update values.yaml as needed, re-submit and re-create a task using BlueAPI and run again using same steps as above. 

docs/tutorials/scripting-plans.md → docs/tutorials/7. scripting-plans.md

@@ -1,4 +1,4 @@
-# Scripting Plans
+# 7. Scripting Plans
 
 While the CLI can be used to query devices and run plans, it can be useful to
 combine multiple plans within a better interface than bash/shell scripting.