Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .prettierrc
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
"files": "*.md",
"options": {
"parser": "markdown",
"proseWrap": "always"
"proseWrap": "never"
}
}
]
Expand Down
25 changes: 7 additions & 18 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,12 @@

A monorepo for managing signed data. Consists of:

- [signed-api](./packages/signed-api/README.md) - A service for storing and accessing signed data. It provides endpoints
to handle signed data for a specific airnode.
- [signed-api](./packages/signed-api/README.md) - A service for storing and accessing signed data. It provides endpoints to handle signed data for a specific airnode.
- [airnode-feed](./packages/airnode-feed/README.md) - A service for pushing data provider signed data.
- [e2e](./packages/e2e/README.md) - End to end test utilizing Mock API, Airnode feed and signed API.
- [performance-test](./packages/performance-test/README.md) - Configurations and scripts to allow running performance
tests and benchmarks on AWS.
- [performance-test](./packages/performance-test/README.md) - Configurations and scripts to allow running performance tests and benchmarks on AWS.

Read the
[specification](https://docs.google.com/document/d/1-kUPIXSD4ZW1SGs_P8HsejC9k9aHB-NXs9_6-OclnmE/edit#heading=h.i307237rdfda)
for more details on the architecture and the components.
Read the [specification](https://docs.google.com/document/d/1-kUPIXSD4ZW1SGs_P8HsejC9k9aHB-NXs9_6-OclnmE/edit#heading=h.i307237rdfda) for more details on the architecture and the components.

## Getting started

Expand All @@ -27,22 +23,15 @@ and to build the packages:
pnpm run build
```

Note, that every time you make a change to a workspace that is used as a dependency of another, you need to rebuild the
changed package (otherwise you might get weird JS/TS errors).
Note, that every time you make a change to a workspace that is used as a dependency of another, you need to rebuild the changed package (otherwise you might get weird JS/TS errors).

## Versioning and release

Signed API and Airnode feed use [semantic versioning](https://semver.org/). Packages are published to NPM to export the
configuration schemas and various utilities. The services are dockerized and published to Docker Hub.
Signed API and Airnode feed use [semantic versioning](https://semver.org/). Packages are published to NPM to export the configuration schemas and various utilities. The services are dockerized and published to Docker Hub.

There is a script that automates the process of creating new NPM packages and Docker images. Full release procedure:

1. `pnpm run create-release:npm [major|minor|patch]` - The script ensures publishing happens from up-to-date `main`
branch. It updates the package versions for `airnode-feed` and `signed-api`, updates fixtures and example files, does
basic checks to ensure the changes are valid and creates a version commit. The command intentionally does not do the
publishing so that the changes can be reviewed before pushing.
1. `pnpm run create-release:npm [major|minor|patch]` - The script ensures publishing happens from up-to-date `main` branch. It updates the package versions for `airnode-feed` and `signed-api`, updates fixtures and example files, does basic checks to ensure the changes are valid and creates a version commit. The command intentionally does not do the publishing so that the changes can be reviewed before pushing.
2. `git show` - To inspect the changes of the version commit.
3. Run the e2e tests locally. This is not automated due to implementation complexity.
4. `git push` - Push the version commit upstream. This will trigger the `tag-and-release` GitHub Actions job and result
in 1) the commit being tagged with the new version, 2) the release being created on GitHub, 3) both packages being
released on npm, and 4) both Docker images being built and pushed to Docker Hub.
4. `git push` - Push the version commit upstream. This will trigger the `tag-and-release` GitHub Actions job and result in 1) the commit being tagged with the new version, 2) the release being created on GitHub, 3) both packages being released on npm, and 4) both Docker images being built and pushed to Docker Hub.
36 changes: 11 additions & 25 deletions packages/airnode-feed/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,10 @@

> A service for storing and accessing signed data.

Airnode feed is a Node.js service, dockerized and deployable on any cloud provider or hostable on premise. It is
continuously running two core loops:
Airnode feed is a Node.js service, dockerized and deployable on any cloud provider or hostable on premise. It is continuously running two core loops:

1. `Fetch beacon data` - Each `triggers.signedApiUpdates` entry defines a group of templates. Airnode feed makes a
template request to the API specified in the OIS to get the template data. Airnode feed's wallet is used to sign the
responses and these are then saved to in-memory storage.
2. `Push signed beacon data to signed API` - For each `triggers.signedApiUpdates`, periodically checks the in-memory
storage and pushes the signed data to the configured API.
1. `Fetch beacon data` - Each `triggers.signedApiUpdates` entry defines a group of templates. Airnode feed makes a template request to the API specified in the OIS to get the template data. Airnode feed's wallet is used to sign the responses and these are then saved to in-memory storage.
2. `Push signed beacon data to signed API` - For each `triggers.signedApiUpdates`, periodically checks the in-memory storage and pushes the signed data to the configured API.

## Configuration

Expand All @@ -24,20 +20,16 @@ The Airnode feed needs a configuration in order to run. The `config` folder cont

To start the the Airnode feed in dev mode run the following:

1. `cp config/airnode-feed.example.json config/airnode-feed.json` - To copy the Airnode feed configuration from the
example. Note, the `airnode-feed.json` file is ignored by git. If you are using Docker Desktop, you need to change
the URL from localhost to `host.docker.internal`. For example:
1. `cp config/airnode-feed.example.json config/airnode-feed.json` - To copy the Airnode feed configuration from the example. Note, the `airnode-feed.json` file is ignored by git. If you are using Docker Desktop, you need to change the URL from localhost to `host.docker.internal`. For example:

```jsonc
"url": "http://host.docker.internal:8090"
```

2. `cp config/secrets.example.env config/secrets.env` - To copy the secrets.env needed for the configuration. This file
is also ignored by git.
2. `cp config/secrets.example.env config/secrets.env` - To copy the secrets.env needed for the configuration. This file is also ignored by git.
3. Set the `NODARY_API_KEY` inside the secrets file. Ask someone from development team for the key.
4. `cp .env.example .env` - To copy the example environment variables. Optionally change the defaults.
5. `pnpm run dev` - To run the Airnode feed. This step assumes already running signed API as specified in the
`airnode-feed.json` configuration.
5. `pnpm run dev` - To run the Airnode feed. This step assumes already running signed API as specified in the `airnode-feed.json` configuration.

### Testing

Expand All @@ -53,8 +45,7 @@ LOGGER_ENABLED=true pnpm run test

### Docker instructions

You can use shorthands from package.json. To understand how the docker image is built, read the
[Dockerfile](../../Dockerfile).
You can use shorthands from package.json. To understand how the docker image is built, read the [Dockerfile](../../Dockerfile).

```sh
pnpm run docker:build
Expand All @@ -65,9 +56,7 @@ pnpm run docker:run

<!-- markdown-link-check-disable -->

To deploy Airnode feed on AWS you can use the Cloud Formation template created by the API integrations team. The
template can be found in the private api-integrations repository
[here](https://github.com/api3dao/api-integrations/blob/main/data/cloudformation-template.json).
To deploy Airnode feed on AWS you can use the Cloud Formation template created by the API integrations team. The template can be found in the private api-integrations repository [here](https://github.com/api3dao/api-integrations/blob/main/data/cloudformation-template.json).

<!-- markdown-link-check-enable -->

Expand All @@ -80,14 +69,11 @@ To run the Airnode feed docker image you need to:
<!-- markdown-link-check-disable -->

1. Mount config folder to `/app/config`. The folder should contain the `airnode-feed.json` and `secrets.env` files.
2. Pass the `-it --init` flags to the docker run command. This is needed to ensure the docker is stopped gracefully. See
[this](https://github.com/nodejs/docker-node/blob/main/docs/BestPractices.md#handling-kernel-signals) for details.
2. Pass the `-it --init` flags to the docker run command. This is needed to ensure the docker is stopped gracefully. See [this](https://github.com/nodejs/docker-node/blob/main/docs/BestPractices.md#handling-kernel-signals) for details.
3. Specify the `--env-file` with the path to the `.env` file containing the [ENV configuration](#environment-variables).
4. Optionally, pass the `--rm` flag to remove the container after it is stopped.
5. If running on Linux, you need to use the `--network host` to access the host network. This has no effect for Docker
Desktop.
6. Lastly, if you are using Docker Desktop and you want to access the host machine, you need to change the host URL from
`localhost` to `host.docker.internal` in the configuration files.
5. If running on Linux, you need to use the `--network host` to access the host network. This has no effect for Docker Desktop.
6. Lastly, if you are using Docker Desktop and you want to access the host machine, you need to change the host URL from `localhost` to `host.docker.internal` in the configuration files.

<!-- markdown-link-check-enable -->

Expand Down
59 changes: 17 additions & 42 deletions packages/airnode-feed/config/configuration.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,10 @@
# Configuration

Airnode feed can be configured via a combination of [environment variables](#environment-variables) and
[configuration files](#configuration-files).
Airnode feed can be configured via a combination of [environment variables](#environment-variables) and [configuration files](#configuration-files).

## Environment variables

Logging needs to be initialized prior the configuration files are loaded. This is done via environment variables. All of
the environment variables are optional and or set with default values for convenience.
Logging needs to be initialized prior the configuration files are loaded. This is done via environment variables. All of the environment variables are optional and or set with default values for convenience.

Example:

Expand Down Expand Up @@ -46,16 +44,13 @@ Default: `true`.
The format of the log output. Options:

- `json` - Specifies JSON log format. This is suitable when running in production and streaming logs to other services.
- `pretty` - Logs are formatted in a human-friendly "pretty" way. Ideal, when running the service locally and in
development.
- `pretty` - Logs are formatted in a human-friendly "pretty" way. Ideal, when running the service locally and in development.

Default: `json`.

### `LOG_HEARTBEAT` _(optional)_

Enables or disables the heartbeat log. The heartbeat log is a cryptographically secure log that is emitted every 60
seconds to indicate that the service is running. The log includes useful information such as the deployment timestamp
and configuration hash. Options:
Enables or disables the heartbeat log. The heartbeat log is a cryptographically secure log that is emitted every 60 seconds to indicate that the service is running. The log includes useful information such as the deployment timestamp and configuration hash. Options:

- `true` - Enables the heartbeat log.
- `false` - Disables the heartbeat log.
Expand All @@ -66,8 +61,7 @@ Default: `true`.

Enables or disables colors in the log output. Options:

- `true` - Enables colors in the log output. The output has special color setting characters that are parseable by CLI.
Recommended when running locally and in development.
- `true` - Enables colors in the log output. The output has special color setting characters that are parseable by CLI. Recommended when running locally and in development.
- `false` - Disables colors in the log output. Recommended for production.

Default: `false`.
Expand All @@ -85,16 +79,13 @@ Default: `info`.

## Configuration files

Airnode feed needs two configuration files, `airnode-feed.json` and `secrets.env`. All expressions of a form
`${SECRET_NAME}` are referring to values from secrets and are interpolated inside the `airnode-feed.json` at runtime.
You are advised to put sensitive information inside secrets file.
Airnode feed needs two configuration files, `airnode-feed.json` and `secrets.env`. All expressions of a form `${SECRET_NAME}` are referring to values from secrets and are interpolated inside the `airnode-feed.json` at runtime. You are advised to put sensitive information inside secrets file.

You can also refer to the example configuration files.

### `templates`

Configuration for the template requests. Each template request is defined by a `templateId` and a `template` object. For
example:
Configuration for the template requests. Each template request is defined by a `templateId` and a `template` object. For example:

```jsonc
// Defines a single template.
Expand Down Expand Up @@ -122,18 +113,15 @@ The ID of the endpoint to which the template request is made.

##### `parameters`

The parameters of the template request. Refer to
[Airnode ABI](https://airnode-docs.api3.org/reference/airnode/latest/specifications/airnode-abi.html) specification for
details.
The parameters of the template request. Refer to [Airnode ABI](https://airnode-docs.api3.org/reference/airnode/latest/specifications/airnode-abi.html) specification for details.

##### `parameters[n]`

Defines one of the parameters of the template request.

`type`

Refer to
[Airnode ABI available types](https://airnode-docs.api3.org/reference/airnode/latest/specifications/airnode-abi.html#details).
Refer to [Airnode ABI available types](https://airnode-docs.api3.org/reference/airnode/latest/specifications/airnode-abi.html#details).

`name`

Expand Down Expand Up @@ -177,8 +165,7 @@ The title of the OIS to which the endpoint belongs.

### `triggers.signedApiUpdates`

Configuration for the signed API update triggers. There can be multiple triggers, each specifying a different update
configuration.
Configuration for the signed API update triggers. There can be multiple triggers, each specifying a different update configuration.

For example:

Expand All @@ -202,15 +189,9 @@ For example:

#### `triggers.signedApiUpdates[n]`

Configuration for one of the signed API update triggers. Airnode feed periodically fetches data from data provider and
pushes it to all Signed APIs defined in the configuration file.
Configuration for one of the signed API update triggers. Airnode feed periodically fetches data from data provider and pushes it to all Signed APIs defined in the configuration file.

Airnode feed only makes a single template request independently of the number of template IDs specified. This is to
reduce the number of data provider calls. This implies that all of the templates in the trigger must use the same
endpoint and parameters. You can use
[OIS processing](https://airnode-docs.api3.org/reference/ois/latest/processing.html) to remove the parameters before
making the request (using pre-processing) and later get the corresponding template value based on the endpoint
parameters (using-processing).
Airnode feed only makes a single template request independently of the number of template IDs specified. This is to reduce the number of data provider calls. This implies that all of the templates in the trigger must use the same endpoint and parameters. You can use [OIS processing](https://airnode-docs.api3.org/reference/ois/latest/processing.html) to remove the parameters before making the request (using pre-processing) and later get the corresponding template value based on the endpoint parameters (using-processing).

##### `templateIds`

Expand Down Expand Up @@ -249,8 +230,7 @@ The URL of the signed API.

#### `authToken`

The authentication token used to authenticate with the signed API. It is recommended to interpolate this value from
secrets.
The authentication token used to authenticate with the signed API. It is recommended to interpolate this value from secrets.

If the signed API does not require authentication, set this value to `null`.

Expand All @@ -266,22 +246,19 @@ Refer to the [OIS documentation](https://airnode-docs.api3.org/reference/ois/lat

### `apiCredentials`

Refer to Airnode's
[API credentials](https://airnode-docs.api3.org/reference/airnode/latest/deployment-files/config-json.html#apicredentials).
Refer to Airnode's [API credentials](https://airnode-docs.api3.org/reference/airnode/latest/deployment-files/config-json.html#apicredentials).

### `nodeSettings`

Contains general deployment parameters of the Airnode feed.

#### `nodeVersion`

The version of the Airnode feed. The version specified in the config must match the version of the Airnode feed at
deployment time.
The version of the Airnode feed. The version specified in the config must match the version of the Airnode feed at deployment time.

#### `airnodeWalletMnemonic`

Mnemonic for the airnode wallet used to sign the template responses. It is recommended to interpolate this value from
secrets. For example:
Mnemonic for the airnode wallet used to sign the template responses. It is recommended to interpolate this value from secrets. For example:

```jsonc
// The mnemonic is interpolated from the "WALLET_MNEMONIC" secret.
Expand All @@ -290,6 +267,4 @@ secrets. For example:

#### `stage`

An identifier of the deployment stage. This is used to distinguish between different deployments of Airnode feed, for
example `dev`, `staging` or `production`. The stage value can have 256 characters at maximum and can only include
lowercase alphanumeric characters and hyphens.
An identifier of the deployment stage. This is used to distinguish between different deployments of Airnode feed, for example `dev`, `staging` or `production`. The stage value can have 256 characters at maximum and can only include lowercase alphanumeric characters and hyphens.
11 changes: 3 additions & 8 deletions packages/e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,9 @@

## Getting started

1. Copy the Airnode feed secrets. Whilst inside `src` run
`cp airnode-feed/secrets.example.env airnode-feed/secrets.env`. If you are using Docker Desktop, you need to change
the URL in `airnode-feed/secrets.env` from `localhost` to `host.docker.internal`, because Airnode feed is running
inside a Docker container.
1. Copy the Airnode feed secrets. Whilst inside `src` run `cp airnode-feed/secrets.example.env airnode-feed/secrets.env`. If you are using Docker Desktop, you need to change the URL in `airnode-feed/secrets.env` from `localhost` to `host.docker.internal`, because Airnode feed is running inside a Docker container.
2. Copy the Signed API secrets. Whilst inside `src` run `cp signed-api/secrets.example.env signed-api/secrets.env`
3. Build the latest Docker images. Run `pnpm run docker:build` from the monorepo root. The e2e flow uses the docker
images.
3. Build the latest Docker images. Run `pnpm run docker:build` from the monorepo root. The e2e flow uses the docker images.
4. This module contains services (or configurations) that are integrated together. Specifically:
- `airnode-feed` - Contains the configuration for the Airnode feed service.
- `signed-api` - Contains the configuration for the signed API service.
Expand All @@ -19,8 +15,7 @@

You are free to modify the configurations to test different scenarios.

5. There are `start:<some-service>` scripts to start the services. It is recommended to start each service in a separate
terminal and in this order:
5. There are `start:<some-service>` scripts to start the services. It is recommended to start each service in a separate terminal and in this order:
1. `pnpm run start:data-provider-api`
2. `pnpm run start:signed-api`
3. `pnpm run start:airnode-feed`
Expand Down
Loading
Loading