Actor Pause/Resume flow

[Dmitry Berkovich (dberkov)]

Implements PAUSED state for issue #119.

The snapshot files are kept locally on node VM in a separate folder. At resume time, scheduler uses a node VM hint and picks up a worker from the same node where files were stored at suspend time.

The local file management solution is temporary and will be replaced once Michelle Au (@msau42) introduces a new component that is supposed to manage files on the node VM.

• Tests pass
• Manual tests with counter demo

>kubectl ate create actor my-counter-1 --template ate-demo-counter/counter
>curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 1
>curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 2
>kubectl ate pause actor my-counter-1 -o json
> curl -X POST -H "Host: my-counter-1.actors.resources.substrate.ate.dev" http://localhost:8000
hello from: 169.254.17.2 | preserved memory count: 3

Breaking change

This PR introduces a breaking change in the Actor proto. All existing actor needs to be recreated, prior testing PAUSE functionality.

[Benjamin Elder (BenTheElder)]

Mostly nits inline.

• Can we add coverage for this to the e2e tests? I think this should work on gvisor+kind and it would be nice to cover in CI. We could extend the demo counter e2e.
• Worker.node_name is only populated when the syncer re-mirrors a pod, so workers created before this rollout won't satisfy the node restriction until re-synced.
• I know the storage situation is temporary, but we have no GC for snapshots and these may fill local disk quickly. Maybe a TODO somewhere or a doc warning?

[Dmitry Berkovich (dberkov)]

Can we add coverage for this to the e2e tests? I think this should work on gvisor+kind and it would be nice to cover in CI. We could extend the demo counter e2e.

I have extended existing e2e demo test to test the PAUSE API.

Worker.node_name is only populated when the syncer re-mirrors a pod, so workers created before this rollout won't satisfy the node restriction until re-synced.

Added a disclaimer in the PR description.

I know the storage situation is temporary, but we have no GC for snapshots and these may fill local disk quickly. Maybe a TODO somewhere or a doc warning?

Added TODO in the code.

[Benjamin Elder (BenTheElder)]

Suggested change

	// Lock TTL is 7 seconds, with 2 seconds padding for workflow timeout
	// Lock TTL is 30 seconds, with 2 seconds padding for workflow timeout

[Dmitry Berkovich (dberkov)]

Julian Gutierrez Oschmann (@juli4n) I copied the comment and content of 30 sec from suspend workflow. The comment says "7" sec. Is it a comment incorrect or the actual timeout set to 30 sec instead of 7?

[Julian Gutierrez Oschmann (juli4n)]

It looks like the comment is outdated. It used to be 7 seconds, then it was bumped to 30. See https://github.com/agent-substrate/substrate/blame/808806db7f7b19304a44a491730004239110ef04/cmd/ateapi/internal/controlapi/workflow.go#L145.