feat: Add affinity to OPA Pods

[sbernauer]

Description

Part of stackabletech/opa-operator#525

Definition of Done Checklist

• Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
• Please make sure all these things are done and tick the boxes

Author

• Changes are OpenShift compatible
• CRD changes approved
• CRD documentation for all fields, following the style guide.
• Helm chart can be installed and deployed operator works
• Integration tests passed (for non trivial changes)
• Changes need to be "offline" compatible
• Links to generated (nightly) docs added
• Release note snippet added

Reviewer

• Code contains useful comments
• Code contains useful logging statements
• (Integration-)Test cases added
• Documentation added or updated. Follows the style guide.
• Changelog updated
• Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

• Feature Tracker has been updated
• Proper release label has been added
• Links to generated (nightly) docs added
• Release note snippet added
• Add type/deprecation label & add to the deprecation schedule
• Add type/experimental label & add to the experimental features tracker

[Maleware]

Thanks for your efforts!

I know, I said that's a good idea, but now where I've thought about things, I'm not that sure anymore.

So affinities to OPA make only sense in the eyes of a deployment of OPA with e.g. preferSameNode. I'm still vastly unsure if using this feature is a good idea in the first place, since it might restrict products talking to one opa pod on its node where it was attracted to be deployed due to affinities. Since this specific feature doesn't care about workload and only about a pod being present, it might degrade the performance of the opa cluster in an non-obvious way.

This said, the current implementation of the OPA deployment tries to fan out pods across all nodes if possible and thus, there can be a use case using this feature and then it is a good idea to support attracting affinities. However, it does not harm to have it, since currently we use localTrafficPolicy: Cluster there will be no behavioural difference apart the pod placement.

Means I'd give this one for Trino a go as a pilot, but would probably not implement this in every other operator where it could be of use or not.

EDIT: I expressed my concerns here