Tag status review

[j616]

Details

This PR is the result of a review of the status of tags in the AppNote0003 registry. The following changes have been made:

• Elevate the flow_status tag to the core spec, as the status Flow parameter
  • This tag is widely used in implementations and seen as stable
• Elevate the auth_classes and language_code tags from Experimental to Proposed
  • These tags have existed for some time without changes and have had their status adjusted to reflect this
• Deprecate the originating_id and originating_timerange tags
  • These tags have largely been replaced by the first_referenced_by_flow parameter in Object metadata
• Marked hls_exclude and hls_segments tags as Implementation specific
  • These parameters have been used in demos and have seen limited use in implementations, but are not widely used. We also strongly recommend the use of TAMS directly and not via HLS adaptation layers as these do not support TAMS timing or partial Object use. Results may be unexpected where HLS is used.

Issue (if relevant)

GitHub Issue: #206
Jira ticker: https://jira.dev.bbc.co.uk/browse/CLOUDFIT-5487

Related PRs

Submitter PR Checks

• PR completes task/fixes bug
• API version has been incremented if necessary
• ADR status has been updated, and ADR implementation has been recorded
• Documentation updated (README, etc.)
• PR added to Jira Issue (if relevant)
• Follow-up stories added to Jira

Reviewer PR Checks

• PR completes task/fixes bug
• Design makes sense, and fits with our current code base
• Code is easy to follow
• PR size is sensible
• Commit history is sensible and tidy

Info on PRs

The checks above are guidelines. They don't all have to be ticked, but they should all have been considered.

[johnbilt]

Elevating the flow status tag makes a lot of sense and is being used now in a number of implementations where people need to understand whether a flow is ingesting. There have been some discussions around whether there is more functionality to add around flow status rather than just elevating the existing tag to the core specification. These include:

1. The role of the store in managing the flow_status field. Currently this is down to the ingest tool to manage the state of the field, and while this is the best practise we do see ingests which have finished but the tag has not been cleared. One option is for the store take responsibility for automatically closing flows which have had no new segments created after a period of time (eg 15 minutes) where the status has not been set to 'awaiting_content'. Similarly, should the store take any responsibility for setting the status in the first place, eg 'awaiting_content' could be set automatically on the flow creation and then 'ingesting' set when the first segments are received. Combined with the auto closing could mean this works regardless of the capabilities of the ingest solution but does put additional work on the store to manage

2. There has been a request to roll up the flow status into the source level and specifically to the top level source. In a world of source first discovery then currently the only way to know if a piece of content is ingesting is to walk the whole tree and evaluate each flow to find out if there are any still ingesting. Having the status ripple up to the top of the data structure would be useful, but again puts more things for the store to manage, plus needs rules defining about how to handle different statuses on different flows.

[j616]

1. The role of the store in managing the flow_status field. Currently this is down to the ingest tool to manage the state of the field, and while this is the best practise we do see ingests which have finished but the tag has not been cleared. One option is for the store take responsibility for automatically closing flows which have had no new segments created after a period of time (eg 15 minutes) where the status has not been set to 'awaiting_content'. Similarly, should the store take any responsibility for setting the status in the first place, eg 'awaiting_content' could be set automatically on the flow creation and then 'ingesting' set when the first segments are received. Combined with the auto closing could mean this works regardless of the capabilities of the ingest solution but does put additional work on the store to manage

I feel like this might cross the line we've traditionally held that TAMS is about advertising available content, as far as possible. We've traditionally steered away from active behaviour like this. And I'm not sure this is in the same ballpark as garbage collection of Objects, for example. But I wonder if it is similar in scope to AppNote0019 retention management. I wonder if an App Note on how this parameter may be sanitized via (potentially external) automation processes is the better approach, which fits more with existing patterns.

2. There has been a request to roll up the flow status into the source level and specifically to the top level source. In a world of source first discovery then currently the only way to know if a piece of content is ingesting is to walk the whole tree and evaluate each flow to find out if there are any still ingesting. Having the status ripple up to the top of the data structure would be useful, but again puts more things for the store to manage, plus needs rules defining about how to handle different statuses on different flows.

I'd be cautious about this behaviour. It feels like while this adds some convenience, it leads to more opportunity for inconsistencies. Its quite possible for only some, not all, Flows of a given Source to be currently ingesting. So marking the top level Flow/Source as ingesting may not be appropriate. I wonder if this would better solved by adding a query param to filter by status, and then using that in combination by the upcoming collected_by filter?

[j616]

I've added a query parameter to filter Flows on status in 421a4c3