Upcoming Boost upgrade in SuperCollider

[dyfer]

Hello FluCoMa devs,

We are planning to upgrade boost in SuperCollider: supercollider/supercollider#5924. That solves at least one issue in SC.

As I understand, this will make the release build of FluCoMa incompatible.

In the long run, it would be best to figure out how to make FluCoMa independent from the version of boost in SC. We could consider extending the plugin interface if that would help solving that, however I have no in-depth knowledge how to address this.

In the short run, what'd be the best way forward? I was thinking of making the boost upgrade in SC 3.13.1, but if this causes effectively a plugin version mismatch for FluCoMa, then we'd maybe wait with it until SC 3.14.0.

Do you have any recommendations how to proceed?

[tremblap]

Hello!

Thanks for this warning, this is very kind of you and very considerate!

@weefuzzy is the grand master in all things deep-architecture, so please take my suggestion with a grain of salt.

I would obviously say that extending the plugin interface to enable us (and the VST extension iirc) to more sturdily linking is the best way forward. How hard is it to do it sooner than later? I'm asking because it would make us future-proof whilst we still have hired people on the project to test it for real!

Waiting for v.3.14 is obviously my choice as we could test it all, especially if there is an interface change which could be on a speculative branch of SC and FluCoMa (and others) so we have time to see if it is sturdy for all.

Thanks again for your feedback and heads-up

p

[weefuzzy]

Thanks for the heads up @dyfer.

It may well cause ABI issues between the current release build and a build of scsynth using the newer Boost. If it causes widespread grief, guess we can mitigate the transition by temporarily maintaining a separate build (although, I'd obviously rather not). Bleeding-edge, confident users simply need to rebuild against whatever version of SC they wish (on Arch, this is already the operating practice, as the Arch SC package opts to use the system Boost by default).

At least the failure conditions are easy to diagnose: all the FluidBuf* objects will cause immediate scsynth crashes.

In the long run, it would be best to figure out how to make FluCoMa independent from the version of boost in SC. We could consider extending the plugin interface if that would help solving that, however I have no in-depth knowledge how to address this.

Great to hear that you're up for this. The actual extensions are pretty simple: just the ability to copy / delete ReplyAddr pointers (without needing to know anything about their insides), and adding a SendReply that can be used from the NRT command thread (which exists internally, it's just not in the plug interface table). IIRC, the main impediment to doing it isn't these specific changes but more general discussions within the SC community about how extensions to the interface table should be handled long term.

If it did happen, this would also potentially unblock supernova support, which might be nice.

In the short run, what'd be the best way forward? I was thinking of making the boost upgrade in SC 3.13.1, but if this causes effectively a plugin version mismatch for FluCoMa, then we'd maybe wait with it until SC 3.14.0.
Do you have any recommendations how to proceed?

I'd say just do it: our CI can be directed to build against whatever version of SC, and – seeing as its our code crime – the onus is on us to try and manage the short-term fallout.

Thanks again for being so considerate.

[dyfer]

Thank you @tremblap and @weefuzzy

Thanks for this warning, this is very kind of you and very considerate!

Thank you! I remembered that there was something going on about FluCoMa and boost when looking at SC boost upgrade PR so I reached out to check.

TL;DR considering your responses I don't want to upgrade boost in the 3.13.x release. That would essentially break ABI compatibility in a bugfix version, even if just for FluCoMa.

It may well cause ABI issues between the current release build and a build of scsynth using the newer Boost.

AFAICT it does. I tested the build from the linked PR on macOS against the develop build of FluCoMa and scsynth indeed crashed when running FluCoMa code.

IIRC, the main impediment to doing it isn't these specific changes but more general discussions within the SC community about how extensions to the interface table should be handled long term.

I don't have enough experience to go in depth into the actual implementation of plugin interface. My personal preference would be to move towards the extensible interface. This was discussed in supercollider/supercollider#5347 but it hasn't moved for quite some time.

I understand that maintaining two versions of FluCoMa would be a necessity if we did upgrade boost version we use, but I really don't like that, especially that the incompatibility wouldn't even be expressed in the "official" part of the plugin interface

If it did happen, this would also potentially unblock supernova support, which might be nice.

Again, I would love that.

With all this in mind, my personal preference would be to consider the plugin interface change for SC 3.14 (either extending it with a fixed functionality, or by making it extensible), and to upgrade boost in SC 3.14 (i.e. pull it into the develop branch, maybe even after the plugin interface change). This is my personal preference and not necessarily representative of the larger SC dev community. In particular, the plugin interface change would need to be agreed on and reviewed by devs who are more experienced in that area.

I'll inquire in supercollider/supercollider#5347 to see if SC would like to move to an extensible interface or not. If not, I'd propose to upgrade/break the plugin interface to accommodate ReplyAddr etc pointers and supernova support. In either case, let's make sure that the upcoming changes accommodates FloCoMa's needs.

[dyfer]

Hello again, we finally updated boost to version 1.86 in SC: supercollider/supercollider#6477

[capital-G]

IIUC the hack occurs because you rely on SendReply which is not exposed via the PluginAPI.

Implementing SendReply "from scratch" is not too difficult, see https://github.com/capital-G/OSCMessenger/ - maybe you just implement it from scratch here as well?

[weefuzzy]

Thanks for the thought! The root problem is actually that we need to be able to send replies from the command fifo thread (rather than the audio thread), and to be able to make copies of ReplyAddr pointers.

supercollider/supercollider#5939

[capital-G]

Just FYI - we are planing to release the first RC for SuperCollider 3.14 next week.

As mentioned, SC had to update boost, so 3.14 will be incompatible w/ the current flucoma version you are distributing.
Maybe you can also distribute a version which is compatible w/ 3.14? :)

BTW - the current plan is to transition to a C-API based plugin interface starting with 3.15 - this breaks all plugins once more, but we can introduce empty slots to the plugin struct this way, which hopefully makes the plugin interface possible to upgrade while maintaining backwards compatibility with plugins - so we can also add your proposed changes easily in 3.15.
But there is no release date for 3.15 yet.

[tremblap]

Thanks for the warning, @weefuzzy will be delighted of the eventual API upgrade.

As for now, I'll wait for the 3.14 to become canonical (aka tip of main in the SC repo) and will trigger a recompile. We always distribute only a version compatible with what is the canonical SC.

[dyfer]

As for now, I'll wait for the 3.14 to become canonical (aka tip of main in the SC repo) and will trigger a recompile. We always distribute only a version compatible with what is the canonical SC.

Please keep in mind that this only happens when the actual version (i.e. 3.14.0) is released; a release candidate is not being merged into main. Depending on how many release candidates there are (i.e. how many issues are discovered) the process may take a few weeks.

If you only build based on main, your next version won't be compatible with SC 3.13 on macOS and Windows. If users complain that they need that, you may consider making two Flucoma releases and use SC branches 3.13 and 3.14 respectively for building them... These are stable, protected branches. Or even better, use tags. I'm just throwing it out there in case the issue arises, I'm not necessarily suggesting that this is the best way forward.

Hopefully we won't have such issues anymore once SC 3.15 arrives, whenever that happens.

[capital-G]

@tremblap we released SC 3.14.0 yesterday and it is tip of main now :)