ADR for openedx_catalog models (and in particular, layering above openedx_content) [FC-0138]

[bradenmacdonald]

Based on the discussion in #812, we need to clarify whether the new catalog app depends on the content app, or vice versa.

See the new ADR itself for all the context: 1. Role of Catalog’s CourseRun and CatalogCourse Models

This PR also revises a Pathways ADR, 7. Pathways: Split Between Catalog and Content

Private ref: FAL-4383

[openedx-webhooks]

Thanks for the pull request, @bradenmacdonald!

This repository is currently maintained by @axim-engineering.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

---

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[bradenmacdonald]

@ormsbee @kdmccormick @ChrisChV and especially @Agrendalath, please let me know your thoughts on this.

[kdmccormick]

thanks for writing this up. "Catalog layers over content" has always felt intuitive to me as well.

that said, applying this same direction to Pathways means that:

• PathwayContents will point to CourseRuns, thus PathwayContent must be in openedx_learning
• CatalogPathways will point to PathwayContents, thus CatalogPathways must also be in openedx_learning

it feels weird that CatalogCourses and CourseRuns would be in openedx_catalog, but CatalogPathways will be in openedx_learning. whereas if we flip the relationship so that openedx_catalog is at the bottom, then everything can point "down" to openedx_catalog. but there are other questions to address if we flip the relationship, so I am not sure yet.

a question that's been on my mind that might guide us a bit: where do you think the LearningContext and Usage tables would go?

[Agrendalath]

Nit: I believe PathwayEnrollment should be tied to the CatalogPathway.

[bradenmacdonald]

I've mention this below, but I've been unclear on where enrollments will live.

[Agrendalath]

@bradenmacdonald, I like this idea! I already implemented these changes in #819.

@kdmccormick,

CatalogPathways will point to PathwayContents, thus CatalogPathways must also be in openedx_learning

We could point the content to the PublishableEntity and ensure we bind the Pathway (content) within the openedx_learning API. This way, we won't handle it at the DB level, but it's a similar mechanism to ContentLibrary pointing to the LearningPackage. Please see the initial implementation: https://github.com/openedx/openedx-core/pull/819/changes#diff-80730dea8b20ad4b376488da48af09bb805b17470a49ec0165c32434bd304449R111-R125

[bradenmacdonald]

it feels weird that CatalogCourses and CourseRuns would be in openedx_catalog, but CatalogPathways will be in openedx_learning.

In some ways, it could also make sense, if we think of catalog and content applets as providing basic primitives and then "Pathways" being an application built with those primitives that lives entirely in openedx_learning. I do like the idea of Pathways being a fully self-contained app that can easily be turned off or on, and only using extension points rather than having its code spread out among many different apps. However, we haven't (yet) designed any suitable extension points in catalog.

I think we've always planned for Pathways to be a first-class thing within the catalog, and I'm also completely fine with that approach. To make it work, we can use the "FK to PublishableEntity" approach @Agrendalath is suggesting in #819 or create a join table in openedx_learning that links CatalogPathway to PathwayContents.

a question that's been on my mind that might guide us a bit: where do you think the LearningContext and Usage tables would go?

We haven't yet had any need for a concrete LearningContext table, and I'm not sure if we ever will. If we do, there's also a question of if we'd need a LearningContextType. So far however, we've gotten by by having LearningContext as a vague concept with disparate concrete implementations (courses and libraries). If we did need it though, I would lean toward putting it in catalog.

Usage tables are a mapping of IDs to content, and so they feel like part of the content to me. I'd put them in content, and assume that they're generally modified alongside the content itself, in a similar manner.

Related to that, @Agrendalath's PR #819 reminds me that we have to decide where the enrollment tables should go (for now just pathways, but long-term we may move course enrollment as well). Since catalog manages enrollables it could arguably be catalog, but learning or even somewhere else could also make sense.

[ormsbee]

Thank you for writing this out—I think I was struggling to understand the disconnect I was having, and this really helps to bring it into focus:

openedx_content is generic infrastructure used by courses, libraries, pathways and future context types, and its applets are deliberately ignorant of what a learning package represents.

1. To me, the ContentLibrary mapping exists in openedx-platform because it wasn't practical to do otherwise, especially with the state of permissions at that time—one of the main motivations for the bottom-first build-out of openedx-core was the difficulty in extracting all the dependencies of higher level concepts. I think that it's actually perfectly reasonable in the long term that an app named openedx_content would contain the API for content libraries. This would be in a new applet that sits above the existing ones in terms of layering.
2. I believe that the Catalog exists to serve the narrow "find a course/pathway and enroll in it" experience, along with various queries for basic metadata like the title. I don't think the Catalog is a place that should be responsible for piecing together the broader notion of a course as students experience it.

My specific worry with the framing of this ADR is that I don't think it's the catalog's responsibility to map a CourseRun to the content that backs it. I believe that the question of, "What content do I display for this CourseRun, and how does that map to LearningPackages?" is a content (and eventually learning) concern. Similarly, I would expect credentials/certificate information to sit in its own app and reference catalog, rather than the other way around. The modeling here doesn't need to be exotic, just a 1:1 table from the content side, referencing CourseRun.

Imagine that some CourseRuns will have their own LearningPackages, but we later decide that some other CourseRuns will point to different root nodes within the same LearningPackage. Maybe some wiki-like content is associated at the CatalogCourse level. Even later, we decide that CCX courses will be modeled as LearningPackage + root node + fkey to some other config. We then go off and have a long discussion about whether multiple CourseRuns should be allowed to point to the exact same content tree. Another conversation surrounds whether we run a data migration to consolidate existing runs into shared LearningPackages or not. All of those would be content modeling decisions, and it shouldn't require changes to openedx_catalog.

We've long been in a world where the enrollment context, the authoring context, and the learning context are all the same, and share the same identifier key. As a platform, I think that's limiting. I should be able to enroll in a Pathway and become associated with a variety of learning contexts inside of it. I should be able to enroll in a CourseRun that points into a content tree that lives in a centrally managed library. I believe that the Catalog is the start of that chain of things, but we should let things outside the Catalog figure out how they're going to map CourseRuns to their own data and areas of responsibility.

[bradenmacdonald]

Imagine that some CourseRuns will have their own LearningPackages, but we later decide that some other CourseRuns will point to different root nodes within the same LearningPackage.... All of those would be content modeling decisions, and it shouldn't live or require changes to openedx_catalog.

I've tried to mention exactly those possibilities and keep them open in the latest version of each ADR, but you make a good point that they don't feel like things that should require changes to openedx_catalog. We can discuss this in our call tomorrow.

We've long been in a world where the enrollment context, the authoring context, and the learning context are all the same, and share the same identifier key. As a platform, I think that's limiting. I should be able to enroll in a Pathway and become associated with a variety of learning contexts inside of it.

This may be only tangentially related, but for me it was often an enormous pain that the edx.org marketing site used completely different course IDs than the LMS, and worked pretty hard to hide the course-v1:... key. Several times I needed to correlate them and it required either actually enrolling into the course or digging through the HTML to find the hidden form with the actual course ID buried somewhere in there.

[ChrisChV]

Was this same pattern of openedx_learning, a neutral layer sitting above both catalog and content, with neither depending on the other, considered for the base CourseRun - LearningPackage mapping?