feat: Add AI-powered spam detection, flagging, and automated moderation for forum discussions - #288
feat: Add AI-powered spam detection, flagging, and automated moderation for forum discussions#288Alam-2U wants to merge 1 commit into
Conversation
0fb6b6e to
cea3944
Compare
Alam-2U
changed the title
| delete_thread(content_id, course_id=course_id) | ||
| log.info(f"AI Moderation Deleted CommentThread: {content_id}") | ||
| elif content_type == "Comment": | ||
| delete_comment(content_id, course_id=course_id) | ||
| log.info(f"AI Moderation Deleted Comment: {content_id}") |
There was a problem hiding this comment.
are these soft deletes? What if the AI is wrong and we need to show the posts again?
There was a problem hiding this comment.
Soft delete has been implemented in the 2U downstream Forum, but it has not been merged upstream yet. I’ll create a PR for the soft-delete functionality once the AI moderation feature is merged.
Once implemented upstream, staff will have the option to restore posts that were incorrectly deleted by AI moderation. This functionality will be optional and will not affect any existing functionality. It can be enabled based on requirements.
| """ | ||
| Service for AI-based content moderation. | ||
|
|
||
| Waffle Flag "discussion.enable_ai_moderation" controls whether AI moderation is active. |
There was a problem hiding this comment.
correct waffle flag. discussions.enable_ai_moderation
There was a problem hiding this comment.
I will update the PR accordingly
| models.Index(fields=["course_id", "is_spam"]), | ||
| models.Index(fields=["author", "course_id", "is_spam"]), |
There was a problem hiding this comment.
are these indexes needed for a staff only feature?
There was a problem hiding this comment.
These indexes were added because we’ve seen a significant amount of spam activity concentrated in certain courses, often from the same accounts. The indexes help optimize queries that filter spam by course_id and is_spam, and by author, course_id, and is_spam, which are relevant for identifying and managing these spam posts efficiently.
3 participants
Description
Adds provider-agnostic AI moderation to forum-v2.
New threads and comments are classified by a configured AI backend. Content classified as spam is flagged, optionally deleted, and recorded in an audit log.
Forum provides the moderation interface and workflow, but does not include a specific AI provider.
AI_MODERATION_BACKENDhas no default; deployments configure the backend they want to use, and each backend handles its own provider-specific credentials and model settings.What's Included
Moderation Pipeline
Located under
forum/ai_moderation/:backends/base.py— AddsBaseModerationBackendas the backend interface andHTTPModerationBackendas a reusable base for JSON-over-HTTP providers.classify()returns{classification, reasoning, confidence_score, full_api_response}orNone. Classification failures never raise exceptions, allowing posting to continue if moderation is unavailable.service.py— Runs the moderation workflow:classify → flag → optionally delete → audit-log. Spam verdicts are cached using a content hash to avoid repeated API calls for identical content. Clean verdicts are not cached.defaults.py— Provides provider-independent defaults, including a ready-to-use spam classification prompt.Hook Points
The following operations trigger moderation after successful content creation:
create_thread,create_parent_comment,create_child_comment.Moderation runs inside a broad
try/except. Any moderation failure is logged and does not cause the original post/comment creation to fail.Data Model
Migration
0006adds:is_spamboolean field toCommentThreadandComment, with indexes onis_spam,(course_id, is_spam), and(author, course_id, is_spam).ModerationAuditLogcontaining content body, full classifier response, reasoning, classification, actions taken, confidence score, original author,moderator_override, andoverride_reason.flag_content_as_spamandunflag_content_as_spambackend methods.Flagged content is attributed to the configured AI moderation user through
abuse_flaggers, allowing it to surface through the existing report path.Admin
Adds
is_spamto thread and comment list displays and filters. Adds a read-onlyModerationAuditLogadmin with search across author, moderator, reasoning, and body.Documentation
Adds
docs/how-tos/configure_ai_moderation.rst, covering backend implementation, required and optional settings, and enabling the course-level waffle flags.Configuration
Required
AI_MODERATION_BACKENDAI_MODERATION_API_URLAI_MODERATION_USER_IDOptional
AI_MODERATION_SYSTEM_MESSAGE,AI_MODERATION_CONNECTION_TIMEOUT(1.0s),AI_MODERATION_READ_TIMEOUT(30s),AI_MODERATION_FLAGGED_CACHE_TTL(24h),AI_MODERATION_FLAGGED_CACHE_PREFIX.Course Waffle Flags
Both flags are off by default:
discussions.enable_ai_moderation,discussions.enable_ai_auto_delete_spam.Testing
Added and updated tests covering service behaviour, caching, flagging, deletion, audit logging, failure handling, backend interface, and HTTP backend.
tests/test_ai_moderation.pytests/test_ai_moderation_backends.pytest_utils/moderation.pytests/test_backends/test_mysql/test_api.pyandtests/test_backends/test_mysql/test_models.pyRollout
The feature has zero impact when unconfigured. With no
AI_MODERATION_BACKENDconfigured and both waffle flags disabled, no moderation calls are made and existing forum behaviour remains unchanged.AI moderation can be enabled on a per-course basis.
Related PRs