I Screened 471 Comments for Viewer Confusion. The Keyword Test Failed

The 471-comment feasibility result

Question form and question intent are different variables

A sentence can end with a question mark while functioning as a statement, challenge, joke, or emotional reaction. 'Why would he do that?' may show that the viewer understood the story perfectly and is judging the character. 'What song is this?' requests metadata, not the premise. 'Who is that?' might be referent confusion, fandom curiosity, or a request for an actor's name.

Natural-language research treats this as an intent-classification problem. Work on non-information-seeking question types separates rhetorical and other pragmatic uses, while a study of question intentions shows that surface form alone does not reveal what a speaker is doing.

Social comments add abbreviations, code-switching, emoji, quoted dialogue, missing punctuation, and shared context. A dictionary is useful for candidate retrieval. It is not a gold-standard labeler, and its candidate rate should never be renamed as viewer confusion prevalence.

The six comment intents the study must separate

Precision was measurable; recall was not

Precision asks: of the comments the lexicon flagged, how many were true premise-confusion cases? Here the answer was one of nine. Recall asks: of all true confusion comments, how many did the lexicon find? The pilot did not manually code the 462 non-candidates, so the denominator required for recall does not exist.

That missing audit is not a footnote. Viewers can express confusion without a question word: 'I have no idea what's happening,' 'context?', a language-specific idiom, or a reply that only becomes clear with the parent comment. A low candidate count can coexist with many missed cases.

The correct publication decision is therefore to stop. Reporting one confirmed comment out of 471 as a 0.2% confusion rate would assume perfect sensitivity, identical comment visibility, and a valid sampling window. None was established.

Newest comments are not early viewer reactions

The connector returned newest available top-level comments, not the first comments after publication. Older videos may have accumulated an audience that understands recurring characters and inside jokes. Moderation, deleted comments, ranking, and reply structure can also change what remains visible.

A study about early confusion needs a fixed window such as the first 24 or 48 hours, a collection timestamp, and a rule for delayed comment loading. It should store the position and thread structure needed for analysis without publishing usernames or raw text unnecessarily.

The denominator also needs two versions: all comments and substantive-text comments. Emoji-only responses cannot contain lexical evidence, but excluding them silently can exaggerate the apparent density. Both counts should be visible.

Caption timing cannot be linked with ten videos

The original idea was tempting: compare caption onset with confusion-comment density and estimate whether late text produces more confusion. Ten videos, one confirmed comment, and an unvalidated classifier cannot support that model. The unit of analysis would be the video, not the individual comment, leaving almost no independent observations.

Even with a larger sample, language, niche, comment volume, audience size, upload age, creator familiarity, text presence, and discovery source could affect both comments and editing style. A raw ratio would mistake those conditions for an effect of caption timing.

The publishable design is a mixed-effects model at the video level after a human-coded gold standard establishes language-specific error rates. The result would remain correlational. A caption-delay multiplier needs stronger evidence than a regression coefficient beside a weak classifier.

How to read your own comments without fooling yourself

• Read the full sentence and nearby thread before labeling a question as confusion.
• Separate premise confusion from requests for names, songs, products, recipes, or sources.
• Distinguish rhetorical reaction from genuine information seeking.
• Record the post-publication window; ten comments from day one are not comparable with ten comments from month six.
• Look for confusion without question words, including 'context?' and declarative statements of uncertainty.
• Treat comments as one evidence source alongside retention and direct viewer tasks, not a ground-truth diagnosis of the edit.
• Preserve privacy by aggregating or paraphrasing unless a user has given permission to quote them.

The gold-standard comment study I would run next

Build a stratified set of at least 1,000 comments across languages, niches, audience sizes, and fixed post-publication windows. Two fluent coders should independently label all six intent categories, with a separate training set and preserved adjudication. Replies should be sampled deliberately rather than mixed accidentally with top-level comments.

Measure precision, recall, and F1 by language and category. Krippendorff's alpha is appropriate for multiple coders and can handle missing ratings; a recent open methods paper explains the coefficient, bootstrap uncertainty, coder training, and the common 0.80 reliability target.

Only after the classifier meets preregistered thresholds should it scale to a larger video sample. The public report must show the confusion matrix, hard examples, subgroup errors, excluded comments, and both comment denominators. An aggregate score without its errors would hide the exact problem this pilot found.

Claims this feasibility pilot cannot support

• Only 0.2% of viewers were confused.
• Question words accurately detect viewer confusion.
• Late captions caused more confusion comments.
• The classifier works equally across English, Hindi, Spanish, and Indonesian comments.
• Newest comments represent the first audience response.
• A comment classifier can replace direct comprehension testing.

Privacy and reproducibility boundary

The internal pilot stored aggregate counts and coding outcomes without retaining comment authors or republishing raw comments. Public accessibility is not permission to build a permanent identity-linked dataset. Any future corpus needs a retention plan, platform-policy review, and a decision about whether paraphrase is sufficient.

The codebook, collection date, candidate lexicon, language coverage, and reviewer decisions should be versioned. Model-generated labels remain proposals until a human audit establishes error rates. If the lexicon changes, old and new results should not be combined without recoding.

No relevant ViralJury customer win exists for this topic. The failed screen is itself the useful proof: the research process rejected a convenient metric before it became a product claim.