Examples For Negative Feedback

How Negative Feedback Actually Works in Training Pipelines

Most teams build out positive signal pipelines first. They spend weeks collecting good outputs, tuning reward models on wins, and celebrating when their evaluator starts producing cleaner results. Then they move on to other problems, leaving negative feedback as an afterthought. This is a mistake. I learned it the hard way about 18 months ago on a project where our model was degrading on edge cases. We had excellent positive test cases but completely missing negative signal coverage. The model looked fine on the happy path and failed silently everywhere else.

What negative feedback is, practically speaking: it's any signal that explicitly marks when an output is wrong, harmful, misaligned, or unacceptable. That's the basic definition. But the useful part is in how you structure those signals. Raw negative labels alone won't help your model. You need them organized into categories, weighted appropriately, and balanced against positive data so the model learns a decision boundary rather than just memorizing what not to do. Here are the types of negative feedback signals you'll encounter in real pipelines, with specifics on how to handle each one. This is the most basic form. The annotator marks an entire output as rejected. It's fast to collect and easy to code, but it's also the least informative type. You know something went wrong, but you don't know why. In practice, I've seen teams rely on these too heavily and then wonder why their model keeps making the same category of mistake. The workaround is adding a secondary tag for the rejection reason even on basic labels. Something as simple as a dropdown for "factual_error," "toxicity," "irrelevance," or "format_violation" turns a useless rejection into a usable training signal. We started doing this at scale and saw our correction rate on factual errors drop by about 40% within two training cycles.

You generate multiple outputs for the same prompt and ask annotators to pick the worst one or rank all of them from best to worst. This is standard in preference-based training approaches. The counter-intuitive part most people miss is that the worst output matters more than the best one for training stability. When you include only the top-ranked response, the model learns a narrow distribution. When you force it to explicitly distinguish between good and bad, the margin between them sharpens. I ran an experiment where we removed the lowest-ranked responses from our training set and the model's failure rate on adversarial prompts increased by nearly three times. It's worth keeping the full ranking rather than truncating to top pairs. Instead of binary accept or reject, you assign a continuous score that goes below a threshold. This is common in moderation and safety-focused systems. The tricky part here is calibration. A score of 0.3 means nothing unless your team has internalized what that number represents. I've seen scoring drift across different annotation sessions because junior annotators treat the scale differently than senior ones. The fix is running calibration rounds where everyone scores the same 20 examples and discussing where they diverge. Do this every two to three weeks during active annotation periods and the drift drops significantly. This is the most detailed form and also the hardest to scale. Instead of a label or score, the annotator writes out exactly what is wrong. Why the answer is incorrect, which step failed, where the logic breaks. This is expensive to collect but produces the highest quality training signal. When you convert these critiques into structured error annotations, you can train error-detection heads alongside your main model. One implementation I worked on used these critiques to build a separate lightweight classifier that flagged potentially problematic outputs before they were even evaluated. That classifier cut our false acceptance rate by about 28% in production, though it added roughly 40 milliseconds of latency per request.

These are deliberately crafted inputs designed to trigger failure modes. Unlike organic negative feedback that comes from real users, adversarial examples are constructed. They're useful for stress testing and robustness training. The risk is that adversarial examples can encode biases in how they're generated. If your adversarial collection process always targets the same type of edge case, your model becomes over-indexed against that specific failure while ignoring others. I recommend mixing adversarial negatives with organic ones at roughly a 1-to-5 ratio. Too much adversarial data and the model starts optimizing for artificial patterns rather than real-world correctness. Here's how to actually put these signals into a training pipeline without breaking everything. Start by defining what negative means for your specific use case. "Bad output" means something completely different for a medical QA system versus a creative writing assistant. Write down five concrete examples of each negative category before you collect a single data point. This forces your team to agree on boundaries early and saves you from weeks of rework later.

Next, set up your collection mechanism. This could be a simple annotation tool, an API endpoint that logs rejected outputs, or a manual review queue. The tool doesn't need to be fancy. It needs to preserve context: the original prompt, the generated output, the rejection reason, and any metadata about when and how the rejection happened. Missing metadata makes it nearly impossible to analyze patterns later. Then handle the class imbalance problem. Negative examples are usually fewer than positive ones. Don't just oversample negatives because that creates artificial distributions. Instead, calculate what fraction of your total training set should be negative based on your target deployment scenario. If your system sees bad inputs roughly 10% of the time in production, don't train it on 50% negative data. Match the distribution. I've seen teams flip this the other direction too, using almost exclusively positive data and treating negatives as a validation-only concern. That produces models that look great in testing and fail in the wild. When you integrate the signals into your loss function, be careful about how you weight them. A common approach is to apply a higher penalty coefficient to negative examples than to positive ones. The exact multiplier depends on your data ratio and your tolerance for false positives versus false negatives. For safety-critical systems, I typically start with a negative weight of 2.5 to 3 times the positive weight. For less critical applications, 1.5 times is usually sufficient. Run ablation studies on the weight parameter. Small changes here produce disproportionately large effects on model behavior.

Pitfalls and Limitations

There are several ways negative feedback pipelines break, and they usually break in subtle ways that are hard to detect. Annotation inconsistency is the most common issue. Different annotators apply different standards. One person might reject an output for being slightly off-topic while another lets the same output pass. This noise compounds over time. The solution is regular calibration sessions, clear guidelines with edge-case examples, and periodic inter-annotator agreement measurements. Aim for at least 0.7 kappa score between annotators before trusting the aggregated labels. Another problem is negative feedback loops. When your model generates negative examples and you then train on those, you can accidentally reinforce its own biases. This is especially dangerous with adversarial generation pipelines. Always validate adversarial examples against an independent reference model or human review before including them in training data.

Sparse negative signals in low-resource domains are a real bottleneck. If your application area has very few documented failure cases, you simply cannot build a robust negative feedback system from scratch. In those cases, transfer learning from a related domain is your best option. Find a domain with richer negative feedback, train your model there first, and then fine-tune on your target domain with whatever negative examples you can collect. This usually gets you 60 to 70% of the performance you'd achieve with native negative data at a fraction of the cost. Finally, don't confuse absence of positive feedback with negative feedback. Just because an output isn't marked as good doesn't mean it's bad. It might be neutral, incomplete, or simply outside the scope of what you asked annotators to evaluate. Make sure your annotation instructions clearly separate "not good enough" from "explicitly wrong." Mixing these two categories corrupts your training data in ways that are very difficult to detect after the fact.

Practical Numbers

For a mid-complexity classification task, expect to spend about 8 to 12 hours per annotator per week on negative example labeling if you want meaningful coverage. This assumes your guidelines are finalized and your tools are functional. If you're still writing guidelines or debugging your annotation interface, double that timeline. A typical dataset for a production model includes roughly 5,000 to 15,000 negative examples across all categories. Start smaller and expand as you validate your process. A well-structured set of 3,000 negative examples with clear categorization will outperform a sloppy set of 20,000 uncategorized rejections every time. The tradeoff is real. Negative feedback improves robustness but it doesn't make your model smarter. It makes it less likely to fail spectacularly. If your goal is to improve capability, you still need strong positive signal. Negative feedback is the safety net, not the engine. Build both.