Earlier this year, I added comments to this blog, knowing full well what would come with them: spam. Sure enough, it didn't take long for an email to arrive with a "Fantastic post, really enjoyed reading this!" from someone who clearly hadn't read the post.

Since then, ads for THC creams, links to unrelated sites and generic praise have become a regular part of my moderation queue. Of the 102 comments left on the site so far, 79 are spam. Each one sends me a "New comment on …" email and sits in the Pending queue next to the real comments, waiting for me to click a button. It isn't a lot of work yet, but it's the kind of work that only grows, and I always knew I'd have to deal with it eventually.

I wasn't completely unprepared: the comment form has a honeypot field and a rate limit, which stop the laziest bots. But they can't do anything about spam that's typed into the form like a real comment, and to catch that, something has to actually read it.

The plan I didn't need

My plan was the classic machine learning one, and I even had a recipe for it. A while ago, I wrote about building a web-friendly AI model from scratch, which classified the sentiment of YouTube comments by generating embeddings for each comment and training a small PyTorch model on top of them. Spam detection is the same kind of problem: take a comment and put it into a category. And every time I moderated a comment, I was creating a labeled example: Approved, Rejected, or Spam. Once I had enough of them, I could train a classifier on those labels, deploy it, and let it flag new comments automatically.

The catch is in "once I had enough of them". The dataset for that sentiment model had 17,872 labeled comments. This blog has 102, so collecting enough examples to train something reliable would take years. Even then, I'd own a training pipeline, a model to host, and a retraining step every time spammers changed tactics.

But what if I didn't need a dataset at all? That's what I found myself asking when I started experimenting with Jev, TypeSafe's first "System One" model. Jev doesn't generate text. You give it some application state and a question, and it returns a typed answer, such as a probability, that your code can act on. TypeSafe describes System One models as "built for decisions inside software". For spam detection, that removed the training step entirely: instead of teaching a model what spam looks like from examples, I describe what spam is, and Jev applies that description to each comment.

If you've been collecting labeled data for a model you never got around to training, it isn't wasted. A hundred labeled examples are far too few to train a classifier, but they're enough to evaluate one and to choose its threshold. That's exactly how the comments I had moderated ended up being used here.

Asking the question

The whole integration is a single question per comment. Jev supports a few question types; I used a Noul, which asks a yes/no question and returns the probability that the answer is yes. A probability is more useful here than a plain boolean, because it lets my code decide how sure it needs to be before acting.

The state Jev sees is a small JSON document with the comment and the article it was posted on:

{
  "article": { "title": "…", "description": "…" },
  "comment": { "author_name": "…", "text": "…", "is_reply": false }
}

And the question:

• Instructions: Is `comment` spam rather than a real reader engaging with `article`?
• Yes when: unsolicited advertising or self-promotion, links to unrelated sites, generic praise that could be posted on any page, bot-generated or templated text, or content unrelated to any blog topic.
• No when: written by a real person responding to the article, even if it is rude, critical, off-topic, or short.

A few decisions are worth explaining:

• The article gives the question its context. A lot of spam is only recognizable as spam relative to the page it was posted on. "Great insights on cloud hosting!" is a reasonable comment on a post about cloud hosting and an obvious bot on a post about WebMCP. Including the article's title and description lets Jev judge whether the comment engages with this post, which is what the instructions ask.
• The "no" criteria matter as much as the "yes" ones. Spam and unwanted comments are different things. A rude or critical comment from a real reader isn't spam, and deciding what to do with it is my job, not the filter's. Spelling that out keeps Jev from treating "negative" as "spam".
• The commenter's email is never sent. The comment form asks for an optional email, but it adds nothing to the judgment, so it doesn't leave my systems.
• One question, no explanations. Jev returns just the probability. I don't need a rationale for each comment; when I disagree with a decision I can see the comment itself.

Does it work?

To find out, I ran every existing comment through Jev and compared the score with how I had moderated it. Spam comments are the positives. Approved and Rejected comments are the negatives: they were written by real people, even when I chose not to publish them.

Once there's a threshold, every comment lands in one of four buckets: spam that was flagged (a true positive), spam that slipped through (a false negative), a real comment that was wrongly flagged (a false positive), and a real comment that was left alone (a true negative). The metrics in this article are different ways of counting those buckets:

• Recall, also called the true positive rate: of all the spam, how much was flagged?
• Precision: of everything that was flagged, how much was actually spam?
• False positive rate (FPR): of all the real comments, how many were wrongly flagged?

Raising the threshold makes the classifier more cautious: precision goes up and the false positive rate goes down, but more spam slips through, so recall drops. Picking a threshold means picking a spot on that trade-off.

Figure 1: Spam probability for each labeled comment. Most spam scores above 0.95, while real comments spread across the lower range. The highest-scoring approved comment scored 0.72.

The separation is clear. 59 of the 79 spam comments scored 0.95 or higher. Real comments are spread across the bottom of the range, and no approved comment scored above 0.72.

The overlap is in the middle, and looking at those comments is more instructive than the numbers. The approved comments with the highest scores are "MCP rocks!!" (0.72), "Great stuff!" (0.58) and "Totally Agree!" (0.50). They're short, generic praise, exactly what the criteria describe as spam, except that real readers wrote them. On the other side, comments I marked as spam that scored low include a few test comments I posted myself ("hi", "great post!") and an abusive message that, honestly, is a real person being rude rather than spam. Some of the "misses" are really labeling problems, so the true recall is a bit higher than the numbers show.

Figure 2: The ROC curve plots the share of spam caught against the share of real comments wrongly flagged, for every possible threshold. The area under the curve (AUC) is 0.922.

The ROC curve sums this up. Its AUC of 0.922 means that a randomly picked spam comment scores higher than a randomly picked real comment about 92% of the time. That isn't perfect, but it's good for a classifier that never saw a single one of my comments before.

Picking the threshold

A probability isn't a decision, though. If Jev says a comment is 80% likely to be spam, should it be hidden or not? I still needed a threshold: comments scoring at or above it are marked Spam automatically.

The two kinds of mistakes don't cost the same. If a spam comment slips under the threshold, nothing changes from before: it stays Pending and I get the usual email, now with its spam score, and moderate it by hand. If a real comment is flagged, it's moved straight to the Spam tab and no email is sent. A reader's comment could sit there unnoticed. So the threshold should favor precision: missing some spam is fine, hiding a real reader isn't.

Figure 3: Precision, recall and false positive rate as the threshold moves from 0.5 to 0.99.

At 0.85, the 102 comments fall into the buckets like this:

I picked 0.85 for these reasons:

• It flags no approved comments. The highest-scoring approved comment is at 0.72, so 0.85 leaves a margin of more than 0.1 for comments that are a little more generic than any I've seen.
• The comments it does flag deserve it. The two "false positives" at 0.85 are Rejected comments: an ad for THC cream (0.97) that I rejected instead of marking as spam, and a comment that just said "test" (0.90).
• It gives up little recall. Moving from 0.8 to 0.85 costs two spam comments. Going to 0.9 would cost two more and flag exactly the same real comments.
• Scores are stable enough. Running the evaluation twice moved scores by about 0.01, which changed some rows by one comment, but never the 0.85 row.

At 0.85, about four out of five spam comments never reach my inbox, and the rest arrive with a high spam score that makes them quick to dismiss.

A threshold only makes sense for the model it was calibrated on. Probabilities from different providers, or even from different versions of the same model, aren't comparable, so switching models means running the evaluation again and picking a new threshold.

What I didn't measure

I didn't build the custom model, so I can't say how Jev compares to one. A model trained on my own comments might well be more accurate: it would learn the exact flavor of spam this blog attracts, and a 0.922 AUC leaves room to improve.

The evaluation is also small. There are only 23 negative examples, so a single comment moves the false positive rate by about 0.04, and the labels have the problems described above. Treat the numbers as a sanity check, not a benchmark.

For this use case, though, "good enough" is the right bar. The goal is to stop clicking through obvious spam, not to moderate automatically. Every flagged comment can still be approved from the Spam tab, and every other comment still reaches me.

Conclusion

The interesting part of this project isn't the spam filter, it's what it didn't take to build one. There was no dataset to collect, no model to train, and nothing to host. I wrote down what spam means for this blog, in plain language, and got back a probability my code could compare against a threshold. The labeled data I had been collecting, which would have been a training set, turned into a small evaluation set for choosing that threshold.

That pattern, describing a decision instead of learning it from thousands of examples, makes classification practical for problems that are too small to justify a custom model. Most problems on a personal blog are that size.