Projects

Featured project

RAG Hallucination Detector

Oct 2026

• Fine-tuned ModernBERT-base in a custom PyTorch loop on 100K RAGTruth sentence pairs to flag unsupported claims in LLM-generated RAG answers, reaching 0.65 F1 and 0.92 AUROC on held-out test vs 0.20 F1 and 0.64 AUROC for a zero-shot NLI baseline.
• Trained with fp16 autocast, gradient accumulation, token-budget batching, and class-weighted loss on a Kaggle T4, then reached 71.5 balanced accuracy on 11 LLM-AggreFact datasets and cut calibration error from 0.043 to 0.016 via Platt scaling.
• Served per-sentence calibrated scores through a FastAPI endpoint in a non-root CPU Docker image, gated by GitHub Actions CI running 46 pytest tests and a container smoke test, and published results to a Tableau Public dashboard.

Research details

RAG Hallucination Detector

Question

Can a small fine-tuned encoder flag unsupported sentences in LLM-generated RAG answers better than a zero-shot NLI model?

Method

Fine-tune ModernBERT-base in a custom PyTorch loop on 100K RAGTruth sentence pairs, using fp16 autocast, gradient accumulation, token-budget batching, and class-weighted loss on a Kaggle T4. Calibrate scores with Platt scaling fitted on validation data.

Findings

On the held-out RAGTruth test set the model reaches 0.65 F1 and 0.92 AUROC, compared with 0.20 F1 and 0.64 AUROC for a zero-shot NLI baseline. It reaches 71.5 balanced accuracy across 11 LLM-AggreFact datasets, and Platt scaling cuts calibration error from 0.043 to 0.016.

Serving & reporting

Calibrated per-sentence scores are served through a FastAPI endpoint in a non-root CPU Docker image, gated by GitHub Actions CI with 46 pytest tests and a container smoke test. Results are published on a Tableau Public dashboard.

Calibration