Ask a legal or financial question with no context passage attached. On the left, Google's Gemma 2 2B exactly as pretrained. On the right, the same model after closed-book Q&A fine-tuning on our legal corpus. Both run 4-bit quantized on one GPU. Watch what fine-tuning adds when there is nothing to retrieve.
This is a controlled comparison. The left column is Google's Gemma 2 2B exactly as released. The right column is that same model after we fine-tuned it on roughly 14,000 question and answer pairs drawn from US court opinions and SEC filings, in closed-book form: only the question goes in, no passage is attached, so the model has to answer from what its weights now hold.
We deliberately included a slice of "I do not have enough information" examples so the fine-tune learns to decline when it genuinely cannot answer, rather than inventing a confident but wrong response.
The honest caveat: many of these questions are about very specific records, so the fine-tune is largely recalling memorized facts, not reasoning about new legal questions. That is exactly the thing worth seeing. A retrieval-augmented setup, where the relevant passage is fetched and handed to the model, is the more robust design and lives in a separate demo.
Both models are served 4-bit quantized (bitsandbytes NF4) on a single GPU on Modal. The first request wakes the GPU and can take 30 to 60 seconds; after that it is quick.