Practice Testing: Why Real Questions Beat Generated Ones

Three weeks out from the exam, you do the sensible thing: you stop re-reading and start testing yourself. Everyone says that's the better use of the time, and they're right.

Then comes the question nobody warns you about. Where do the questions come from? Ask ChatGPT, Gemini or Claude for twenty practice questions on any topic and you'll get twenty in about eight seconds. They'll look right — four answer choices, plausible distractors, reasonable difficulty, the right vocabulary throughout. Free, at midnight, for any subject.

Whether that's worth anything depends on what you're preparing for. Take AP as the worked example, because it makes the principle unmistakable.

Then you sit the actual exam in May and something is off. The questions are longer than you practiced. They embed the statistics inside a context paragraph you have to decode first. The free-response section asks you to justify a choice in a specific way, and you have no idea how many points that justification is worth or what the reader is looking for.

Nothing went wrong with the generated questions. They were competent questions about statistics. The AP exam is not a test of statistics in general — it is a specific, standardized, rubric-scored instrument, and practicing against a good approximation of it is a different activity from practicing against it.

Here's what actually differs, and how to build a prep routine around it.

1. What a real exam tests that a generated question can't reach

Three things, and they compound.

The question format is the skill. AP questions have conventions: how much context comes before the actual ask, how distractors are built, how a multi-part free-response question builds from part (a) to part (d). Students who've done fifty real questions recognize the shape and spend their reading time on the right sentence. Fifty generated questions practice the content and not the format.

FRQ scoring is rubric-based, and the rubric is the exam. This is the big one. A free-response answer isn't graded on whether it's correct in a general sense — it's graded against a published point-by-point rubric. You can be right and score two out of four because you didn't state the assumption the rubric required. A generated question can produce a plausible FRQ prompt; it cannot tell you which four things a reader is instructed to award points for, because that comes from the actual scoring guidelines.

Difficulty calibration comes from real administration data. Real questions have been sat by hundreds of thousands of students. "Make it harder" in a prompt produces a model's idea of harder — usually longer or more obscure rather than more discriminating.

None of this makes generated questions worthless. It makes them a different tool, and the sequencing matters.

2. Where practice questions come from, honestly compared

Two rows deserve a note. Khan Academy is the College Board's official practice partner for AP and its content coverage is good, but it calls its own AP courses supplemental and doesn't give feedback on free-response work — the part this piece keeps returning to. And the two flashcard rows are further apart than they look: Anki schedules each card across days and weeks from how you rated it; Quizlet's Learn mode works within a session. For material you need in three months, that matters.

Read the bottom row without dismissiveness. When you've done every released question on a topic and still feel shaky, generating more is genuinely useful, and general assistants are good at explaining why an answer is wrong once you have one. What generated questions can't do is teach you the instrument — which is why the sequence runs real questions first, generated volume second, never the reverse.

3. Building the routine

Start with a diagnostic, not a study plan. Before deciding what to study, do a timed set of real questions across all units and see where you lose points. Most students find their fear was misallocated — the unit they dreaded is fine, and the one they felt neutral about has a hole in it.

Sovi's AP Test Prep is built around this shape. Practice Test lets you filter by unit, difficulty and question count. That's the shape a diagnostic needs: all units, mixed difficulty, a small set. Time yourself while you do it. Then, once you know where you're weak, the same filters narrow to the unit you're actually losing points on.

Do units in mixed sets, not blocks. Twenty questions on one unit feels productive because repetition within a topic gets smooth fast. That smoothness is not retention.

Taylor and Rohrer tested this directly with mathematics practice: students who practiced interleaved problems performed worse during the practice session, then roughly doubled the blocked group's score on a test a day later. The mechanism they identified is the one that matters for AP — interleaving improves your ability to pair each problem with the right procedure, which is precisely what a mixed-unit exam demands. (Their study was with school-age students on a narrower task than an AP exam, so read it as a strong signal about practice structure rather than a measurement of your situation.)

Your actual exam jumps between units. Your practice should.

Sit at least two full-length mock exams. Students skip it, and it decides more marks than another week of content review. A Full Length Mock Exam with both MCQ and FRQ sections tells you four things nothing else will:

• Pacing. A large share of exam disasters are time management, not knowledge. You find out here or in the room.
• Stamina. Three hours of concentration is a physical capacity, and it has to be rehearsed at least once.
• The MCQ-to-FRQ switch. Moving from rapid multiple choice into extended written justification is a gear change, uncomfortable the first time.
• Where you break under pressure, which is rarely where you break at your own pace.

Track the score, not the feeling. Progress tracking exists so you can see whether your practice is moving the number. A week that felt productive and moved nothing is data worth having in March rather than May.

4. Where general AI genuinely helps

Being specific about this, because the honest answer is "quite a lot, in one particular place."

• Explaining a question you got wrong. You missed it, you have the correct answer, and you don't understand why. ChatGPT, Gemini and Claude are all excellent here, and this is the single highest-value use of general AI in any kind of practice testing.
• Unpacking a concept from scratch. "Explain sampling distributions like I've never seen them." Patient, repeatable, no source needed.
• Extra volume after real questions run out. Genuinely useful for a specific weak unit.
• Drafting your own explanation to check. "I'm going to explain why this answer is right — tell me what I left out."

For the first of those, the format matters as much as the content. A wall of text explaining a multi-step statistics problem is hard to follow at 11pm; watching the steps worked through is easier to hold onto. Sovi's Video Explanation under Ask Sovi is aimed at that gap. And when a question stays confusing after the explanation, Expert Help routes it to a real expert who works it through by hand — which is the one thing no general assistant offers, because there's nobody on the other end of it.

5. Five ways practice testing gets wasted

• Practicing content instead of the assessment. Knowing statistics and scoring well on a statistics exam are related but not identical. The gap is format and rubric.
• Never sitting a full-length paper. You will discover your pacing problem somewhere. Better in March.
• Doing FRQs without checking the scoring guidelines. The rubric is the exam. Answering one without comparing against how it's actually scored is half the exercise.
• Blocking by unit until the last week. Feels organized, produces worse exam performance than mixing.
• Grinding generated questions before exhausting real ones. Real questions are finite and free. Use them first.

<br>

Frequently Asked Questions

Q1: Can AI make good practice questions?

It can make competent questions about the subject matter in seconds, which is useful for extra volume on a weak topic once you've worked through the real ones. What it can't reproduce is the assessment as an instrument: the question format, the difficulty calibration that comes from real administration data, and, most importantly, the point-by-point rubric that free-response answers are scored against. Use generated questions as a supplement, not as your core practice.

Q2: How many full-length practice tests should I do?

Content-wise, as many unit-level question sets as it takes to stop losing points there. Format-wise, at least two full-length timed mock exams including both MCQ and FRQ, and ideally three: one early enough to change your study plan, one in the final fortnight to check pacing, and one in between. Students who've never sat a complete three-hour paper before exam day consistently report the same surprise, and it's always about time.

Q3: How should I study for AP free-response questions specifically?

Answer the question, then score yourself against the actual scoring guidelines rather than against your own sense of whether you were right. The gap between those two is where the marks are. Most students find they lose points not on the mathematics but on stating an assumption, justifying a choice, or answering in context — all of which are rubric requirements you can only learn from the rubric. Then rewrite the answer to earn the points you missed.

Q4: Is it worth using AI at all for practice testing?

Yes, for one thing above all: understanding why you got something wrong. At 11pm, nothing else you have access to comes close to a patient explanation that doesn't get tired of you. Where it doesn't help is in replacing the real question bank or the rubric. The efficient split: real questions and full-length mocks for the practice, AI for the explanation afterward, a human when neither of those gets you there.

Reference

• Taylor & Rohrer, The Effects of Interleaved Practice, Applied Cognitive Psychology (2010)
• Rohrer et al., Interleaved Practice Improves Mathematics Learning (2015)