45d2f38b0c
scripts/import_examgoal.py pulls the MIT-licensed examgoal-derived 14,973-question pickle and keeps the 9,813 fully-text MCQs with single correct answers (2002-2025, all subjects). chapterGroup becomes our chapter, chapter slug becomes our subtopic. Idempotent: skips stems already present. views.rs gains rich() — a read-side sanitizer matching the importer's allowlist — and all six stem/option/solution render sites use it. KaTeX auto-render (trust:false) added to head() for $$..$$ math. db.rs gains nine CREATE INDEX statements; the chapter loader query drops from ~5s to 6ms on the 9.8k-question bank. DATA.md rewritten around the real pipeline with attribution.
2.6 KiB
Data sourcing — JEE Main question bank
Status
scripts/import_examgoal.py imports 9,813 real JEE Main PYQs (2002–2025,
all three subjects) from the MIT-licensed
jee_mains_pyqs_data_base
release asset (1762787474-DataBaseChapters-v007.pkl, ~24 MB, 14,973 questions
total). The keep rule is: type=mcq, fully text (no <img> anywhere in
stem/options/explanation), exactly one correct option in A–D. Skipped: 1,909
image-questions, 2,875 answer-less, 19 multi-correct, ~357 with stray images
in options/explanations.
python3 scripts/import_examgoal.py path/to/app.db [--pkl db.pkl]
Safe to re-run: INSERTs skip stems already present, so the 24 hand-written
seed questions are never duplicated. Existing deployments pick up the new
CREATE INDEX statements on boot (init_db runs them idempotently).
The demo seed (src/db.rs, 24 hand-written questions) remains as the offline
fallback when the pickle cannot be fetched. scripts/expand_seed.py is kept
for synthetic load testing only.
Attribution
Question content: HostServer001/jee_mains_pyqs_data_base
(MIT), itself reverse-engineered from a subscription site's API and cached.
Upstream model credit: examgoal database schema. No NTA PDFs were scraped —
NTA publishes image-rendered PDFs with no machine-readable question text
(pdftotext yields only question IDs).
Schema mapping
| pickle field | our column |
|---|---|
chapterGroup (11 groups) |
chapters.name |
chapter (91 slugs) |
subtopics.name |
topic (472 slugs) |
folded into subtopic name when distinct |
correct_options: ["X"] |
answer |
question.en.content (HTML+$$) |
stem (sanitized) |
options[].content |
opt_a..opt_d (sanitized) |
explanation |
solution (sanitized) |
paperTitle Morning/Evening |
shift S1/S2 (examDate hour as fallback) |
difficulty easy/medium/hard |
Easy/Medium/Hard |
Sanitizer contract (both sides match)
Write side (import_examgoal.py::sanitize) and read side
(views.rs::rich) share one allowlist: br p sup sub b strong i em u ol ul li table tbody thead tr th td span h3, attributes always stripped,
<picture>/<source>/<style>/<col> blocks dropped whole. KaTeX renders
$$..$$ display math and \(..\) inline at page load with trust: false.
Adding more questions later
Re-run the importer against a newer pickle release. For NUM questions the
upstream data carries no answers, so they are excluded until a verified
answer source exists. The answer column for MCQ is a single letter.