cc0cd75ff9
Integer explanations end with the answer but phrasing varies enough that regex extraction would mis-key questions. A wrong key is worse than no question. NUM stays hand-added until NTA keys transcribed.
71 lines
3.2 KiB
Markdown
71 lines
3.2 KiB
Markdown
# Data sourcing — JEE Main question bank
|
||
|
||
## Status
|
||
|
||
**`scripts/import_examgoal.py` imports 9,813 real JEE Main PYQs** (2002–2025,
|
||
all three subjects) from the MIT-licensed
|
||
[jee_mains_pyqs_data_base](https://github.com/HostServer001/jee_mains_pyqs_data_base)
|
||
release asset (`1762787474-DataBaseChapters-v007.pkl`, ~24 MB, 14,973 questions
|
||
total). The keep rule is: type=mcq, fully text (no `<img>` anywhere in
|
||
stem/options/explanation), exactly one correct option in A–D. Skipped: 1,909
|
||
image-questions, 2,875 answer-less, 19 multi-correct, ~357 with stray images
|
||
in options/explanations.
|
||
|
||
```
|
||
python3 scripts/import_examgoal.py path/to/app.db [--pkl db.pkl]
|
||
```
|
||
|
||
Safe to re-run: INSERTs skip stems already present, so the 24 hand-written
|
||
seed questions are never duplicated. Existing deployments pick up the new
|
||
`CREATE INDEX` statements on boot (`init_db` runs them idempotently).
|
||
|
||
The demo seed (`src/db.rs`, 24 hand-written questions) remains as the offline
|
||
fallback when the pickle cannot be fetched. `scripts/expand_seed.py` is kept
|
||
for synthetic load testing only.
|
||
|
||
## Attribution
|
||
|
||
Question content: [HostServer001/jee_mains_pyqs_data_base](https://github.com/HostServer001/jee_mains_pyqs_data_base)
|
||
(MIT), itself reverse-engineered from a subscription site's API and cached.
|
||
Upstream model credit: examgoal database schema. No NTA PDFs were scraped —
|
||
NTA publishes image-rendered PDFs with no machine-readable question text
|
||
(`pdftotext` yields only question IDs).
|
||
|
||
## Schema mapping
|
||
|
||
| pickle field | our column |
|
||
|---|---|
|
||
| `chapterGroup` (11 groups) | `chapters.name` |
|
||
| `chapter` (91 slugs) | `subtopics.name` |
|
||
| `topic` (472 slugs) | folded into subtopic name when distinct |
|
||
| `correct_options: ["X"]` | `answer` |
|
||
| `question.en.content` (HTML+`$$`) | `stem` (sanitized) |
|
||
| `options[].content` | `opt_a..opt_d` (sanitized) |
|
||
| `explanation` | `solution` (sanitized) |
|
||
| `paperTitle` Morning/Evening | `shift` S1/S2 (examDate hour as fallback) |
|
||
| `difficulty` easy/medium/hard | `Easy`/`Medium`/`Hard` |
|
||
|
||
## Sanitizer contract (both sides match)
|
||
|
||
Write side (`import_examgoal.py::sanitize`) and read side
|
||
(`views.rs::rich`) share one allowlist: `br p sup sub b strong i em u ol ul
|
||
li table tbody thead tr th td span h3`, attributes always stripped,
|
||
`<picture>/<source>/<style>/<col>` blocks dropped whole. KaTeX renders
|
||
`$$..$$` display math and `\(..\)` inline at page load with `trust: false`.
|
||
|
||
## Adding more questions later
|
||
|
||
Re-run the importer against a newer pickle release. For NUM questions the
|
||
upstream data carries no answers, so they are excluded until a verified
|
||
answer source exists. The `answer` column for MCQ is a single letter.
|
||
|
||
## Why NUM answers are not extracted from explanations
|
||
|
||
Audited 2026-10-12: integer-question explanations do end with the numeric
|
||
result (`= 80`, `Total 11`, `is 8`), but the phrasing varies per question
|
||
(`Hence`, `Clearly`, `no. of integers in S is`, bare `= X` mid-derivation)
|
||
and several tails contain multiple numbers. A regex would mis-key a
|
||
meaningful fraction, and a wrong answer key teaches the wrong result —
|
||
worse than no question. NUM import stays out until NTA final answer keys
|
||
(authoritative per-shift source) are transcribed. The grader already
|
||
supports NUM with 0.1% tolerance for hand-added questions.
|