45d2f38b0c
scripts/import_examgoal.py pulls the MIT-licensed examgoal-derived 14,973-question pickle and keeps the 9,813 fully-text MCQs with single correct answers (2002-2025, all subjects). chapterGroup becomes our chapter, chapter slug becomes our subtopic. Idempotent: skips stems already present. views.rs gains rich() — a read-side sanitizer matching the importer's allowlist — and all six stem/option/solution render sites use it. KaTeX auto-render (trust:false) added to head() for $$..$$ math. db.rs gains nine CREATE INDEX statements; the chapter loader query drops from ~5s to 6ms on the 9.8k-question bank. DATA.md rewritten around the real pipeline with attribution.
60 lines
2.6 KiB
Markdown
60 lines
2.6 KiB
Markdown
# Data sourcing — JEE Main question bank
|
||
|
||
## Status
|
||
|
||
**`scripts/import_examgoal.py` imports 9,813 real JEE Main PYQs** (2002–2025,
|
||
all three subjects) from the MIT-licensed
|
||
[jee_mains_pyqs_data_base](https://github.com/HostServer001/jee_mains_pyqs_data_base)
|
||
release asset (`1762787474-DataBaseChapters-v007.pkl`, ~24 MB, 14,973 questions
|
||
total). The keep rule is: type=mcq, fully text (no `<img>` anywhere in
|
||
stem/options/explanation), exactly one correct option in A–D. Skipped: 1,909
|
||
image-questions, 2,875 answer-less, 19 multi-correct, ~357 with stray images
|
||
in options/explanations.
|
||
|
||
```
|
||
python3 scripts/import_examgoal.py path/to/app.db [--pkl db.pkl]
|
||
```
|
||
|
||
Safe to re-run: INSERTs skip stems already present, so the 24 hand-written
|
||
seed questions are never duplicated. Existing deployments pick up the new
|
||
`CREATE INDEX` statements on boot (`init_db` runs them idempotently).
|
||
|
||
The demo seed (`src/db.rs`, 24 hand-written questions) remains as the offline
|
||
fallback when the pickle cannot be fetched. `scripts/expand_seed.py` is kept
|
||
for synthetic load testing only.
|
||
|
||
## Attribution
|
||
|
||
Question content: [HostServer001/jee_mains_pyqs_data_base](https://github.com/HostServer001/jee_mains_pyqs_data_base)
|
||
(MIT), itself reverse-engineered from a subscription site's API and cached.
|
||
Upstream model credit: examgoal database schema. No NTA PDFs were scraped —
|
||
NTA publishes image-rendered PDFs with no machine-readable question text
|
||
(`pdftotext` yields only question IDs).
|
||
|
||
## Schema mapping
|
||
|
||
| pickle field | our column |
|
||
|---|---|
|
||
| `chapterGroup` (11 groups) | `chapters.name` |
|
||
| `chapter` (91 slugs) | `subtopics.name` |
|
||
| `topic` (472 slugs) | folded into subtopic name when distinct |
|
||
| `correct_options: ["X"]` | `answer` |
|
||
| `question.en.content` (HTML+`$$`) | `stem` (sanitized) |
|
||
| `options[].content` | `opt_a..opt_d` (sanitized) |
|
||
| `explanation` | `solution` (sanitized) |
|
||
| `paperTitle` Morning/Evening | `shift` S1/S2 (examDate hour as fallback) |
|
||
| `difficulty` easy/medium/hard | `Easy`/`Medium`/`Hard` |
|
||
|
||
## Sanitizer contract (both sides match)
|
||
|
||
Write side (`import_examgoal.py::sanitize`) and read side
|
||
(`views.rs::rich`) share one allowlist: `br p sup sub b strong i em u ol ul
|
||
li table tbody thead tr th td span h3`, attributes always stripped,
|
||
`<picture>/<source>/<style>/<col>` blocks dropped whole. KaTeX renders
|
||
`$$..$$` display math and `\(..\)` inline at page load with `trust: false`.
|
||
|
||
## Adding more questions later
|
||
|
||
Re-run the importer against a newer pickle release. For NUM questions the
|
||
upstream data carries no answers, so they are excluded until a verified
|
||
answer source exists. The `answer` column for MCQ is a single letter.
|