oiupoyt/whaleprep
1
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Import 9,813 real JEE Main PYQs, KaTeX math, DB indexes

Browse source 
scripts/import_examgoal.py pulls the MIT-licensed examgoal-derived
14,973-question pickle and keeps the 9,813 fully-text MCQs with single
correct answers (2002-2025, all subjects). chapterGroup becomes our
chapter, chapter slug becomes our subtopic. Idempotent: skips stems
already present.

views.rs gains rich() — a read-side sanitizer matching the importer's
allowlist — and all six stem/option/solution render sites use it.
KaTeX auto-render (trust:false) added to head() for $$..$$ math.

db.rs gains nine CREATE INDEX statements; the chapter loader query
drops from ~5s to 6ms on the 9.8k-question bank. DATA.md rewritten
around the real pipeline with attribution.
This commit is contained in:
dev 2026-10-12 10:55:20 +05:30
parent 5c4b8ab7f2
commit 45d2f38b0c
4 changed files with 341 additions and 45 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

80
DATA.md
View file

@ -2,49 +2,59 @@
## Status
The demo seed contains **24 hand-written questions** spread across 9 chapters and 19 subtopics. `scripts/expand_seed.py` inflates this to 500+ synthetic questions so the app can be demonstrated at scale; those questions have real syllabus metadata but sparse placeholder stems.
## Why there is no real scraper
NTA publishes every JEE Main question paper at `jeemain.nta.nic.in/question-papers/`. The files are multi-MB PDFs. The problem:
**Every NTA question paper PDF renders questions as images, not text.** Running `pdftotext` extracts only question IDs and section metadata — no stems, no options. This is intentional (anti-copy protection). Every aggregator site (Embibe, Allen, Aakash, MathonGo) that has structured data built it through manual data-entry pipelines over years.
**`scripts/import_examgoal.py` imports 9,813 real JEE Main PYQs** (2002–2025,
all three subjects) from the MIT-licensed
[jee_mains_pyqs_data_base](https://github.com/HostServer001/jee_mains_pyqs_data_base)
release asset (`1762787474-DataBaseChapters-v007.pkl`, ~24 MB, 14,973 questions
total). The keep rule is: type=mcq, fully text (no `<img>` anywhere in
stem/options/explanation), exactly one correct option in A–D. Skipped: 1,909
image-questions, 2,875 answer-less, 19 multi-correct, ~357 with stray images
in options/explanations.
```
$ pdftotext "B Tech 2nd Apr 2026 Shift 1.pdf" - | head -30
Question Number : 1 Question Id : 6911211 Question Type : MCQ ...
Options :
6911211. ← this is just the option ID, not the text
6911212.
python3 scripts/import_examgoal.py path/to/app.db [--pkl db.pkl]
```
## Realistic paths to a real bank
Safe to re-run: INSERTs skip stems already present, so the 24 hand-written
seed questions are never duplicated. Existing deployments pick up the new
`CREATE INDEX` statements on boot (`init_db` runs them idempotently).
| Method | Effort | Quality |
|---|---|---|
| MathonGo / Embibe API | Paid (₹ ~5–20k/yr) | Verified, tagged, 10k+ questions |
| Allen/Aakash raw exports | Partnership required | Best tagging, solutions included |
| Manual entry from NTA PDFs | ~2 min/question → 350 person-hours for 10k | Accurate, controllable |
| OCR (Tesseract) on NTA PDFs | Medium effort, high error rate on math | Needs human review pass |
| Scrape a coaching site | Legally grey, brittle to layout changes | Varies |
The demo seed (`src/db.rs`, 24 hand-written questions) remains as the offline
fallback when the pickle cannot be fetched. `scripts/expand_seed.py` is kept
for synthetic load testing only.
## What `scripts/expand_seed.py` does
## Attribution
Generates synthetic questions using the real JEE Main syllabus taxonomy (15 chapters, 45 subtopics, correct MCQ/NUM split, correct difficulty distribution per topic). The stems are deliberately sparse (`[Phy/Kinematic/2024S1] Practice question 42: Kinematics concept application.`) so no-one mistakes them for NTA content.
Question content: [HostServer001/jee_mains_pyqs_data_base](https://github.com/HostServer001/jee_mains_pyqs_data_base)
(MIT), itself reverse-engineered from a subscription site's API and cached.
Upstream model credit: examgoal database schema. No NTA PDFs were scraped —
NTA publishes image-rendered PDFs with no machine-readable question text
(`pdftotext` yields only question IDs).
Run with `--count N` to produce N additional questions:
```
python3 scripts/expand_seed.py path/to/app.db --count 1000
```
## Schema mapping
## Adding real questions
| pickle field | our column |
|---|---|
| `chapterGroup` (11 groups) | `chapters.name` |
| `chapter` (91 slugs) | `subtopics.name` |
| `topic` (472 slugs) | folded into subtopic name when distinct |
| `correct_options: ["X"]` | `answer` |
| `question.en.content` (HTML+`$$`) | `stem` (sanitized) |
| `options[].content` | `opt_a..opt_d` (sanitized) |
| `explanation` | `solution` (sanitized) |
| `paperTitle` Morning/Evening | `shift` S1/S2 (examDate hour as fallback) |
| `difficulty` easy/medium/hard | `Easy`/`Medium`/`Hard` |
The schema expects:
```sql
INSERT INTO questions(subject, chapter_id, subtopic_id, qtype,
stem, opt_a, opt_b, opt_c, opt_d, answer,
difficulty, year, shift, solution)
VALUES(...)
```
## Sanitizer contract (both sides match)
`chapter_id` and `subtopic_id` must exist in `chapters` and `subtopics` first. The expand script shows how to upsert those. For NUM questions set `opt_a..opt_d` to empty strings and `answer` to the numeric string (e.g. `"20.00"`).
Write side (`import_examgoal.py::sanitize`) and read side
(`views.rs::rich`) share one allowlist: `br p sup sub b strong i em u ol ul
li table tbody thead tr th td span h3`, attributes always stripped,
`<picture>/<source>/<style>/<col>` blocks dropped whole. KaTeX renders
`$$..$$` display math and `\(..\)` inline at page load with `trust: false`.
## Adding more questions later
Re-run the importer against a newer pickle release. For NUM questions the
upstream data carries no answers, so they are excluded until a verified
answer source exists. The `answer` column for MCQ is a single letter.

216
scripts/import_examgoal.py Normal file
View file

@ -0,0 +1,216 @@
#!/usr/bin/env python3
"""Import real JEE Main PYQs from the HostServer001 examgoal-derived database.
Source: https://github.com/HostServer001/jee_mains_pyqs_data_base (MIT)
Release asset: 1762787474-DataBaseChapters-v007.pkl (~24 MB, 14,973 questions).
What this does:
1. Downloads the pickle (or uses --pkl if you already have it).
2. Keeps only: type=mcq, fully text (no <img anywhere in stem/options/
explanation), exactly one correct option in A-D.
3. Sanitizes HTML to a safe allowlist, unwraps $$..$$ math for KaTeX.
4. Maps chapterGroup -> our chapter, chapter -> our subtopic, creating
rows as needed. Keeps the existing 24 seed questions untouched
(dedupes on exact stem match).
5. Shifts: Morning Slot/Shift -> S1, Evening -> S2, Offline/AIEEE -> S1.
Usage:
python3 scripts/import_examgoal.py path/to/app.db [--pkl db.pkl]
Safe to re-run: INSERTs skip stems that already exist.
"""
import argparse, html, pickle, re, sqlite3, sys, urllib.request
from pathlib import Path
PKL_URL = ("https://github.com/HostServer001/jee_mains_pyqs_data_base"
"/releases/download/v007/1762787474-DataBaseChapters-v007.pkl")
ALLOWED = {"br", "p", "sup", "sub", "b", "strong", "i", "em", "u",
"ol", "ul", "li", "table", "tbody", "thead", "tr", "th", "td",
"span", "h3"}
SUBJECT_FIX = {"mathematics": "Maths", "physics": "Physics", "chemistry": "Chemistry"}
def title_case(slug: str) -> str:
return " ".join(w.capitalize() for w in slug.replace("-", " ").split())
def sanitize(raw: str) -> str:
"""Strip to safe HTML. Unescape entities first so &lt; renders as <. KaTeX
handles $$..$$ at render time; nothing else needs dollar-escaping."""
if not raw:
return ""
text = html.unescape(str(raw))
# Drop whole <picture>/<source>/<style> blocks (responsive image variants).
text = re.sub(r"<picture.*?</picture>", "", text, flags=re.S | re.I)
text = re.sub(r"<source[^>]*>", "", text, flags=re.I)
text = re.sub(r"<style.*?</style>", "", text, flags=re.S | re.I)
text = re.sub(r"<colgroup.*?</colgroup>", "", text, flags=re.S | re.I)
text = re.sub(r"</?(?:col|colgroup)[^>]*>", "", text, flags=re.I)
# Keep only allowlisted tags, drop all attributes.
def tag(m):
closing, name = m.group(1), m.group(2).lower()
if name in ALLOWED:
return f"<{closing}{name}>"
return ""
text = re.sub(r"<(/?)([a-zA-Z][a-zA-Z0-9]*)(\s[^>]*)?>", tag, text)
# Collapse whitespace runs but keep single spaces inside math.
text = re.sub(r"[ \t\xa0]+", " ", text)
text = re.sub(r"\n\s*\n+", "\n", text)
return text.strip()
def shift_of(paper_title: str, exam_date) -> str:
t = (paper_title or "").lower()
if "evening" in t:
return "S2"
if "morning" in t:
return "S1"
if exam_date:
try:
hour = int(str(exam_date).split()[1].split(":")[0])
return "S2" if hour >= 12 else "S1"
except (IndexError, ValueError):
pass
return "S1"
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("db")
ap.add_argument("--pkl", default=None)
args = ap.parse_args()
pkl_path = Path(args.pkl) if args.pkl else Path("/tmp/pyqeval/db.pkl")
if not pkl_path.exists():
print(f"downloading {PKL_URL} ...", flush=True)
pkl_path.parent.mkdir(parents=True, exist_ok=True)
urllib.request.urlretrieve(PKL_URL, pkl_path)
print(f"loading {pkl_path} ({pkl_path.stat().st_size / 1e6:.1f} MB) ...", flush=True)
sys.path.insert(0, "/tmp/pyqeval")
from pkg.core.question import Question
from pkg.core.chapter import Chapter
class FixUnpickler(pickle.Unpickler):
def find_class(self, module, name):
if name == "Question":
return Question
if name == "Chapter":
return Chapter
return super().find_class(module, name)
with open(pkl_path, "rb") as f:
data = FixUnpickler(f).load()
con = sqlite3.connect(args.db)
existing = {r[0] for r in con.execute("SELECT stem FROM questions")}
print(f"existing stems in db: {len(existing)}")
chap_cache: dict = {}
sub_cache: dict = {}
def chapter_id(subject: str, chapter: str) -> int:
key = (subject, chapter)
if key not in chap_cache:
row = con.execute(
"SELECT id FROM chapters WHERE subject=? AND name=?",
(subject, chapter)).fetchone()
chap_cache[key] = (row[0] if row
else con.execute(
"INSERT INTO chapters(subject,name) VALUES(?,?)",
(subject, chapter)).lastrowid)
return chap_cache[key]
def subtopic_id(ch_id: int, sub: str) -> int:
key = (ch_id, sub)
if key not in sub_cache:
row = con.execute(
"SELECT id FROM subtopics WHERE chapter_id=? AND name=?",
(ch_id, sub)).fetchone()
sub_cache[key] = (row[0] if row
else con.execute(
"INSERT INTO subtopics(chapter_id,name) VALUES(?,?)",
(ch_id, sub)).lastrowid)
return sub_cache[key]
kept = skipped = 0
diff_map = {"easy": "Easy", "medium": "Medium", "hard": "Hard"}
for _key, chap in data.items():
for q in chap.question_dict.values():
if getattr(q, "type", None) != "mcq":
skipped += 1
continue
if getattr(q, "isImgQuestion", False):
skipped += 1
continue
co = getattr(q, "correct_options", None) or []
if len(co) != 1 or co[0] not in ("A", "B", "C", "D"):
skipped += 1
continue
stem = sanitize(getattr(q, "question", ""))
if not stem or "<img" in stem:
skipped += 1
continue
opt_contents = []
bad = False
for opt in (getattr(q, "options", None) or []):
c = sanitize(opt.get("content", "") if isinstance(opt, dict) else "")
if not c or "<img" in c:
bad = True
break
opt_contents.append(c)
if bad or len(opt_contents) != 4:
skipped += 1
continue
solution = sanitize(getattr(q, "explanation", ""))
if "<img" in solution:
skipped += 1
continue
if stem in existing:
skipped += 1
continue
subject = SUBJECT_FIX.get(
str(getattr(chap, "parent_subject", "")).lower(),
str(getattr(chap, "parent_subject", "")).title())
chapter = title_case(str(getattr(q, "chpaterGroup", None)
or getattr(q, "chapter", "General")))
subtopic = title_case(str(getattr(q, "topic", None)
or getattr(q, "chapter", "General")))
ch_id = chapter_id(subject, chapter)
st_id = subtopic_id(ch_id, subtopic)
diff = diff_map.get(str(getattr(q, "difficulty", "medium")).lower(),
"Medium")
try:
year = int(getattr(q, "year", 0) or 0)
except (TypeError, ValueError):
year = 0
shift = shift_of(getattr(q, "paperTitle", ""), getattr(q, "examDate", None))
con.execute(
"INSERT INTO questions(subject,chapter_id,subtopic_id,qtype,"
"stem,opt_a,opt_b,opt_c,opt_d,answer,difficulty,year,shift,solution)"
" VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?)",
(subject, ch_id, st_id, "MCQ", stem,
opt_contents[0], opt_contents[1],
opt_contents[2], opt_contents[3],
co[0], diff, year, shift, solution))
existing.add(stem)
kept += 1
con.commit()
n_q = con.execute("SELECT COUNT(*) FROM questions").fetchone()[0]
n_c = con.execute("SELECT COUNT(*) FROM chapters").fetchone()[0]
n_s = con.execute("SELECT COUNT(*) FROM subtopics").fetchone()[0]
print(f"imported {kept}, skipped {skipped}")
print(f"db now: {n_q} questions / {n_c} chapters / {n_s} subtopics")
con.close()
if __name__ == "__main__":
main()

11
src/db.rs
View file

@ -13,7 +13,16 @@ pub async fn init_db(pool: &SqlitePool) -> anyhow::Result<()> {
CREATE TABLE IF NOT EXISTS mistake_tags(attempt_id TEXT NOT NULL, question_id INTEGER NOT NULL, tag TEXT NOT NULL, PRIMARY KEY(attempt_id, question_id, tag));
CREATE TABLE IF NOT EXISTS sessions(token TEXT PRIMARY KEY, user_id TEXT NOT NULL, created_at TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS pkce(state TEXT PRIMARY KEY, verifier TEXT NOT NULL, created_at TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS seen_questions(user_id TEXT NOT NULL, question_id INTEGER NOT NULL, PRIMARY KEY(user_id, question_id));",
CREATE TABLE IF NOT EXISTS seen_questions(user_id TEXT NOT NULL, question_id INTEGER NOT NULL, PRIMARY KEY(user_id, question_id));
CREATE INDEX IF NOT EXISTS idx_questions_chapter ON questions(chapter_id);
CREATE INDEX IF NOT EXISTS idx_questions_subtopic ON questions(subtopic_id);
CREATE INDEX IF NOT EXISTS idx_questions_subject ON questions(subject);
CREATE INDEX IF NOT EXISTS idx_questions_year ON questions(year);
CREATE INDEX IF NOT EXISTS idx_subtopics_chapter ON subtopics(chapter_id);
CREATE INDEX IF NOT EXISTS idx_papers_user ON papers(user_id);
CREATE INDEX IF NOT EXISTS idx_attempts_paper ON attempts(paper_id);
CREATE INDEX IF NOT EXISTS idx_attempts_user ON attempts(user_id);
CREATE INDEX IF NOT EXISTS idx_answers_attempt ON answers(attempt_id);",
)
.execute(pool)
.await?;

79
src/views.rs
View file

@ -17,6 +17,58 @@ pub fn esc(s: &str) -> String {
o
}
/// Render question body HTML stored in the DB. The importer sanitizes at write
/// time; this re-sanitizes on read so a hand-edited row can never inject a
/// script. Only the sanitizer's allowlist survives — see
/// scripts/import_examgoal.py for the matching write-side list.
pub fn rich(s: &str) -> String {
const ALLOWED: &[&str] = &[
"br", "p", "sup", "sub", "b", "strong", "i", "em", "u", "ol", "ul",
"li", "table", "tbody", "thead", "tr", "th", "td", "span", "h3",
];
let mut o = String::with_capacity(s.len());
let b = s.as_bytes();
let mut i = 0;
while i < b.len() {
if b[i] != b'<' {
// Escape text nodes so a stray "<3" or raw & cannot break markup.
match b[i] as char {
'&' => o.push_str("&amp;"),
'<' => o.push_str("&lt;"),
'>' => o.push_str("&gt;"),
c => o.push(c),
}
i += 1;
continue;
}
let Some(end) = s[i..].find('>') else {
o.push_str("&lt;");
i += 1;
continue;
};
let inner = &s[i + 1..i + end];
let inner = inner.trim();
let (closing, name) = match inner.strip_prefix('/') {
Some(rest) => (true, rest.split_whitespace().next().unwrap_or("")),
None => (false, inner.split_whitespace().next().unwrap_or("")),
};
let name = name.trim_end_matches('/').to_ascii_lowercase();
// Reject anything with attributes or a non-allowlisted tag. A trailing
// slash (self-closing <br/>) is fine and needs no attribute parsing.
let has_attrs = inner.contains(char::is_whitespace) && !inner.ends_with('/');
if !has_attrs && ALLOWED.contains(&name.as_str()) {
o.push('<');
if closing {
o.push('/');
}
o.push_str(&name);
o.push('>');
}
i += end + 1;
}
o
}
pub struct SubtopicRow {
pub id: i64,
pub name: String,
@ -40,9 +92,18 @@ fn head(ctx: &Ctx, title: &str) -> String {
format!(
r#"<!doctype html><html lang="en"><head><meta charset="utf-8">
<meta name="viewport" content="width=device-width,initial-scale=1">
<title>{t} · PYQ</title><link rel="stylesheet" href="{b}/static/style.css">
<title>{t} · WhalePrep</title><link rel="stylesheet" href="{b}/static/style.css">
<link rel="preconnect" href="https://fonts.googleapis.com">
<link href="https://fonts.googleapis.com/css2?family=JetBrains+Mono:wght@400;500;600&display=swap" rel="stylesheet">
<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.11/dist/katex.min.css">
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.11/dist/katex.min.js"></script>
<script defer src="https://cdn.jsdelivr.net/npm/katex@0.16.11/dist/contrib/auto-render.min.js"></script>
<script>document.addEventListener("DOMContentLoaded", function() {{
if (window.renderMathInElement) renderMathInElement(document.body, {{
delimiters: [{{left: "$$", right: "$$", display: true}}, {{left: "\\(", right: "\\)", display: false}}],
trust: false, strict: false, throwOnError: false
}});
}});</script>
</head><body>"#,
t = esc(title),
b = ctx.base
@ -51,7 +112,7 @@ fn head(ctx: &Ctx, title: &str) -> String {
fn nav(ctx: &Ctx, user: Option<&str>) -> String {
let brand = format!(
r#"<header class="topbar"><a class="brand" href="{b}/">PYQ<span>jee main practice</span></a><nav class="nav">"#,
r#"<header class="topbar"><a class="brand" href="{b}/">WhalePrep<span>jee main practice</span></a><nav class="nav">"#,
b = ctx.base
);
let links = match user {
@ -451,7 +512,7 @@ pub fn attempt(ctx: &Ctx, user: &str, attempt_id: &str, questions: &[Question],
id = q.id,
letter = letter,
checked = checked,
t = esc(&text)
t = rich(&text)
)
})
.collect()
@ -481,7 +542,7 @@ pub fn attempt(ctx: &Ctx, user: &str, attempt_id: &str, questions: &[Question],
ch = esc(&q.chapter),
qt = esc(&q.qtype),
revflag = if rev { " · <span class=\"warn-t\">marked</span>" } else { "" },
stem = esc(&q.stem),
stem = rich(&q.stem),
sub2 = esc(&q.subtopic),
year = q.year,
shift = esc(&q.shift),
@ -605,7 +666,7 @@ pub fn result(ctx: &Ctx, user: &str, _attempt_id: &str, a: &Analysis, rows: &[Ro
let is_ans = r.q.answer.trim().eq_ignore_ascii_case(l);
let is_sel = r.selected.trim().eq_ignore_ascii_case(l);
let style = if is_ans { "ok-t" } else if is_sel { "bad-t" } else { "dim" };
format!(r#"<span style="margin-right:10px" class="{style}">{l}: {t}{mark}</span>"#, style = style, l = l, t = esc(&r.q.opts[k]), mark = if is_ans { " (correct)" } else if is_sel { " (you)" } else { "" })
format!(r#"<span style="margin-right:10px" class="{style}">{l}: {t}{mark}</span>"#, style = style, l = l, t = rich(&r.q.opts[k]), mark = if is_ans { " (correct)" } else if is_sel { " (you)" } else { "" })
})
.collect()
};
@ -620,7 +681,7 @@ pub fn result(ctx: &Ctx, user: &str, _attempt_id: &str, a: &Analysis, rows: &[Ro
cls = cls,
state = state,
marks = r.marks,
stem = esc(&r.q.stem),
stem = rich(&r.q.stem),
opts = opts,
sub2 = esc(&r.q.subtopic),
ch = esc(&r.q.chapter),
@ -628,7 +689,7 @@ pub fn result(ctx: &Ctx, user: &str, _attempt_id: &str, a: &Analysis, rows: &[Ro
shift = esc(&r.q.shift),
difficulty = esc(&r.q.difficulty),
qtime = if r.time_secs > 0 { format!("{}s spent", r.time_secs) } else { "no time recorded".to_string() },
sol = esc(&r.q.solution)
sol = rich(&r.q.solution)
)
})
.collect();
@ -784,7 +845,7 @@ pub fn shared(ctx: &Ctx, title: &str, duration: i64, questions: &[Question]) ->
["A", "B", "C", "D"]
.iter()
.enumerate()
.map(|(k, l)| format!("<p><b>{l}</b> {}</p>", esc(&q.opts[k])))
.map(|(k, l)| format!("<p><b>{l}</b> {}</p>", rich(&q.opts[k])))
.collect()
};
format!(
@ -792,7 +853,7 @@ pub fn shared(ctx: &Ctx, title: &str, duration: i64, questions: &[Question]) ->
n = i + 1,
subj = esc(&q.subject),
sub2 = esc(&q.subtopic),
stem = esc(&q.stem),
stem = rich(&q.stem),
opts = opts
)
})