dev 3df1d62002 Add expand_seed and document data sourcing

scripts/expand_seed.py generates synthetic questions across the full JEE Main
syllabus taxonomy (15 chapters, 45 subtopics, correct MCQ/NUM split). Stems
are deliberately sparse so no-one mistakes them for NTA content.

DATA.md documents why there is no real scraper: NTA PDFs render questions as
images; pdftotext extracts only IDs. Every aggregator with structured data
built it through manual entry pipelines. The document lists realistic paths
to a real bank and the SQL schema for loading one.

2026-10-12 05:19:03 +05:30

2.5 KiB

Raw Blame History

Data sourcing — JEE Main question bank

Status

The demo seed contains 24 hand-written questions spread across 9 chapters and 19 subtopics. scripts/expand_seed.py inflates this to 500+ synthetic questions so the app can be demonstrated at scale; those questions have real syllabus metadata but sparse placeholder stems.

Why there is no real scraper

NTA publishes every JEE Main question paper at jeemain.nta.nic.in/question-papers/. The files are multi-MB PDFs. The problem:

Every NTA question paper PDF renders questions as images, not text. Running pdftotext extracts only question IDs and section metadata — no stems, no options. This is intentional (anti-copy protection). Every aggregator site (Embibe, Allen, Aakash, MathonGo) that has structured data built it through manual data-entry pipelines over years.

$ pdftotext "B Tech 2nd Apr 2026 Shift 1.pdf" - | head -30
Question Number : 1 Question Id : 6911211 Question Type : MCQ ...
Options :
6911211.       ← this is just the option ID, not the text
6911212.

Realistic paths to a real bank

Method	Effort	Quality
MathonGo / Embibe API	Paid (₹ ~5–20k/yr)	Verified, tagged, 10k+ questions
Allen/Aakash raw exports	Partnership required	Best tagging, solutions included
Manual entry from NTA PDFs	~2 min/question → 350 person-hours for 10k	Accurate, controllable
OCR (Tesseract) on NTA PDFs	Medium effort, high error rate on math	Needs human review pass
Scrape a coaching site	Legally grey, brittle to layout changes	Varies

What `scripts/expand_seed.py` does

Generates synthetic questions using the real JEE Main syllabus taxonomy (15 chapters, 45 subtopics, correct MCQ/NUM split, correct difficulty distribution per topic). The stems are deliberately sparse ([Phy/Kinematic/2024S1] Practice question 42: Kinematics concept application.) so no-one mistakes them for NTA content.

Run with --count N to produce N additional questions:

python3 scripts/expand_seed.py path/to/app.db --count 1000

Adding real questions

The schema expects:

INSERT INTO questions(subject, chapter_id, subtopic_id, qtype,
  stem, opt_a, opt_b, opt_c, opt_d, answer,
  difficulty, year, shift, solution)
VALUES(...)

chapter_id and subtopic_id must exist in chapters and subtopics first. The expand script shows how to upsert those. For NUM questions set opt_a..opt_d to empty strings and answer to the numeric string (e.g. "20.00").

2.5 KiB Raw Blame History Unescape Escape