Tübel

A self-contained English-language web search engine for content about Tübingen, built for the INFO4271 “Modern Search Engines” project. Crawling, indexing and first-stage retrieval are written from scratch on top of requests, BeautifulSoup, NLTK (stemmer and stopword lists only), numpy/scipy and scikit-learn (clustering only), with Flask for the UI.

There are three ranking modes: bm25 (BM25F over documents), rm3 (BM25F plus RM3 pseudo-relevance feedback, the default) and neural (BM25 over passages, re-ranked by an instruction-tuned language model).

Crawled index: https://cloud.marvinborner.de/s/NbwpqibnZmY9AMf (move to data/)

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt # (depending on GPU setup, you may require a different version of torch)

NLTK’s stopwords corpus is the only data download; the crawler fetches it on first run if it is missing.

The classical modes need nothing else. neural additionally pulls google/flan-t5-base (~1 GB).

Quick Start

First crawl some pages after the setup

python crawler.py           # crawl until the frontier is empty
python crawler.py 5000      # stop after 5000 newly fetched pages

Then run Tübel

python app.py            # http://localhost:5000

Files

File	What it does
`crawler.py`	Focused crawler. Priority frontier and fetched pages in `data/crawl.db`.
`seeds.txt`	Start URLs.
`indexer.py`	Parses stored pages into fields, tokenizes and stems, writes the index. Also the shared tokenizer.
`index_utils.py`	Index loading and IDF, query terms, page text, snippets, domain cap, result rows.
`rm3.py`	BM25F over whole documents plus RM3 query expansion.
`neural.py`	BM25 over passages, re-ranked by query likelihood under `flan-t5-base`, fused by RRF.
`retrieval.py`	One `search(query, k, mode)` over the two pipelines.
`batch.py`	Batch mode.
`app.py`, `templates/search.html`, `static/`	Flask UI: list view and graph view (d3 is vendored).
`graph.py`	tf-idf similarity over the index, k-means clusters, graph construction.
`analyze_index.py`, `analysis_plots.py`	Offline PNG report on crawl coverage and index quality.
`data/`	`crawl.db`, `search_index.pkl`, `page_text.db`

Crawling

python crawler.py           # crawl until the frontier is empty
python crawler.py 5000      # stop after 5000 newly fetched pages

Ctrl-C at any point is safe: every page and every frontier change is committed immediately, so rerunning simply continues where it stopped. Nothing already in pages is fetched twice.

Our crawled corpus in data/crawl.db is 63k fetched pages; 205k were rejected as off-topic, 120k by the language gate, and 94k did not return HTTP 200. Roughly 450k URLs are still queued. Due to its size, we have not published it here.

Indexing

python indexer.py                  # data/crawl.db -> the two files below
python indexer.py data/other.db

Writes data/search_index.pkl (postings, document frequencies, field lengths, passage layout, document metadata) and data/page_text.db (page bodies, read per query for snippets and re-ranker input rather than held in memory).

Our current index holds 43,105 documents, 714,225 passages and 667k terms.

Searching

python retrieval.py "tübingen attractions"                # default: rm3
python retrieval.py --mode bm25 "tübingen attractions"
python retrieval.py --mode neural "tübingen attractions"

python rm3.py "tübingen attractions"                      # each pipeline also runs alone
python neural.py --no-rerank "tübingen attractions"

Batch mode takes the same mode names. Input is qid<TAB>query per line, output is qid<TAB>rank<TAB>url<TAB>score, 100 lines per query:

python batch.py data/queries.txt data/results.txt rm3

Once the index is loaded a query costs 0.3-1.8 s in bm25 and rm3. neural is far slower, since it runs 150 passages through the model, and is opt-in for that reason.

Web app

python app.py            # http://localhost:5000

The list view shows the top 100 results with query terms bolded in title and snippet, the RM3 expansion terms as related-term hints under the search box, and the k-means cluster labels as chips that filter the list. The ranking mode is switchable in the footer.

The graph view draws the result neighbourhood: the query in the middle, pages as nodes coloured by cluster and sized by relevance to the result centroid, edges between pages with similar content. Clicking a node opens it in the side panel, and any node can be made the new centre. Relatedness is tf-idf cosine over the same inverted index the ranker uses, inverted lazily into a sparse document-term matrix on the first graph request. Graph data without the browser:

python graph.py tübingen attractions

Known limitations

The index is a pickle of Python dicts held fully in memory.
The crawler is single-threaded, so throughput is bounded by the per-host politeness delay.
The language gate still admits a few pages in languages NLTK has no stopword list for.
Duplicate removal at index time is exact-match only, so near-duplicates survive and can fill the RM3 feedback set.
Output scores are raw retrieval scores, not normalized to a fixed range.
English Tübingen content is dominated by the university, the research institutes, and GitHub repositories.