1. 1 Projects
  2. 2 Details
  3. 3 Code on GitHub

Open source | Python | CLI and library

research-harvest

A command-line tool and Python library for literature screening. It searches PubMed and OpenAlex, merges and deduplicates the results, reads every abstract for the methods used, the cohort size and the best score reported, and writes it all to JSONL, CSV or Excel.

  • Python 3.9+
  • requests
  • pytest
  • responses
  • ruff
  • openpyxl
  • GitHub Actions
115
tests, all running offline
95%
test coverage, and CI fails below 90%
5
Python versions in CI, 3.9 to 3.13
114
method names it recognises, in six families

Overview

What it does

Screening papers by hand means opening every abstract to find out what was done, on how much data, and how well it worked. research-harvest does that first pass. Give it a query and it returns one row per paper with the models the abstract mentions, the size of the dataset or cohort, and the headline score, ready to sort and filter.

It talks to both catalogues over their official REST APIs and paces itself to each one's rate limit: 3 requests a second for PubMed, or 9 with an NCBI key, and 5 for OpenAlex. If one source fails, the error is recorded and the run carries on with the other.

Type
CLI and Python library
Language
Python 3.9 or newer
Dependencies
requests, plus openpyxl for Excel output
Sources
PubMed E-utilities and OpenAlex
Output
JSONL, CSV or Excel
Licence
MIT

How it works

From a search query to a spreadsheet

  1. Read the command

    main() validates the flags, exits with status 2 on bad input, and turns the filters into a FilterSpec.

    src/research_harvest/cli.py

  2. Run every source

    harvest() runs each source in turn. A SourceError is recorded in the result and the remaining sources still run.

    src/research_harvest/harvest.py

  3. Search PubMed

    esearch returns the matching ids as JSON, and efetch returns the records as XML, 200 per call.

    sources/pubmed.py

  4. Search OpenAlex

    Results come back page by page with a cursor, and each abstract is rebuilt from OpenAlex's inverted index. Both sources share one HTTP client with a rate limiter and produce the same Article model.

    sources/openalex.py, sources/base.py, models.py

  5. Clean the text

    Unescape HTML entities, strip tags and cut the copyright notices that publishers append to abstracts.

    pipeline/clean.py

  6. Remove duplicates

    Records match on DOI first, then on source id, then on normalised title. When two match, the fuller one stays: the longer abstract, then more authors, then a DOI, then a year. A report counts what was removed at each step.

    pipeline/dedupe.py

  7. Read the abstracts

    find_terms(), extract_dataset_size() and extract_headline_metric() pull the methods, the cohort size and the best score out of each abstract.

    pipeline/enrich.py

  8. Filter

    Year range, required abstract or DOI, words that must be mentioned, method families and a minimum dataset size.

    pipeline/filters.py

  9. Export

    JSONL to a file or to stdout, CSV, or an Excel workbook. The run summary goes to stderr, so it never mixes with the data.

    src/research_harvest/export.py

Engineering decisions

The parts that took the most thought

Narrow extraction, on purpose

In the words of the code: anything clever enough to catch every phrasing will also catch a p-value or a year and report it as a cohort size. So a number only counts as a cohort when it sits next to a cue, as in "n = 500", "1,234 patients" or "a dataset of 2000". Values from 2 to 50,000,000 are kept, and the largest wins.

"Published in 2021 with p < 0.001." gives no cohort at all.

Scores on one scale

The metric name has to come right before the number, optionally joined by "of" or "=". Percentages and rates above 1 are divided by 100, anything still outside 0 to 1 is dropped, and error measures such as RMSE and MAE never outrank a rate. "accuracy of 97.8%" becomes ACCURACY 0.978.

Methods matched longest first

114 method names in six families are tried longest first, so "Convolutional Neural Network" wins over "CNN". Word-boundary lookarounds stop short names from matching inside other words, so "GAN" never fires on "organ".

Paced to the limit, retried with care

Requests are paced to each API's published ceiling, which the code notes is also faster than sleeping a fixed amount. HTTP 429 and 5xx responses get three attempts with backoff. Any other 4xx fails at once, because a 400 will fail the same way forever.

Real output

Three rows from a live run

A live PubMed query for "blood brain barrier machine learning" on 24 September 2026, limited to three papers with abstracts. Where a cell says none, the extractor found nothing it was confident about.

TitleYearMethodsDataset sizeBest score
Blood Brain Barrier Permeability Prediction Using Machine Learning Techniques: An Update2019nonenonenone
Machine learning based dynamic consensus model for predicting blood-brain barrier permeability2023XGBoost; Random Forest8153ACCURACY 0.978
Exploring blood-brain barrier passage using atomic weighted vector and machine learning2024Gradient Boosting; SVMnoneACCURACY 0.98

What deduplication reports, from the sample run in the README

400 record(s) in, 361 kept, 39 duplicate(s) removed (doi: 37, source id: 0, title: 2).
12 record(s) removed by filters. 349 record(s) kept.

Tests and CI

How it is checked

115 tests run with pytest in under five seconds: pipeline 48, sources 26, cli 14, export 11, models 10 and harvest 6. None of them touch the network. The responses library answers every HTTP call from recorded fixtures, a real PubMed query and a set of OpenAlex works. Coverage is 95%.

The GitHub Actions workflow tests on Python 3.9, 3.10, 3.11, 3.12 and 3.13 and fails below 90% coverage, lints with ruff, and on main runs a live smoke test against PubMed that checks every row has a title and an abstract.

Run it yourself

git clone https://github.com/vipul21435/Web-Scraping-Tools-for-Research-Paper.git
cd Web-Scraping-Tools-for-Research-Paper
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"

research-harvest "blood brain barrier permeability" \
  --source pubmed openalex --year-from 2018 \
  --require-abstract --format xlsx -o out/bbb.xlsx

Step 3 of 3

The code

The package, the recorded fixtures, the test suite and the CI workflow are all in the repository. MIT licensed.

github.com/vipul21435/Web-Scraping-Tools-for-Research-Paper