- 1 Projects
- 2 Details
- 3 Code on GitHub
Open source | Python | CLI and library
research-harvest
A command-line tool and Python library for literature screening. It searches PubMed and OpenAlex, merges and deduplicates the results, reads every abstract for the methods used, the cohort size and the best score reported, and writes it all to JSONL, CSV or Excel.
- Python 3.9+
- requests
- pytest
- responses
- ruff
- openpyxl
- GitHub Actions
- 115
- tests, all running offline
- 95%
- test coverage, and CI fails below 90%
- 5
- Python versions in CI, 3.9 to 3.13
- 114
- method names it recognises, in six families
Overview
What it does
Screening papers by hand means opening every abstract to find out what was done, on how much data, and how well it worked. research-harvest does that first pass. Give it a query and it returns one row per paper with the models the abstract mentions, the size of the dataset or cohort, and the headline score, ready to sort and filter.
It talks to both catalogues over their official REST APIs and paces itself to each one's rate limit: 3 requests a second for PubMed, or 9 with an NCBI key, and 5 for OpenAlex. If one source fails, the error is recorded and the run carries on with the other.
- Type
- CLI and Python library
- Language
- Python 3.9 or newer
- Dependencies
- requests, plus openpyxl for Excel output
- Sources
- PubMed E-utilities and OpenAlex
- Output
- JSONL, CSV or Excel
- Licence
- MIT
How it works
From a search query to a spreadsheet
Read the command
main()validates the flags, exits with status 2 on bad input, and turns the filters into aFilterSpec.src/research_harvest/cli.py
Run every source
harvest()runs each source in turn. ASourceErroris recorded in the result and the remaining sources still run.src/research_harvest/harvest.py
Search PubMed
esearch returns the matching ids as JSON, and efetch returns the records as XML, 200 per call.
sources/pubmed.py
Search OpenAlex
Results come back page by page with a cursor, and each abstract is rebuilt from OpenAlex's inverted index. Both sources share one HTTP client with a rate limiter and produce the same
Articlemodel.sources/openalex.py, sources/base.py, models.py
Clean the text
Unescape HTML entities, strip tags and cut the copyright notices that publishers append to abstracts.
pipeline/clean.py
Remove duplicates
Records match on DOI first, then on source id, then on normalised title. When two match, the fuller one stays: the longer abstract, then more authors, then a DOI, then a year. A report counts what was removed at each step.
pipeline/dedupe.py
Read the abstracts
find_terms(),extract_dataset_size()andextract_headline_metric()pull the methods, the cohort size and the best score out of each abstract.pipeline/enrich.py
Filter
Year range, required abstract or DOI, words that must be mentioned, method families and a minimum dataset size.
pipeline/filters.py
Export
JSONL to a file or to stdout, CSV, or an Excel workbook. The run summary goes to stderr, so it never mixes with the data.
src/research_harvest/export.py
Engineering decisions
The parts that took the most thought
Narrow extraction, on purpose
In the words of the code: anything clever enough to catch every phrasing will also catch a p-value or a year and report it as a cohort size. So a number only counts as a cohort when it sits next to a cue, as in "n = 500", "1,234 patients" or "a dataset of 2000". Values from 2 to 50,000,000 are kept, and the largest wins.
"Published in 2021 with p < 0.001." gives no cohort at all.
Scores on one scale
The metric name has to come right before the number, optionally joined by "of" or "=". Percentages and rates above 1 are divided by 100, anything still outside 0 to 1 is dropped, and error measures such as RMSE and MAE never outrank a rate. "accuracy of 97.8%" becomes ACCURACY 0.978.
Methods matched longest first
114 method names in six families are tried longest first, so "Convolutional Neural Network" wins over "CNN". Word-boundary lookarounds stop short names from matching inside other words, so "GAN" never fires on "organ".
Paced to the limit, retried with care
Requests are paced to each API's published ceiling, which the code notes is also faster than sleeping a fixed amount. HTTP 429 and 5xx responses get three attempts with backoff. Any other 4xx fails at once, because a 400 will fail the same way forever.
Real output
Three rows from a live run
A live PubMed query for "blood brain barrier machine learning" on 24 September 2026, limited to three papers with abstracts. Where a cell says none, the extractor found nothing it was confident about.
| Title | Year | Methods | Dataset size | Best score |
|---|---|---|---|---|
| Blood Brain Barrier Permeability Prediction Using Machine Learning Techniques: An Update | 2019 | none | none | none |
| Machine learning based dynamic consensus model for predicting blood-brain barrier permeability | 2023 | XGBoost; Random Forest | 8153 | ACCURACY 0.978 |
| Exploring blood-brain barrier passage using atomic weighted vector and machine learning | 2024 | Gradient Boosting; SVM | none | ACCURACY 0.98 |
What deduplication reports, from the sample run in the README
400 record(s) in, 361 kept, 39 duplicate(s) removed (doi: 37, source id: 0, title: 2).
12 record(s) removed by filters. 349 record(s) kept.
Tests and CI
How it is checked
115 tests run with pytest in under five seconds: pipeline 48, sources 26, cli 14, export 11, models 10 and harvest 6. None of them touch the network. The responses library answers every HTTP call from recorded fixtures, a real PubMed query and a set of OpenAlex works. Coverage is 95%.
The GitHub Actions workflow tests on Python 3.9, 3.10, 3.11, 3.12 and 3.13 and fails below 90% coverage, lints with ruff, and on main runs a live smoke test against PubMed that checks every row has a title and an abstract.
Run it yourself
git clone https://github.com/vipul21435/Web-Scraping-Tools-for-Research-Paper.git
cd Web-Scraping-Tools-for-Research-Paper
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
research-harvest "blood brain barrier permeability" \
--source pubmed openalex --year-from 2018 \
--require-abstract --format xlsx -o out/bbb.xlsx
Step 3 of 3
The code
The package, the recorded fixtures, the test suite and the CI workflow are all in the repository. MIT licensed.
github.com/vipul21435/Web-Scraping-Tools-for-Research-Paper