Skip to content

← All projects In preparation

A harmonised global dataset of preprints, 1990–2025

One clean record for every preprint, harmonised across Crossref, DataCite and OpenAlex.

Bar chart of the top 15 preprint sources from 1990 to 2025, led by arXiv, SSRN and HAL — Top 15 preprint sources, 1990–2025, in the Preprint Servers Explorer. Preprint Servers Explorer, data collected 16 May 2026.

Dashboard overview: 109 sources, 7,347,311 preprints in range, 24% with published links — Overview: 109 sources and 7.3 million preprints in the 1990–2025 range; 24% link to a published version. Preprint Servers Explorer.

Problem

Preprint metadata are split across Crossref, DataCite and OpenAlex, with inconsistent server names and duplicate records.

Approach

Built harvesters for each source, wrote rules to identify preprint servers, linked versions and removed duplicates.

Result

7,905,699 harvested records reduced to 7,332,492 parent records from 109 servers; dataset paper in preparation for Quantitative Science Studies.

My role

Lead author and data engineer at ScholCommLab, Simon Fraser University.

Outputs

Harmonised dataset; harvesters; manuscript in preparation

Methods and tools

OpenAlex
Crossref
DataCite
Python
DOI normalization
SQL

Skills

Data engineering
entity resolution
metadata quality
reproducibility