NewAnswers MCP: one question, one checked answer for your agent.See how it works

Guides

How a research agent legally gets the full text of a paywalled paper

By , founder of HyperresearchUpdated

An AI agent can legally get the full text of a paper by asking free open-access services where a lawful copy is, then reading only copies whose licence allows it. Give it a DOI: Unpaywall or OpenAlex says whether an open copy exists, which version it is and under what licence; Europe PMC and PubMed Central hand back parsed XML for papers in their open-access subsets; CORE, arXiv and bioRxiv cover repository copies and preprints. None needs a subscription, and when no open copy exists the agent gets an abstract or nothing. Below are the request for each, the limits and licence terms as checked on 2026-11-03, and what Hyperresearch, the AI deep research API, does with them.

Endpoints, auth, limits and licences, checked on 2026-11-03

Service Endpoint for one paper Auth Rate limit Licence of what you get
Unpaywall api.unpaywall.org/v2/{doi}?email= Real email, checked “Please limit use to 100,000 calls per day” Locations only; each copy carries its own license
OpenAlex api.openalex.org/works/doi:{doi} Free key, or none 10 req/s and $0.10/day without a key; 30 req/s and $1/day with a free key; DOI lookups cost $0 Metadata CC0; each copy’s licence in license
OpenAlex content content.openalex.org/works/{id}.pdf or .grobid-xml Any key $10 per 1,000 downloads The work’s own licence
Europe PMC ebi.ac.uk/europepmc/webservices/rest/{PMCID}/fullTextXML None None published Open-access subset only, under the article’s CC or similar licence
PubMed Central BioC API ncbi.nlm.nih.gov/research/bionlp/RESTful/pmcoa.cgi/BioC_json/{PMCID}/unicode, or the pmc-oa-opendata S3 bucket None None published Open-access subset and author manuscripts, grouped by licence
CORE api.core.ac.uk/v3/search/works/?q=doi:"{doi}" None, or Bearer key 100 tokens/day, 10/min without a key; 1,000/day, 25/min with a free personal key No licence field; check Unpaywall or OpenAlex
Crossref api.crossref.org/works/{doi}?mailto= None; mailto moves you to the polite pool 5 req/s, 1 at once; 10 req/s, 3 at once with mailto Bibliographic metadata is “Facts, not subject to copyright”; abstracts stay under publisher or author copyright
Retraction Watch Crossref update-to, or the CSV at gitlab.com/crossref/retraction-watch-data None As Crossref CC0
arXiv export.arxiv.org/api/query?id_list={id} None One request every 3 seconds, one connection at a time Metadata CC0; e-prints under the licence the author chose
bioRxiv api.biorxiv.org/details/biorxiv/{doi} None None published The licence the author chose, in license, including no reuse
Semantic Scholar api.semanticscholar.org/graph/v1/paper/DOI:{doi}?fields=openAccessPdf x-api-key header, or none 1,000 req/s shared by all keyless users; 1 req/s on a new key Its API licence agreement; third-party papers keep their own terms

Unpaywall finds the open copy and names its version

Terminal window
curl -sS 'https://api.unpaywall.org/v2/10.7717/peerj.4375?email=you@your-domain.org'

Read is_oa first, then oa_status, which is one of gold, hybrid, bronze, green or closed. Then read best_oa_location. Its useful fields are url_for_pdf, url_for_landing_page, url, host_type, license and version. The docs define url as “the url_for_pdf if there is one; otherwise landing page URL”. oa_locations lists every copy Unpaywall knows about. You will need it, because the best copy is not always one you can fetch.

For this DOI the call returns is_oa: true, oa_status: "gold", a publishedVersion under cc-by, and ten locations. url_for_pdf is null. That is normal. Fall back to url, then try the other locations.

There is no key, but email is required and checked: email=test@example.com returns HTTP 422 with “Please use your own email address in API calls.” The only limit is a request: “Please limit use to 100,000 calls per day.” Two location fields, evidence and updated, are deprecated and always return the string deprecated. Do not fetch from oa_locations_embargoed. The docs describe those copies as “not yet available”.

Unpaywall now runs on OpenAlex’s data. OpenAlex’s help center says it “moved Unpaywall onto the OpenAlex database” in late 2025, and that an Unpaywall record’s is_oa, oa_status and PDF link are the same values as the OpenAlex work’s. Two consequences for an agent. Unpaywall covers Crossref DOIs only, so a DataCite DOI returns 404; ask OpenAlex instead. And a preprint DOI for a paper later published in a journal answers with the journal article’s open-access status.

Europe PMC is the one that returns structured text

Most of the others point you at a PDF. Europe PMC returns JATS XML with real sections and paragraphs, which an agent reads far better than a two-column PDF.

Terminal window
curl -sS 'https://www.ebi.ac.uk/europepmc/webservices/rest/search?query=DOI:%2210.7717/peerj.4375%22&format=json&resultType=core'
curl -sS 'https://www.ebi.ac.uk/europepmc/webservices/rest/PMC5815332/fullTextXML'

Ask for resultType=core. The default lite result has the PMCID and the open-access flag but leaves out the licence and the full-text links. In the result, check isOpenAccess and inEPMC, and take the PMCID from pmcid. Do not rely on hasFullTextXML: it came back null on this record, which does have XML. In fullTextUrlList.fullTextUrl[], keep only entries with availabilityCode OA. The same record also lists a subscription link.

The second call returns application/xml, about 128 KB, with an <article> root and 88 paragraphs in the body. The same call for an article outside the open-access subset returned HTTP 500, so ask only when isOpenAccess is Y.

Open access does not mean anything goes. Europe PMC says articles in the subset “are still protected by copyright” and are made available under a Creative Commons licence “or similar”. Some of those licences are non-commercial. Read license before you reuse text. For bulk work the rule is explicit: “It is not permissible to use any kind of automated process to bulk download other content from Europe PMC.” The open-access subset has its own bulk download. Nothing else does.

There is no API key and no published rate limit. The only figure is a 2020 post by the Europe PMC team on their own forum: “Currently we apply some throttling of 10 requests per second and 500 requests per minute.” That is six years old. Stay well under it.

PubMed Central retired its OA Web Service

Many scripts still call PMC’s oa.fcgi to find an article’s package. It is gone. PMC’s page now says “the PMC OA Web Service is no longer available,” and its FTP page says “All legacy PMC Article Dataset files on the FTP and Cloud Services were removed the week of August 24, 2026.” The old URL returned 404 on 2026-11-03.

For one article, use the BioC API:

Terminal window
curl -sS 'https://www.ncbi.nlm.nih.gov/research/bionlp/RESTful/pmcoa.cgi/BioC_json/PMC5815332/unicode'

It covers the PMC open-access subset and the author manuscript collection, in BioC XML or JSON, by PMID or PMCID. The call above returned the article as passages, with "license": "CC BY" in the document’s infons.

For many articles, use the PMC Cloud Service. The bucket is pmc-oa-opendata on AWS, and PMC says “No account is necessary.” Each article version has its own folder, such as PMC5815332.1/, holding the XML, plain text, PDF, figures and a JSON metadata file that carries the licence.

PMC sorts the subset into three licence groups: commercial use allowed (CC0, CC BY, CC BY-SA, CC BY-ND), non-commercial use only (the CC BY-NC licences), and other, which includes custom licences and none. It also names the only routes a machine may use: the Cloud Service, OAI-PMH, E-Utilities and the BioC API. “Systematic retrieval (or bulk retrieval) of articles through any other automated process is prohibited.”

CORE casts the widest net, and holds back text without a key

Terminal window
curl -sS 'https://api.core.ac.uk/v3/search/works/?q=doi%3A%2210.7717%2Fpeerj.4375%22'

Keep the trailing slash after works. Without it you get a 301. There is no DOI endpoint, so you search on the doi field.

You do not need to register. The docs say “Access to the CORE API is free and requires no authentication”, and the call above works with no key. A key from CORE’s registration form goes in an Authorization: Bearer header. A bad key is worse than none: it returns 401.

The docs’ footnote is the part that matters: “Full-text is not available for unauthenticated API users.” Without a key, fullText comes back as the string "Not available for public API users." An agent that only checks whether the field is empty will store that sentence as the paper. What you do get without a key is downloadUrl (here a CORE-hosted PDF) and sourceFulltextUrls, which for this paper lists the CORE copy and the publisher’s PDF.

Limits are counted in tokens. A simple query costs one, and harder queries cost three to five. Without a key you get “100 tokens per day, maximum 10 per minute”. A registered personal key gets “1,000 tokens per day, maximum 25 per minute”. Registered academic users at a non-supporting institution get “5,000 tokens per day, maximum 10 per minute”. The x-ratelimit-retry-after header is a timestamp, not a number of seconds. Our first keyless call on 2026-11-03 got a 429, so expect one on a shared address and honour that timestamp.

The work record has no version, open-access status or licence field. If you need them, look up the DOI in Unpaywall or OpenAlex.

OpenAlex charges in dollars, and DOI lookups are free

A free API key, sent as api_key or a Bearer header, is how you identify yourself. An agent can ask for one for its user with POST https://api.openalex.org/api-keys and the user’s email; the key works once the user clicks the emailed link.

Terminal window
curl -sS 'https://api.openalex.org/works/doi:10.7717/peerj.4375?api_key=YOUR_KEY'

Read open_access and best_oa_location. For this DOI, open_access says is_oa: true, oa_status: "gold" and any_repository_has_fulltext: true. best_oa_location has version: "publishedVersion", license: "cc-by", a null pdf_url, and is_accepted and is_published flags. The record also carries is_retracted and indexed_in. That retraction flag is a reason to call OpenAlex even when you already have the text.

Limits are a daily budget in dollars: $0.10 a day without a key, $1 a day with a free key, reset at midnight UTC. There is also a per-second limit: 10 requests a second without a key, 30 with a free key or prepaid usage, and 50 on an annual plan. A per-second 429 costs nothing; wait and retry. A single lookup by ID or DOI costs nothing. List and filter calls cost $0.10 per 1,000 and search costs $1 per 1,000. Their own table lists retrieving 1,000,000 works by DOI as free. The response headers confirm it: X-RateLimit-Cost-USD: 0 on a DOI lookup, 0.0001 on a filter call.

OpenAlex also serves its own cached copies. Its content API holds a PDF and a GROBID TEI XML parse for more than 60 million open-access works, at content.openalex.org/works/{work_id}.pdf or .grobid-xml, for $10 per 1,000 downloads; a HEAD request is free. The work record says what exists: has_content lists the formats and content_urls the links. On this record both formats were false, although the paper is gold open access, so do not treat a missing cached copy as a closed paper.

Do not mix up has_fulltext and open_access.any_repository_has_fulltext either. On this record the first is false and the second is true. They answer different questions.

Terminal window
curl -sS 'https://api.crossref.org/works/10.7717/peerj.4375?mailto=you@your-domain.org'

Crossref does not find open copies, but publishers deposit two useful fields there. license[] holds licence URLs with start dates. link[] holds full-text URLs, and entries marked intended-application: text-mining are where the publisher serves files for machines. A link is not a licence. Check license before you use what it points to.

Add mailto and you move to the polite pool. The response headers show 10 requests per second with 3 at once, against 5 per second with 1 at once without it.

Crossref also carries the Retraction Watch database, which it acquired in 2023 and publishes under CC0. In the REST API, retractions sit in a work’s update-to field, and each entry’s source is publisher or retraction-watch. https://api.crossref.org/works?filter=update-type:retraction lists retraction notices; it counted 81,306 on 2026-11-03. For a local copy, the same data is a CSV at gitlab.com/crossref/retraction-watch-data, “updated once per working day”; the last commit was dated 2026-11-02. Match on OriginalPaperDOI, and read RetractionNature, because the file also holds corrections, expressions of concern and reinstatements.

arXiv and bioRxiv serve preprints, under the author’s licence

arXiv’s API returns metadata and a link to the PDF:

Terminal window
curl -sS 'https://export.arxiv.org/api/query?id_list=1706.03762'

Its terms ask you to “make no more than one request every three seconds” and to “limit requests to a single connection at a time”, across all your machines. Metadata is CC0. The e-prints are not: most are under arXiv’s non-exclusive licence to distribute, and the terms say not to store and serve them from your own servers unless the paper’s licence allows it. Read the paper for your own research use and link to the abstract page.

bioRxiv’s API returns metadata, the licence and a JATS XML link:

Terminal window
curl -sS 'https://api.biorxiv.org/details/biorxiv/10.1101/2020.03.20.000141'

This record returned license: "cc_by", a jatsxml URL and the DOI of the journal version. Authors choose from CC BY, CC BY-NC, CC BY-ND, CC BY-NC-ND, CC0, “or no reuse”, and “By posting on bioRxiv, authors explicitly consent to text mining of their work.” Text mining is not redistribution. bioRxiv says tools built on its full text “must therefore link back to the text hosted at bioRxiv rather than re-host content.” Bulk full text is in a requester-pays S3 bucket, s3://biorxiv-src-monthly.

Terminal window
curl -sS 'https://api.semanticscholar.org/graph/v1/paper/DOI:10.7717/peerj.4375?fields=title,isOpenAccess,openAccessPdf' \
-H 'x-api-key: YOUR_KEY'

openAccessPdf is null or an object with a url and a status such as GOLD. Without a key you share “1000 requests per second shared among all unauthenticated users”, and that pool is often full: three keyless calls from our network on 2026-11-03 all returned 429. A new key gets 1 request per second. Use is under Semantic Scholar’s API licence agreement, which requires an attribution to Semantic Scholar and says open-access works may carry their own third-party terms.

Field names do not line up across services

What you want Unpaywall OpenAlex Europe PMC CORE
Is it open is_oa open_access.is_oa isOpenAccess not given
Open-access status oa_status open_access.oa_status not given not given
Best PDF best_oa_location.url_for_pdf best_oa_location.pdf_url fullTextUrlList entries downloadUrl
Landing page best_oa_location.url_for_landing_page best_oa_location.landing_page_url not applicable not given
Every copy oa_locations[] locations[] fullTextUrlList[] sourceFulltextUrls[]
Cached full text not given has_content, content_urls fullTextXML endpoint fullText (key only)
Version version version, is_accepted, is_published not given not given
Licence license license, license_id license not given
Retraction flag not given is_retracted not given not given

Record which version you got

Unpaywall uses the DRIVER Guidelines v2.0. submittedVersion “is not yet peer-reviewed”. acceptedVersion “is peer-reviewed, but lacks publisher-specific formatting”. publishedVersion “is the version of record”. They are not the same text. Page numbers differ, and wording and figures can change between the accepted manuscript and the final proof.

Store four things with the text: which service answered, the URL the text came from, the version, and the licence. Then show them. A reader who sees a journal citation should know when the text was an accepted manuscript or a preprint. A quote from a submitted version may not appear in the published paper.

What not to do

Sci-Hub and its mirrors. Courts in several countries have issued judgments and blocking orders against it. A commercial pipeline cannot use it.

Someone else’s login. Do not use a customer’s or an employee’s institutional login, do not share a proxy session, and do not let an agent sign in anywhere. An institutional licence lets a person read. It does not let a service copy.

Pages behind a paywall. A page that needs payment or a session is not a public page. When a fetch hits one, record it and ask a person.

Crawling the websites. Europe PMC, PMC and bioRxiv all name their machine routes and bar bulk crawling of their sites. Use the API or the bulk bucket.

How Hyperresearch does it

Hyperresearch reads each source through a fetch ladder: the workspace vault, an arXiv route for arXiv papers, a direct fetch, open access, licensed articles (built but switched off), browser rendering, an archived copy from the Wayback Machine for report and government pages, then a person. How a run works covers the ladder. This section covers the scholarly steps.

An arXiv URL is read through arXiv’s own API for the metadata and its HTML or PDF for the body. Every request from the platform shares one limiter set to one request every 3 seconds, the ceiling arXiv’s terms name.

The open-access rung runs when the direct fetch was blocked, hit a login wall, failed or came back thin, and the run has an identifier for the paper. For a biomedical article with a DOI, PubMed ID or PMCID, it first makes one Europe PMC lookup and, if the article is in the open-access subset, one full-text fetch. For any DOI, it asks Unpaywall, Europe PMC and CORE where open copies are, then tries them in this order: Unpaywall’s PDF, Unpaywall’s landing page, Europe PMC, then a CORE repository copy. It fetches at most three copies for one source, and it doesn’t look an article up on Europe PMC twice. The recovered text must pass the same length and junk checks as any other page. If no copy passes, the ladder moves on.

Every URL a resolver hands back is checked before anything fetches it, because it arrives inside someone else’s JSON. The check covers the scheme, credentials in the URL, and whether the host resolves to a public address, and it runs again on every redirect.

You can call the same lookup yourself. It needs the vault:read scope and returns every location it found, each with its version, licence and whether it passed the URL check.

Terminal window
curl -sS https://api.hyperresearch.ai/v1/scholar/oa \
-H 'Authorization: Bearer <YOUR_API_KEY>' \
-H 'Content-Type: application/json' \
-d '{"doi": "10.7717/peerj.4375"}'

A recovered source is labelled one of two ways. substituted means the original page loaded but was thin, so only the body came from the open copy. rescued means nothing came from the original page, so the title and authors are the open copy’s too. Each source row in the run result has an oa block with the kind, the resolver, the URL the text came from, the version and body_is_not_from_source: true. A rescued source also has nothing_from_source: true. When the resolver reported a licence, the note keeps it. The source is kept in the vault with the rung that read it.

The standalone citation verification API reads cited pages through a shorter ladder. Its open-access step uses only the Europe PMC path for biomedical articles. Retracted papers are covered on retraction-aware research.

We have not measured how often each rung recovers a paper, so this page gives no recovery rate.

The rules are the ones the open-source harness keeps, described in Claude Code as a research harness: public pages, logged out, no credentials, no CAPTCHA solving. When a paper needs a person, the run gives you a task. You can upload a copy you hold.