Download

Utilities for downloading database source files and detecting releases.

Download and release detection for biological database sources.

The datasource-agnostic dispatch logic lives in mapkgsutils.download; this module just binds it to pysec2pri’s datasource registries (see pysec2pri.downloads).

exception CloudflareBlockedError(url: str)[source]

Bases: Exception

Raised when a download is blocked by Cloudflare bot protection.

Parameters:: url – The URL that was blocked.

Store url and build a message explaining the manual-download workaround.

class ReleaseInfo(datasource: str, version: str | None, release_date: datetime | None, is_new: bool, files: dict[str, str])[source]

Bases: object

Information about a datasource release.

check_chebi_release() → ReleaseInfo[source]

Check for the latest ChEBI release.

Returns:: ReleaseInfo with the latest ChEBI release details.

check_ensembl_release() → ReleaseInfo[source]

Check for the latest Ensembl release.

The release number is global across every species (Ensembl cuts all species’ core databases under the same release number), so this always checks against the default species (human).

Returns:: ReleaseInfo with the latest Ensembl release details.

check_hgnc_release() → ReleaseInfo[source]

Check for the latest HGNC release from the quarterly archive.

Queries the Google Cloud Storage API to list files in the HGNC quarterly archive bucket and finds the latest release.

Returns:: ReleaseInfo with the latest HGNC release details.

check_hmdb_release() → ReleaseInfo[source]

Check for the latest HMDB release by downloading and checking XML.

Returns:: ReleaseInfo with the latest HMDB release details.

check_ncbi_release() → ReleaseInfo[source]

Check for the latest NCBI Gene release.

Returns:: ReleaseInfo with the latest NCBI release details.

check_release(datasource: str, current_version: str | None = None, current_date: datetime | None = None) → ReleaseInfo[source]

Check if a new release is available for a datasource.

Parameters:

datasource – Name of the datasource.
current_version – Current version string to compare against.
current_date – Current release date to compare against.

Returns:

ReleaseInfo with is_new indicating if update is available.

check_uniprot_release() → ReleaseInfo[source]

Check for the latest UniProt release.

Returns:: ReleaseInfo with the latest UniProt release details.

download_datasource(datasource: str, output_dir: Path, decompress: bool = True, version: str | None = None, keys: list[str] | None = None, **kwargs: Any) → dict[str, Path][source]

Download all files for a datasource.

For datasources with dynamic URLs (like HGNC quarterly archive), this function first checks for the latest release and uses those URLs. If a version is specified, it downloads that specific version.

Parameters:

datasource – Name of the datasource.
output_dir – Directory to save files.
decompress – Whether to decompress .gz files.
version – Specific version to download. Format depends on datasource.
keys – Optional list of file-key names to download. When given, only URLs whose key is in this list are fetched. Defaults to all keys.
**kwargs – Datasource-specific knobs (subset for ChEBI, species for Ensembl); see get_download_urls().

Returns:

Dictionary mapping file keys to downloaded paths.

download_datasource_with_release(datasource: str, output_dir: Path, decompress: bool = True, version: str | None = None, keys: list[str] | None = None, **kwargs: Any) → tuple[dict[str, Path], str | None, datetime | None][source]

Download all files for a datasource; report its resolved version and release date.

Parameters:

datasource – Name of the datasource.
output_dir – Directory to save files.
decompress – Whether to decompress .gz files.
version – Specific version to download, or None for latest.
keys – Optional list of file-key names to download.
**kwargs – Datasource-specific knobs (subset for ChEBI, species for Ensembl); see get_download_urls().

Returns:

Tuple of (file-key -> downloaded path mapping, resolved version, release date). None for a datasource with no release checker.

download_file(url: str, output_path: Path, decompress_gz: bool = True, timeout: float | None = None, show_progress: bool = True, description: str | None = None) → Path[source]

Download a file from URL to the specified path.

Parameters:

url – URL to download from.
output_path – Where to save the file.
decompress_gz – Whether to decompress .gz files automatically.
timeout – Request timeout in seconds.
show_progress – Whether to show a progress bar.
description – Description for the progress bar.

Returns:

Path to the downloaded (and optionally decompressed) file.

get_download_urls(datasource: str, version: str | None = None, **kwargs: Any) → dict[str, str][source]

Get download URLs for a datasource.

Parameters:

datasource – Name of the datasource.
version – Specific version to get URLs for.
**kwargs – Datasource-specific knobs (e.g. subset for ChEBI, species for Ensembl/NCBI) – which ones apply depends on the datasource’s config (see DatasourceConfig.subset/ .species); irrelevant kwargs are simply ignored.

Returns:

Dictionary mapping file keys to URLs.

get_file_last_modified(url: str, timeout: float = 30.0) → datetime | None[source]

Get the Last-Modified date from a URL via HEAD request.

Parameters:

url – URL to check.
timeout – Request timeout in seconds.

Returns:

The Last-Modified datetime or None if unavailable.

get_latest_release_info(datasource: str) → ReleaseInfo[source]

Get release information for a datasource.

Parameters:: datasource – Name of the datasource (chebi, hmdb, hgnc, ncbi, uniprot).
Returns:: ReleaseInfo with the latest release details.
Raises:: ValueError – If the datasource is not supported.

list_versions(datasource: str) → Any[source]

List all available archive versions for a datasource.

Delegates to the datasource’s downloader class list_versions() method, which contains all source-specific retrieval logic.

For datasources that publish versioned archives (ChEBI, HGNC, UniProt), returns all available version strings sorted in ascending order.

NCBI and HMDB do not maintain versioned archives; calling this function for those datasources raises ValueError.

Parameters:

datasource – Datasource name ("chebi", "hgnc", or "uniprot").

Returns:

chebi: integer release numbers, e.g. ["200", "201", ..., "245"]
hgnc: ISO dates, e.g. ["2023-01-01", ..., "2026-04-07"]
uniprot: release identifiers, e.g. ["2024_01", "2024_02", ...]

Return type:

Sorted list of version strings. Format depends on the datasource

Raises:

ValueError – If the datasource is unknown or has no versioned archive.

resolve_release_date(datasource: str, version: str | None = None, **kwargs: Any) → datetime | None[source]

Resolve the upstream release date for a datasource/version.

This is the date used for the SSSOM mapping_date of generated mapping sets. It does not download the data files (it may issue a lightweight HEAD request to read a Last-Modified header). Prefer download_datasource_with_release() when you are downloading anyway, to avoid an extra round-trip.

Parameters:

datasource – Name of the datasource.
version – Specific version, when applicable.
**kwargs – Datasource-specific knobs (subset for ChEBI, species for Ensembl/NCBI); see get_download_urls().

Returns:

The release date, or None when it cannot be determined.