{ "cells": [ { "cell_type": "markdown", "id": "21a5be08", "metadata": {}, "source": [ "# Example: GTEx\n", "\n", "GTEx's gene median TPM table names every gene twice, in two columns:\n", "\n", "| column | holds | example |\n", "| --- | --- | --- |\n", "| `Name` | Ensembl gene ID, versioned | `ENSG00000223972.6` |\n", "| `Description` | gene symbol | `DDX11L1` |\n", "\n", "Any identifier or gene name could go out of date.\n", "\n", "Since downstream analyses or integration with\n", "your data could depend on one of these columns, `pysec2pri` can be used to\n", "update these IDs and labels or spot issues.\n", "\n", "The last part of the notebook shows how to use the sec2pri mapping sets\n", "with additional context (HGNC's HGNC symbol->Ensembl ID mappings) to solve\n", "the remaining ambiguity issues." ] }, { "cell_type": "code", "execution_count": null, "id": "eaef4375", "metadata": { "execution": { "iopub.execute_input": "2026-07-15T11:47:33.749180Z", "iopub.status.busy": "2026-07-15T11:47:33.748756Z", "iopub.status.idle": "2026-07-15T11:47:35.697333Z", "shell.execute_reply": "2026-07-15T11:47:35.695812Z" } }, "outputs": [ { "ename": "", "evalue": "", "output_type": "error", "traceback": [ "\u001b[1;31mFailed to start the Kernel. \n", "\u001b[1;31m[IPKernelApp] WARNING | Kernel is running over TCP without encryption. All communication (including code and outputs) is sent in plain text and is susceptible to eavesdropping. Use IPC transport or launch with kernel manager-provisioned CurveZMQ keys to enable transport encryption. \n", "\u001b[1;31mView Jupyter log for further details." ] } ], "source": [ "import pandas as pd\n", "\n", "URL = (\n", " \"https://storage.googleapis.com/adult-gtex/bulk-gex/v11/rna-seq/\"\n", " \"GTEx_Analysis_2025-08-22_v11_RNASeQCv2.4.3_gene_median_tpm.gct.gz\"\n", ")\n", "df = pd.read_csv(URL, sep=\"\\t\", skiprows=2, usecols=[\"Name\", \"Description\"], dtype=str)\n", "\n", "# GTEx versions its IDs; nothing else here does. Drop it.\n", "df[\"ensembl\"] = df[\"Name\"].str.split(\".\").str[0]\n", "# Ensembl's own mapping set writes IDs as CURIEs; HGNC's crosswalk does not.\n", "df[\"ensembl_curie\"] = \"ENSEMBL:\" + df[\"ensembl\"]\n", "df.head(3)" ] }, { "cell_type": "markdown", "id": "3f5d085e", "metadata": {}, "source": [ "## Symbols\n", "\n", "`update_labels` adds `Description_current`: the symbol HGNC uses today." ] }, { "cell_type": "code", "execution_count": null, "id": "3be46a36", "metadata": { "execution": { "iopub.execute_input": "2026-07-15T11:47:35.701359Z", "iopub.status.busy": "2026-07-15T11:47:35.700970Z", "iopub.status.idle": "2026-07-15T11:47:55.422810Z", "shell.execute_reply": "2026-07-15T11:47:55.421357Z" } }, "outputs": [], "source": [ "from pysec2pri import generate_labels, update_labels\n", "\n", "hgnc = generate_labels(\"hgnc\", show_progress=False)\n", "plain = update_labels(df, hgnc, at=\"Description\")\n", "\n", "current = plain[\"Description_current\"]\n", "renamed = plain[(current != \"\") & (current != plain[\"Description\"])]\n", "renamed[[\"Description\", \"Description_current\"]].head()" ] }, { "cell_type": "markdown", "id": "9c95d5fb", "metadata": {}, "source": [ "Some rows come back empty. Those symbols are ambiguous: the symbol is both a retired name of one gene and the current name of another, so there is no safe answer and pysec2pri does not guess." ] }, { "cell_type": "code", "execution_count": null, "id": "87b0c45c", "metadata": { "execution": { "iopub.execute_input": "2026-07-15T11:47:55.426854Z", "iopub.status.busy": "2026-07-15T11:47:55.426413Z", "iopub.status.idle": "2026-07-15T11:47:55.453458Z", "shell.execute_reply": "2026-07-15T11:47:55.451011Z" } }, "outputs": [], "source": [ "ambiguous = plain[plain[\"Description_current\"] == \"\"]\n", "print(f\"renamed: {len(renamed)}\")\n", "print(f\"ambiguous: {len(ambiguous)}\")\n", "ambiguous[[\"Name\", \"Description\"]].head()" ] }, { "cell_type": "markdown", "id": "8d97aa17", "metadata": {}, "source": [ "## Symbols, settled by the Ensembl column\n", "\n", "The issues with HGNC symbols can be repaired by adding a custom cross-reference.\n", "\n", "Here `xref_source(\"hgnc_custom\")` (a link to the custom download mapping HGNC\n", "symbols to Ensembl IDs at https://www.genenames.org/download/custom/) is used,\n", "but any table works as a hint, as long as its `object_id` matches the IDs of\n", "the mapping set you are resolving against.\n", "\n", "See more at the [Update IDs and labels](update_ids.rst) page.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "b01cf096", "metadata": { "execution": { "iopub.execute_input": "2026-07-15T11:47:55.459495Z", "iopub.status.busy": "2026-07-15T11:47:55.459103Z", "iopub.status.idle": "2026-07-15T11:48:03.987550Z", "shell.execute_reply": "2026-07-15T11:48:03.985949Z" } }, "outputs": [], "source": [ "from mapkgsutils.context import download_xref_source\n", "from mapkgsutils.parsers.config import get_datasource_config\n", "\n", "src = get_datasource_config(\"hgnc\", config_package=\"pysec2pri.config\").xref_source(\"hgnc_custom\")\n", "print(src.note)\n", "\n", "ens_to_hgnc = download_xref_source(src, src.subject_id_cols[\"ensembl\"], show_progress=False)\n", "len(ens_to_hgnc.records)" ] }, { "cell_type": "code", "execution_count": null, "id": "4d4286c8", "metadata": { "execution": { "iopub.execute_input": "2026-07-15T11:48:03.991419Z", "iopub.status.busy": "2026-07-15T11:48:03.990993Z", "iopub.status.idle": "2026-07-15T11:48:20.013999Z", "shell.execute_reply": "2026-07-15T11:48:20.012587Z" } }, "outputs": [], "source": [ "hinted = update_labels(\n", " df,\n", " hgnc,\n", " at=\"Description\",\n", " xref=\"ensembl\",\n", " xref_mapping=ens_to_hgnc,\n", " report_path=\"symbol_decisions.tsv\",\n", ")\n", "\n", "settled = ambiguous.index.difference(hinted[hinted[\"Description_current\"] == \"\"].index)\n", "print(f\"ambiguous before: {len(ambiguous)}\")\n", "print(f\"settled by the hint: {len(settled)}\")\n", "hinted.loc[settled, [\"Description\", \"Description_current\", \"Description_current_id\"]].head()" ] }, { "cell_type": "markdown", "id": "03feef84", "metadata": {}, "source": [ "The hint is only consulted for ambiguous rows, and every decision is written down." ] }, { "cell_type": "code", "execution_count": null, "id": "44aaeadd", "metadata": { "execution": { "iopub.execute_input": "2026-07-15T11:48:20.021469Z", "iopub.status.busy": "2026-07-15T11:48:20.020911Z", "iopub.status.idle": "2026-07-15T11:48:20.045637Z", "shell.execute_reply": "2026-07-15T11:48:20.043915Z" } }, "outputs": [], "source": [ "report = pd.read_csv(\"symbol_decisions.tsv\", sep=\"\\t\")\n", "report.head(3)" ] }, { "cell_type": "markdown", "id": "b5dd4959", "metadata": {}, "source": [ "## The other direction\n", "\n", "Now resolve the Ensembl IDs against Ensembl, and hint with the symbol column." ] }, { "cell_type": "code", "execution_count": null, "id": "02642e2c", "metadata": { "execution": { "iopub.execute_input": "2026-07-15T11:48:20.050535Z", "iopub.status.busy": "2026-07-15T11:48:20.050009Z", "iopub.status.idle": "2026-07-15T11:48:57.704824Z", "shell.execute_reply": "2026-07-15T11:48:57.703438Z" } }, "outputs": [], "source": [ "from pysec2pri import generate_ids, update_ids\n", "\n", "ensembl = generate_ids(\"ensembl\", version=\"115\", species=\"9606\", show_progress=False)\n", "plain_ids = update_ids(df, ensembl, at=\"ensembl_curie\")\n", "\n", "changed = plain_ids[plain_ids[\"ensembl_curie_primary\"] != plain_ids[\"ensembl_curie\"]]\n", "stale = plain_ids[plain_ids[\"ensembl_curie_primary\"] == \"\"]\n", "print(f\"retired or withdrawn: {len(changed)}\")\n", "print(f\"ambiguous: {len(stale)}\")\n", "changed[[\"ensembl_curie\", \"ensembl_curie_primary\", \"Description\"]].head()" ] }, { "cell_type": "markdown", "id": "85ff691e", "metadata": {}, "source": [ "`sssom:NoTermFound` means Ensembl withdrew the gene without a replacement.\n", "\n", "The same HGNC table read the other way round gives symbol -> Ensembl ID." ] }, { "cell_type": "code", "execution_count": null, "id": "e5ca8f8e", "metadata": { "execution": { "iopub.execute_input": "2026-07-15T11:48:57.708071Z", "iopub.status.busy": "2026-07-15T11:48:57.707740Z", "iopub.status.idle": "2026-07-15T11:49:14.249273Z", "shell.execute_reply": "2026-07-15T11:49:14.248087Z" } }, "outputs": [], "source": [ "from mapkgsutils.context import XrefMapping, XrefRecord\n", "\n", "raw = pd.read_csv(src.url, sep=\"\\t\", dtype=str)\n", "pairs = raw[[src.object_label_col, src.subject_id_cols[\"ensembl\"]]].dropna()\n", "sym_to_ens = XrefMapping(\n", " records=[\n", " XrefRecord(subject_id=s, object_id=f\"ENSEMBL:{e}\") for s, e in pairs.itertuples(index=False)\n", " ]\n", ")\n", "\n", "hinted_ids = update_ids(\n", " df, ensembl, at=\"ensembl_curie\", xref=\"Description\", xref_mapping=sym_to_ens\n", ")\n", "print(f\"ambiguous before: {len(stale)}\")\n", "print(f\"ambiguous after: {len(hinted_ids[hinted_ids['ensembl_curie_primary'] == ''])}\")" ] } ], "metadata": { "kernelspec": { "display_name": ".venv (3.14.0)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.14.0" } }, "nbformat": 4, "nbformat_minor": 5 }