ECOD changed its distribution format: neither the version header nor the column layout is recognised #1139

After submitting the issues/PRs yesterday, I spent overnight digging the exact reason for the exception on the "version test" until I found it and so fixed it. I asked my AI agent to continue the effort.

Sub-issue of #1135, cause 2 of 3 — and the earlier diagnosis there was wrong. See the last section.

What changed upstream

ECOD redesigned its distribution at v294.1. Two things in the file break us.

1. The version comment

EcodInstallation.EcodParser matches ^\s*#.*ECOD\s*version\s+(\S+).*, which the new form does not satisfy. parsedVersion stays null and getVersion() falls back to the string that was requested — the literal "latest". That is precisely what the test asserts against:

EcodInstallationTest.testVersion:285  Values should be different. Actual: latest

2. The column layout

The parser accepts only 13, 14 or 15 fields, so every data line in v294.1 and v295 is rejected. Other things that changed with it:

• manual_rep now holds True/False rather than MANUAL_REP/AUTO_NONREP;
• assembly_id and domain_id_short are declared but empty on all 2,945,500 rows;
• the ligand list moved from the last column to ligand_comp_ids (column 24);
• f_name is empty rather than F_UNCLASSIFIED for unclassified domains, while f_id now carries a fourth level (1.1.1.3);
• uid restarts from 0;
• 1,573,776 of the 2,945,500 rows (53%) are domains classified from AlphaFold models, with ecod_domain_id values like P44140_F1_nD2 and an empty pdb column. EcodDomain is keyed by PdbId and cannot represent them.

ecod.latest.domains.txt itself still resolves, so no URL change is needed. The new /ecod/af2_pdb/distribution index links into the same /ecod/distributions/ directory — it is a second index over one file set, not a second host.

Impact beyond CI

This is not only a test failure. Anyone calling EcodFactory.getEcodDatabase() today — which defaults to latest — gets an empty domain list, silently:

EcodDatabase ecod = EcodFactory.getEcodDatabase();
ecod.getVersion();          // "latest", not "v295"
ecod.getAllDomains();       // empty
ecod.getDomainsForPdb("1cbs"); // null

The parser logs "Unexpected number of fields", capped at 10 warnings, and then hands back nothing. No exception is thrown.

Correcting the earlier diagnosis

#1135 recorded this as "EcodInstallationTest.testVersion downloads 657 MB from a slow host" and described it as a judgement call rather than a bug. That was wrong, and I would rather say so plainly than quietly move on. The download completes, the file is read, and the assertion then fails on the version string. The 657 MB is a genuine CI cost — pulled once per matrix job, five times per pull request — but it is not why the test fails, and no amount of making the download faster would have fixed it. That cost is an argument for #1137, not a bug in itself.

The reason a wholesale format change surfaced as a single one-line assertion failure is that EcodInstallationTest pins VERSION = "develop204" everywhere else. Only testVersion touches latest.

Reproducing

mvn verify -pl biojava-integrationtest -Dtest='EcodInstallationTest#testVersion'

Or without the download:

curl -r 0-400 http://prodata.swmed.edu/ecod/distributions/ecod.latest.domains.txt

Fix

A pull request follows. It reads files that declare a column header by column name rather than by position, which covers all four layouts at once and is the only approach that will not need revisiting the next time a column moves; keeps a positional fallback for pre-develop101 files that have no header; matches both version comment forms; and adds an offline unit test pinning every layout, so the next format change is caught in milliseconds rather than by a 657 MB download that only one test exercises.