Refactoring the Vault Protein Complex Pipeline: From 15 Billion Sequences to a 34-Hour 3D Reconstruction

1. A structural puzzle left hanging for forty years

In 1986, a biology team isolated a previously undescribed class of particles from rat liver homogenate. On electron micrographs, its cross-sectional profile looked like the vault of a Gothic cathedral: a barrel-shaped shell, highly symmetric, open inside. That morphological association gave it its name — the vault. Later characterization produced more concrete parameters: a mass roughly three times that of a ribosome, making it one of the largest protein complexes known inside cells. In mammalian cells it is built from 78 protein chains, with two “half-barrels” of 39 chains each that close together to enclose a hollow barrel-shaped container.

The odd part: tens of thousands of these empty barrels sit in nearly every human cell, consuming a substantial biosynthetic cost, and the field has never been able to say what they are for. Three lines of evidence point to “absence” with no consequence:

• The most widely used model organisms — yeast, Drosophila, C. elegans — contain no homologs at all;
• When the major vault protein gene is knocked out in mice, development and reproduction remain normal;
• Without a reliable functional hypothesis, structural work has no validation standard; and an unclear structure in turn blocks functional inference.

So it became a classic “placeholder problem”: neither falsifiable nor easily advanced. The shared trait of such problems is not a lack of data, but the absence of a path that converts hypotheses quickly into measurable experiments. Moving it forward required not a breakthrough at a single step, but a simultaneous speedup across the whole research pipeline.

2. Where the real bottleneck lies for giant complexes

Giant protein complexes are difficult not because the molecules are big, but because uncertainty at every step is amplified along the pipeline.

Sequence level: a thin homology signal. Few known sequences and high sequence divergence mean that similarity-based retrieval easily misses genuine distant homologs. The family size stays small, so every downstream screen runs inside a small sample.

Expression level: a narrow assembly window. For a machine built from 78 chains, correct assembly in a heterologous host depends on expression strength, folding rate, and matching of the stoichiometry between chains. Most constructs precipitate as inclusion bodies, and the soluble expression rate is usually the hardest constraint at the screening stage.

Purification level: insufficient throughput. The traditional route relies on sucrose density gradient ultracentrifugation: the sample is layered on a gradient, spun for hours or overnight, then fractionated tube by tube by hand and interpreted by band position. One round takes days and depends heavily on operator experience — which means the number of candidate proteins has to be kept artificially low.

Structure determination level: compute and scheduling. Single-particle cryo-EM reconstruction is a compute-heavy process covering motion correction, contrast transfer function (CTF) estimation, particle picking, 2D classification, 3D classification and refinement. Repeated iterations on a single dataset easily consume tens of GPU-hours, and cycles of weeks or even months have historically been common.

Decision level: interpretation still needs human scientific judgment. AI can generate plans, but a judgment like “does this barrel have biological meaning” still has to be converged through an experimental feedback loop.

The conclusion is clear: any single-point speedup gets eaten by the slowest link. What works is engineering the whole chain so that sequence screening, construct design, purification optimization and data processing become orchestrable, traceable and reproducible — with an agent making local decisions along the chain.

3. Step one: distant homology search across 15 billion sequences

The team’s entry point was a change of scale: rather than looking for functional clues inside structurally complex higher organisms, widen the search radius into the prokaryotic world. After a broad-scale search across roughly 15 billion protein sequences, the result was that this family of proteins is broadly distributed across 29 bacterial phyla and 4 archaeal phyla, with large numbers of homologous genes also hidden in the genomes of phages that infect bacteria.

The known family size therefore expanded from 41 members to 1507, an increase of about 36-fold. That change in magnitude is itself information: once the sample moves from “single digits” into a statistically usable range, macro-level patterns such as phylogenetic distribution, domain combinations and gene neighborhood become meaningful. Two ecological clues were identified accordingly — the family is quite rare among prokaryotes, appearing in fewer than 1% of species, and it frequently co-localizes with antiviral defense gene clusters. This pushed the functional hypothesis from “unknown” to a testable direction: a prokaryotic anti-phage defense device.

Engineering-wise, this step is a typical batch heterogeneous retrieval problem: database version, search toolchain, parallel sharding, resumable runs, threshold tuning, result deduplication. An illustrative pipeline definition:

# Illustrative large-scale homology search pipeline
# (replace tool and library names with those of your actual environment)
# Goal: reproducibility -- input sequence set + database release
#       => candidate list with an evidence trail
STAGES = [
    {"name": "fetch",     "cmd": "download_db --ref {DB_RELEASE} --out /mnt/data/db"},
    {"name": "search",    "cmd": "remote_blast --query shards/*.faa "
                                "--db /mnt/data/db --evalue 1e-3 "
                                "--max-target-seqs 500 --threads 16"},
    {"name": "filter",    "cmd": "filter_hits --coverage-min 0.3 --identity-min 15 "
                                "--out hits.tsv"},
    {"name": "cluster",   "cmd": "mmseqs cluster hits.tsv rep_seqs.fasta "
                                "--min-seq-id 0.3 --cov-mode 1"},
    {"name": "neighborhood", "cmd": "extract_context --genes-upstream 10 "
                                   "--genes-downstream 10 --out loci.tsv"},
    {"name": "report",    "cmd": "summarize --group-by phylum,isolate_source "
                                "--out family_report.html"},
]

Three engineering points matter here. First, write the database version and parameters into the output metadata; otherwise a conclusion like “1507 members” cannot be re-verified. Second, shard in parallel and requeue failed shards — a single full search can consume hundreds of thousands of core-hours, so resumability is not an optimization but a requirement. Third, retrieval thresholds must be recorded explicitly, because distant-homology calls are extremely sensitive to thresholds; a small shift on the same dataset changes the family boundary.

4. Step two: from 44 candidates to 15 soluble constructs

With a candidate list in hand, the next step was turning sequences into protein. The team selected 44 candidates and handed them to an agent to handle synthetic DNA design, expression cassette configuration and the ordering plan: codon optimization matched to host preference, control of repetitive sequences and extreme GC regions, placement of restriction sites, plus a cloning strategy and primer plan, ending in a design list ready to be ordered. After transfer into E. coli, 15 candidate sequences yielded soluble protein.

The value of this step is not “writing fast” but “satisfying constraints simultaneously.” Construct design is essentially a multi-constraint combinatorial optimization problem: sequence length, synthesis cost, host codon preference, secondary-structure risk, tag position, uniqueness of restriction sites. Any single conflict invalidates the whole design. Checking each one by hand is both slow and error-prone, while encoding it as an executable rule set plus validation scripts filters out most unsynthesizable designs in minutes.

# Illustrative multi-constraint validation for construct design
def validate_construct(record, host_codon_table):
    problems = []
    if host_codon_table.cai(record.cds) < 0.75:
        problems.append("codon_adaptation_too_low")
    if max_homopolymer_run(record.cds) > 8:
        problems.append("homopolymer_risk")           # synthesis difficulty and sequencing quality
    if not (0.35 <= record.gc_fraction <= 0.65):
        problems.append("gc_out_of_range")
    sites = record.enzyme_sites()
    if len(sites) != len(set(sites)):
        problems.append("duplicate_restriction_site")
    if not record.has_unique_cloning_landmark():
        problems.append("cloning_strategy_ambiguous")
    return problems

It is worth stressing that a drop from 44 to 15 soluble constructs is not a poor rate — for a giant multi-subunit complex, solubility in a heterologous host is a rare event in itself. What this filter ratio provides is a large enough candidate base for the next step; without it, there is nothing to iterate on during purification.

5. Step three: compressing a three-day purification into 90 minutes

Traditional vault isolation depends on sucrose density gradient ultracentrifugation: the sample is layered on top of the gradient, spun for hours or overnight, then fractionated from top to bottom, with the target component judged from band position and intensity. The method is reasonably accurate, but throughput is a hard weakness — three days per round locks the number of purification iterations into the single digits.

The breakthrough came from the agent pushing back. The initial instruction was to optimize parameters inside the sucrose gradient framework (gradient concentration, rotor speed, centrifugation time). It judged that the ceiling of that path was three days, which could not support batch screening, and instead proposed abandoning ultracentrifugation for a chromatography route based on molecular-sieve (size-exclusion) effects, along with a candidate scheme matrix. After switching to chromatography, a single purification round was compressed to 90 minutes, and the daily capacity of the same instrument went from a fraction of a run to more than ten runs.

The methodological point deserves to be called out separately: when the ceiling of the optimization objective is set by the method itself, continuing to tune parameters is wasted effort — you have to change paradigms. The agent’s contribution here was not faster computation but recognizing the structural problem that the objective function had been capped by the methodology.

Three supporting engineering changes were equally important:

• Parameters as configuration. Gradient concentration, flow rate and buffer recipes are all written into versioned configuration files, so one experiment corresponds to one replayable configuration;
• Output as data. Chromatograms, fractions and purity assessments from every round are written automatically to object storage and indexed by candidate protein and round, instead of being scattered across lab notebooks;
• Interpretation as scripts. Fraction selection is performed automatically by quantitative rules such as peak area and target band position, reducing variance introduced by manual interpretation.

Together, these turn purification from a craft into a repeatable, callable service interface — which is precisely the precondition for AI to intervene and keep optimizing it.

6. Step four: 3D reconstruction in 34 hours

Once samples were ready, the workflow moved into single-particle cryo-EM analysis. The standard process covers motion correction of raw frames, contrast transfer function (CTF) estimation, automatic particle picking, 2D classification to remove junk, ab initio initial model generation, 3D classification to eliminate conformational heterogeneity, and non-uniform refinement with local resolution optimization. Every stage requires a decision on whether to continue or go back, and the basis for that decision is usually empirical statistics such as resolution curves, particle orientation distribution and anisotropy metrics.

The full run took 34 hours, compressing the historical cycle of weeks or even months into the order of one day. Four categories of attributable speedup:

1. Candidates already converged. The throughput gain at the purification stage greatly increased certainty about sample quality before submission, avoiding cryo-EM queue time waiting for samples and repeated preparation;
2. Predictable orientation coverage. Judging in advance — from 3D classification and orientation distribution — whether sample preparation conditions need adjustment, rather than redoing work after refinement fails;
3. Orchestrated workflow. Stages are chained as containerized tasks into a DAG; the output of one step automatically triggers the next, and failed nodes are rerun individually;
4. Elastic compute scheduling. GPU nodes are requested on demand, tasks are sharded and run in parallel, and intermediate products are cached and reused.

# Illustrative orchestration of a single-particle cryo-EM pipeline
stages:
  - {name: motion_correction, depends_on: [], gpu: 1}
  - {name: ctf_estimation,    depends_on: [motion_correction]}
  - {name: particle_pick,     depends_on: [ctf_estimation], gpu: 1}
  - {name: class2d,           depends_on: [particle_pick], gpu: 2}
  - {name: abinitio_refine,   depends_on: [class2d], gpu: 4}
  - {name: class3d,           depends_on: [abinitio_refine], gpu: 4}
  - {name: nonuniform_refine, depends_on: [class3d], gpu: 8}
artifacts: {root: "s3://project/vault/structure-v1", version_by: "run_id"}
policy: {retry: 2, cache_intermediate: true, notify_on_fail: true}

7. The infrastructure that supports pipelines like this

To chain the four steps above into a reproducible pipeline, the infrastructure needs to meet a few general requirements, independent of any particular vendor:

Capability	Role in the pipeline	Key requirements
Elastic GPU compute	Sequence retrieval, cryo-EM data processing	On-demand provisioning, sharded parallelism, tolerant of preemption
Object storage	Sequence databases, raw cryo-EM frames, intermediate products	Large-file throughput, versioning, lifecycle management
Container platform and orchestration	Environment isolation and scheduling for heterogeneous toolchains	Reproducible images, DAG orchestration, failure retries
Message queue and job scheduling	Decoupling long-running jobs, load smoothing, status reporting	At-least-once delivery, idempotent consumption
Logging and monitoring/alerting	Observability for long-cycle jobs, anomaly localization	Per-stage timing statistics, metrics dashboards
Data versioning and experiment tracking	Traceability of parameters, samples and results	One conclusion mapped to one complete metadata record

One caution: the role of large models and AI coding tools in this kind of work is author of the pipeline and local decision maker, not the compute engine. What actually consumes resources is deterministic computation such as retrieval and refinement; the model’s contribution is turning scattered manual steps into executable, verifiable, replayable automated workflows, and proposing alternative hypotheses when experimental feedback turns anomalous. On platforms such as Baidu AI Cloud, these capabilities are typically offered as combinations of general-purpose cloud services, so research teams can use capacity elastically at peak and avoid holding hardware long-term for one-off heavy computation.

8. A sober view: four red lines that must be set

Cases of “AI across the whole workflow” easily create the illusion that research has already been automated. From an engineering standpoint, the following risks have to be managed explicitly.

Reproducibility of conclusions. A number like 1507 family members is sensitive to thresholds and must be archived together with the database version, search parameters and clustering thresholds, or it cannot be re-verified. The recommendation is to write “parameters + version + input hash” into the metadata of every conclusion.

The hidden nature of hallucination. When generating experimental plans, a model will confidently fabricate reagent concentrations, centrifugation parameters or existing literature conclusions that do not exist. The defense is to route model output into deterministic validation: sequence-level checks with synthesis-feasibility scripts, workflow-level checks with parameter ranges and dependency relationships, rejecting anything out of bounds before it enters the next stage.

The value of negative results. Only 15 of 44 candidates expressed solubly, and this kind of “failure” data often never reaches a paper — yet it is critical for training downstream screening models. Engineering-wise, failed samples should be written to storage in full by default rather than discarded in the pipeline.

Human judgment cannot be outsourced. Whether structure corresponds to function, and whether a given anti-phage defense hypothesis holds, requires a wet-lab feedback loop and domain judgment. AI compresses the time from hypothesis to experiment, not the truth of the hypothesis itself.

9. What is transferable for R&D teams

Abstracting this chain reveals a reusable pattern: turn a long-stalled research problem into a pipeline problem of “high-throughput screening + fast experimental feedback + a data loop.”

Four concrete moves:

1. Do the dimensional conversion first. Translate “very hard” into concrete numbers — hours per round, rounds per day, success rate. Every bottleneck shows up in the numbers, not in intuition;
2. Identify objectives capped by methodology. If the theoretical ceiling of parameter optimization still fails to meet the requirement, changing paradigms beats tuning — and that is exactly where an agent adds value;
3. Turn manual steps into interfaces. Versioned configuration, standardized outputs and scripted interpretation are preconditions for automation to intervene, and the order cannot be reversed;
4. Apply deterministic constraints to model output. Let the model generate plans and let scripts judge compliance, keeping the two strictly separated.

What the vault case ultimately reveals is not the solving speed of any single model, but how much the “effective bandwidth” of an individual researcher is amplified once research infrastructure matures. Behind three numbers — family expansion from 41 to 1507, purification from 3 days to 90 minutes, reconstruction from weeks to 34 hours — lies the same thing: when the experimental feedback cycle drops from days to hours, the iteration speed of a scientific question makes an order-of-magnitude jump.