Beyond the Benchmark: Tackling Data Bias to Build Generalizable Affinity Prediction Models

Abstract

Accurate prediction of drug-target binding affinity is crucial for computational drug discovery, yet the generalization capability of many deep learning models has been severely overestimated due to pervasive data bias. This article explores the critical issue of train-test data leakage and dataset redundancy in public benchmarks like PDBbind and CASF. We examine how these biases inflate performance metrics, present novel methodological solutions like the PDBbind CleanSplit protocol and similarity-aware evaluation frameworks for robust model training, and discuss advanced architectures that maintain performance on strictly independent tests. For researchers and drug development professionals, this synthesis provides a roadmap for developing and validating truly generalizable affinity prediction models to enhance real-world drug discovery pipelines.

The Benchmarking Mirage: Exposing Data Bias in Affinity Prediction

The Critical Role of Binding Affinity Prediction in Modern Drug Discovery

Drug-target binding affinity (DTA), which quantifies the strength of interaction between a small molecule (drug) and its protein target, serves as a fundamental metric in drug discovery and development. Accurate prediction of DTA is crucial for efficiently identifying promising drug candidates, understanding molecular interactions, and accelerating the lengthy and costly drug development process [1]. Traditional drug discovery is notoriously expensive, time-consuming, and prone to failure, often requiring over a decade and billions of dollars to bring a single drug to market [2] [3]. In this context, artificial intelligence (AI) and computational methods have emerged as potent substitutes over the last decade, providing strong answers to challenging biological issues and offering reliable alternatives that diminish the constraints of traditional experimental methods [2].

The evolution of DTA prediction has transitioned from physics-based simulations and traditional machine learning to sophisticated deep learning architectures. Early computational strategies relied mainly on physics-based methods like molecular docking and molecular dynamics simulations, which, while providing detailed structural insights, demand extensive computational resources and accurate structural input, limiting their applicability in large-scale screening [3]. The last decade has witnessed a paradigm shift with the widespread adoption of deep learning, which can handle large datasets and learn complex non-linear relationships, thus enabling more accurate and scalable DTA predictions [2].

However, a critical challenge has emerged that threatens the validity of many reported advances: data bias and inadequate generalization. Recent studies have revealed that train-test data leakage between standard benchmarks has severely inflated the performance metrics of many deep-learning-based models, leading to an overestimation of their true capabilities [4] [5]. This whitepaper provides an in-depth technical examination of DTA prediction methodologies, the critical issue of generalization, and the experimental frameworks essential for robust model development.

Key Methodologies in Binding Affinity Prediction

Evolution of Computational Approaches

The journey of DTA prediction methodologies can be broadly categorized into three distinct eras, each marked by increasing sophistication and performance.

• Conventional Physics-Based Methods: These early approaches, such as molecular docking, predict stable binding conformations and estimate affinities using scoring functions based on physical force fields, empirical data, or knowledge-based statistical potentials [1] [3]. While they offer valuable structural insights, their accuracy is often limited, and they are computationally intensive, making them unsuitable for large-scale virtual screening.

• Traditional Machine Learning Methods: From around 2005, methods like KronRLS and SimBoost began to gain traction [3]. These models learned from known drug-target binding data using manually curated features or similarity metrics (e.g., drug-drug and target-target similarity) [2] [1]. They demonstrated improved accuracy over conventional methods but were still constrained by their reliance on human-engineered features.

• Deep Learning-Based Methods: The increase in available structural and affinity data, coupled with enhanced computational power, facilitated the rise of deep learning. A significant advantage of deep learning is its ability to automatically learn relevant features from raw data, thus overcoming the limitation of manual feature selection [2]. Early deep learning models utilized convolutional neural networks (CNNs) and recurrent neural networks (RNNs) on one-dimensional sequences of drugs (e.g., SMILES strings) and proteins (amino acid sequences) [2]. Subsequently, the field has progressed through several advanced paradigms:

  • Graph-Based Models: These represent molecules as graphs, where atoms are nodes and bonds are edges. Models like GraphDTA use Graph Neural Networks (GNNs) to capture intricate structural information, providing a richer representation than sequences [3].
  • Attention-Based and Multimodal Architectures: Modern frameworks, such as HPDAF, integrate multiple data types (e.g., protein sequences, drug graphs, and binding pocket structures) using hierarchical attention mechanisms. This allows the model to dynamically focus on the most critical features for prediction [3].
  • Language Model Derivatives: The development of domain-specific large language models (LLMs) like ChemBERTa (for drugs) and ProtBERT (for proteins) has enabled the extraction of semantic features from chemical and biological sequences. The embeddings from these models can be combined with other architectures for enhanced prediction [2].
  • Equivariant Graph Networks: Cutting-edge approaches, such as DockBind, leverage equivariant graph neural networks (e.g., MACE) that respect physical symmetries to model detailed atomic environments from 3D docking poses, further incorporating physical and chemical descriptors [6].

Comparative Analysis of Deep Learning Architectures

Table 1: Comparison of Key Deep Learning Architectures for DTA Prediction.

Model Type	Key Features	Representative Models	Advantages	Limitations
Sequence-Based	Uses 1D SMILES for drugs and amino acid sequences for proteins.	DeepDTA, DeepAffinity [3]	Simple input; good performance improvement over pre-deep learning methods.	Ignores 3D structural information and specific binding pockets.
Graph-Based	Represents drugs and/or proteins as graphs to capture topology.	GraphDTA, GEMS [4] [3]	Better representation of molecular structure and atomic interactions.	Early models did not fully incorporate protein pocket data.
Pocket-Aware	Integrates structural information from protein-binding pockets.	PocketDTA, DeepDTAF [3]	Captures the local chemical environment where binding occurs, enhancing accuracy.	Relies on accurate pocket identification and definition.
Multimodal	Fuses multiple data types (sequence, graph, structure).	HPDAF, DockBind [6] [3]	Leverages complementary information; dynamic feature importance via attention.	Complex architecture; requires diverse and high-quality input data.
Physics-Informed	Incorporates physical principles and/or docking poses.	DockBind [6]	Provides a more physically realistic model of interactions.	Computationally expensive; depends on the accuracy of pose generation.

The following diagram illustrates the logical progression and relationships between these key methodological paradigms in the field.

Diagram 1: The evolution of methodologies in binding affinity prediction.

The Critical Challenge of Data Bias and Generalization

The PDBbind-CASF Data Leakage Problem

A groundbreaking study published in Nature Machine Intelligence (2025) exposed a fundamental flaw in the evaluation of deep-learning-based scoring functions [4] [5]. The field has heavily relied on the PDBbind database for training models and the Comparative Assessment of Scoring Functions (CASF) benchmark for testing. The study revealed a substantial train-test data leakage between these datasets, meaning that models were being tested on data that was highly similar to what they were trained on, rather than on truly novel challenges.

The researchers proposed a novel structure-based clustering algorithm to quantify the similarity between protein-ligand complexes in PDBbind and CASF. This algorithm uses a combined assessment of:

• Protein similarity (TM-scores)
• Ligand similarity (Tanimoto scores)
• Binding conformation similarity (pocket-aligned ligand root-mean-square deviation)

This analysis identified nearly 600 highly similar pairs between the training and test sets, affecting 49% of all CASF complexes [4]. This leakage allows models to "cheat" by memorizing structural similarities and associated affinity labels, rather than learning the underlying principles of protein-ligand interactions. Alarmingly, some models were found to perform comparably well on CASF benchmarks even after omitting all protein or ligand information, confirming that their predictions were not based on a genuine understanding of interactions [4].

The PDBbind CleanSplit Solution and Its Impact

To address this critical issue, the study introduced PDBbind CleanSplit, a new training dataset curated using their filtering algorithm to eliminate train-test data leakage and reduce redundancies within the training set itself [4]. The creation of CleanSplit involved two key steps:

• Removing train-test leakage: All training complexes that closely resembled any CASF test complex (based on the combined similarity metrics) were excluded. This also included training complexes with ligands identical to those in the test set (Tanimoto > 0.9).
• Reducing training set redundancy: The algorithm identified and iteratively removed complexes from large similarity clusters within the training data, which discourages mere memorization and encourages better generalization.

The impact of retraining existing state-of-the-art models on CleanSplit was profound. Models like GenScore and Pafnucy, which had previously shown excellent benchmark performance, saw their performance drop markedly when trained on the cleaned dataset [4]. This confirmed that their prior high scores were largely driven by data leakage. In contrast, the authors' Graph Neural Network for Efficient Molecular Scoring (GEMS), which leverages a sparse graph model and transfer learning from language models, maintained high performance when trained on CleanSplit, demonstrating robust generalization to strictly independent test data [4].

Experimental Protocols and Benchmarking

Standardized Evaluation Metrics and Datasets

Robust evaluation of DTA models requires standardized benchmarks and multiple metrics to assess different aspects of predictive power. The primary datasets used for training and evaluation include PDBbind, CASF, BindingDB, and others [1]. As discussed, the critical importance of using leakage-free splits like CleanSplit cannot be overstated for a genuine assessment of generalizability [4].

Table 2: Key Datasets for Drug-Target Binding Affinity Prediction.

Dataset	Complexes	Affinities	3D Structures	Primary Use
PDBbind	~19,588	~19,588	Yes	Primary training database for many models.
CASF	285	285	Yes	Standard benchmark for scoring power, docking power, ranking power.
BindingDB	~1.69 million	~1.69 million	Partial	Large-scale database for binding measurements; useful for pre-training.
Davis	N/A	Kinase-inhibitor data	N/A	Used for specific validation studies (e.g., kinase binding) [6].

Evaluation typically focuses on several "powers":

• Scoring Power: The ability to predict absolute binding affinity values, measured by the Pearson correlation coefficient (R) and the root-mean-square error (RMSE) between predicted and experimental values [1].
• Ranking Power: The ability to correctly rank ligands based on their affinity for a specific target, often measured by the Spearman correlation coefficient [1].
• Docking Power: The ability to identify the native binding pose among decoy poses [1].

Detailed Methodology: The HPDAF Framework

The HPDAF (Hierarchically Progressive Dual-Attention Fusion) framework exemplifies a modern, multimodal approach to DTA prediction [3]. Its experimental workflow and architecture provide a template for robust model development.

1. Data Representation and Input Modalities:

• Protein Sequences: Amino acid sequences are used as input.
• Drug Molecular Graphs: Drugs are represented as graphs with atoms as nodes and bonds as edges.
• Protein-Ligand Interaction Graphs: Structural information from protein-binding pockets is represented as graphs, capturing the local atomic environment crucial for binding.

2. Specialized Feature Extraction Modules:

• Each input modality is processed by a dedicated deep learning module (e.g., CNNs for sequences, GNNs for molecular graphs) to extract high-level, representative features.

3. Hierarchical Dual-Attention Fusion:

• This is the core innovation of HPDAF. The extracted features are fused using a two-tiered attention mechanism:
  • Modality-Aware Cross-Attention (MACN): This focuses on learning the importance of features within each modality (e.g., which atoms or residues are most critical).
  • Affinity-Aware Attention (AACN): This operates across modalities, dynamically calibrating and weighting the contributions of the protein, drug, and pocket features to the final affinity prediction.

4. Ablation Studies:

• To validate the contribution of each component, HPDAF employed ablation studies. These experiments systematically removed or altered parts of the model (e.g., using only sequence data, or removing the attention mechanisms). The results confirmed that the full multimodal model with dual-attention fusion achieved the best performance, significantly outperforming ablated versions and other state-of-the-art models on benchmarks like CASF-2016 [3].

The following workflow diagram outlines the key stages of a robust DTA prediction experiment, from data preparation to model validation.

Diagram 2: Workflow for robust binding affinity prediction experiments.

The Scientist's Toolkit: Essential Research Reagents

Table 3: Key Computational Tools and Resources for DTA Prediction Research.

Tool / Resource	Type	Primary Function	Relevance to DTA Prediction
PDBbind CleanSplit	Curated Dataset	Provides a leakage-free training and benchmark dataset.	Essential for training models that generalize to novel complexes; addresses data bias [4].
GEMS (Graph Neural Network for Efficient Molecular Scoring)	Software Model	A GNN model for binding affinity prediction.	Demonstrates robust generalization when trained on CleanSplit; uses sparse graphs and transfer learning [4].
HPDAF	Software Framework	A multimodal deep learning tool for DTA.	Integrates sequences, drug graphs, and pocket structures via hierarchical attention [3].
DockBind	Software Framework	A physics-informed DTA prediction framework.	Leverages docking poses from DiffDock and equivariant GNNs (MACE) to enhance affinity estimation [6].
ProtInter	Computational Tool	Calculates non-covalent interactions from PDB files.	Used to extract features (H-bonds, hydrophobic interactions) for machine learning models [7].
ESM & ChemBERTa	Pre-trained Language Model	Provides semantic embeddings for proteins and drugs.	Used for transfer learning, providing crucial sequence-based features for downstream DTA models [2] [6].

The field of binding affinity prediction is at a pivotal juncture. The exposure of widespread data bias has necessitated a re-evaluation of model performance and a renewed focus on true generalization. Future research will likely focus on several key areas:

• Advanced Data Curation: Widespread adoption of rigorous, structure-based dataset splitting, as exemplified by PDBbind CleanSplit, will become the standard to ensure fair and meaningful model evaluation [4].
• Integration of AI Virtual Cells (AIVCs): The FDA's move to phase out animal testing is accelerating the development of AI-driven in silico models. AIVCs offer a systems-level framework for modeling molecular interactions in dynamic, cell-specific contexts. Progress in DTA prediction will strengthen the molecular foundations of AIVCs, which in turn will provide more realistic simulation environments for testing affinity predictors [1].
• Temporal Dynamics and Multi-Omics Integration: Future models will need to move beyond static structures to simulate the temporal dynamics of binding and integrate multi-omics data to understand binding in a broader biological context, supporting more accurate and personalized therapeutic outcomes [1].

In conclusion, binding affinity prediction is a cornerstone of modern computational drug discovery. While deep learning has driven remarkable progress, the community must prioritize addressing data bias to build models that genuinely understand protein-ligand interactions. By leveraging multimodal architectures, physics-informed learning, and rigorously curated data, the next generation of DTA predictors will play an even more critical role in reducing the time and cost of bringing new medicines to patients.

The development of accurate scoring functions to predict protein-ligand binding affinity is a cornerstone of computational drug design. In recent years, deep learning models have promised to revolutionize this field. However, a critical and widespread issue has undermined their real-world applicability: a significant overestimation of their generalization capabilities due to train-test data leakage between the primary training database, PDBbind, and the standard evaluation benchmark, the Comparative Assessment of Scoring Functions (CASF) [4]. This leakage has created an illusion of performance, where models appear highly accurate during benchmarking but fail dramatically when faced with truly novel protein-ligand complexes. This problem strikes at the core of a broader thesis on data bias in affinity prediction research, revealing how biases in dataset construction can compromise the scientific validity of an entire field. The recent discovery that nearly half of the CASF test complexes have overly similar counterparts in the PDBbind training set has forced a major re-evaluation of model performance claims and dataset curation practices [4]. This whitepaper details the nature of this data leakage, its quantifiable impact on model performance, and the emerging solutions that aim to restore rigor and reliability to binding affinity prediction.

The Anatomy of Data Leakage in PDBbind and CASF

The PDBbind Database and CASF Benchmark

The PDBbind database is a comprehensive, curated collection of protein-ligand complexes sourced from the Protein Data Bank (PDB), each annotated with experimentally measured binding affinities [8]. It is typically divided into a "general" set used for training and a "refined" set of higher-quality complexes. The CASF benchmark, developed to assess the "scoring power" of predictive models, is often derived from this refined set [4] [8]. For years, the standard protocol involved training models on the general or refined PDBbind set and evaluating their performance on the CASF core sets (e.g., CASF-2013, CASF-2016). This practice was presumed to provide a fair assessment of a model's ability to generalize to unseen data. However, this protocol contained a fundamental flaw: the assumption that the CASF test sets were independent of the training data. It is now understood that this assumption was incorrect, leading to a systematic inflation of reported performance metrics across numerous published models [4].

Mechanisms of Train-Test Contamination

The data leakage between PDBbind and CASF is not merely a result of random overlap but stems from deep structural similarities between complexes in the training and test sets. Traditional sequence-based splitting methods, which rely on protein sequence identity, have proven insufficient to guarantee true independence. The leakage occurs through several specific mechanisms:

• Protein Structure Similarity: Complexes can share highly similar protein structures (high TM-scores) even when their sequence identity is low [4]. This allows models to recognize protein structural patterns from training data during testing.
• Ligand Chemical Similarity: Ligands in the test set may be chemically nearly identical (high Tanimoto similarity) to those in the training set, enabling prediction based on ligand memorization rather than understanding of interactions [4].
• Binding Conformation Similarity: The three-dimensional positioning of the ligand within the protein binding pocket (measured by pocket-aligned ligand RMSD) can be nearly identical between training and test complexes, providing almost identical input data points to the models [4].

When combined, these factors create a scenario where a test complex is not a genuinely new challenge for a trained model but rather a slight variation of what it has already encountered during training.

Quantifying the Overlap: A Structural Clustering Analysis

A Multimodal Filtering Algorithm

To rigorously quantify the extent of data leakage, a recent study introduced a novel structure-based clustering algorithm [4]. Unlike traditional methods that rely primarily on sequence identity, this algorithm performs a multimodal assessment of similarity between any two protein-ligand complexes by evaluating three key metrics simultaneously:

• Protein Similarity: Calculated using the TM-score, a metric for protein structural similarity that is more sensitive than sequence alignment, especially for proteins with low sequence identity [4].
• Ligand Similarity: Computed using the Tanimoto coefficient, a standard measure for comparing molecular fingerprints and assessing ligand chemical similarity [4].
• Binding Conformation Similarity: Determined by the pocket-aligned root-mean-square deviation (RMSD) of the ligand atoms, which measures how similarly the ligand is positioned in the binding pocket [4].

By combining these three metrics, the algorithm provides a robust and detailed comparison of protein-ligand complex structures, capable of identifying complexes with similar interaction patterns even when their protein sequences are divergent.

Quantitative Evidence of Widespread Leakage

The application of this filtering algorithm to the PDBbind and CASF datasets revealed a startling degree of data leakage. The analysis identified nearly 600 unacceptably close similarities between complexes in the PDBbind training set and those in the CASF benchmark set [4]. These structurally redundant pairs involved 49% of all CASF test complexes [4]. This means that nearly half of the test cases in the standard evaluation benchmark were not truly novel, but had highly similar counterparts in the training data. Consequently, models could achieve high benchmark performance not by learning general principles of binding but by exploiting these memorized similarities. The table below summarizes the key quantitative findings of the overlap analysis.

Table 1: Quantified Data Leakage Between PDBbind Training and CASF Test Sets

Metric of Similarity	Threshold for "Leakage"	Number of Leaky Pairs	Percentage of CASF Test Set Affected
Overall Structural Similarity	Combined assessment of TM-score, Tanimoto, and RMSD	~600 pairs	49%
Protein Structure (TM-score)	High similarity despite potential low sequence identity	Data not specified	Implied to be significant [4]
Ligand Chemistry (Tanimoto)	> 0.9	Data not specified	Addressed by filtering [4]

This widespread redundancy had a direct impact on model evaluation. To illustrate the effect, a simple search algorithm was devised that predicted the affinity of a CASF test complex by averaging the affinities of its five most similar training complexes. This straightforward, non-learning-based approach achieved a competitive Pearson correlation (R = 0.716) on the CASF2016 benchmark, rivaling some published deep-learning scoring functions [4]. This experiment starkly demonstrated that high benchmark performance could be achieved through data exploitation rather than genuine learning.

The CleanSplit Solution: A Methodology for Rigorous Data Curation

The PDBbind CleanSplit Protocol

In response to the data leakage crisis, the PDBbind CleanSplit dataset was created [4]. Its development involved a rigorous, multi-step filtering protocol designed to eliminate both train-test leakage and internal training set redundancies. The following diagram illustrates the workflow for creating this cleaned dataset.

Diagram 1: Workflow for creating the PDBbind CleanSplit dataset.

The methodology can be broken down into two primary phases:

• Eliminating Train-Test Leakage: The algorithm first performs an all-against-all comparison between CASF test complexes and PDBbind training complexes using the multimodal similarity assessment. Any training complex that exceeds similarity thresholds (e.g., Tanimoto > 0.9 for ligands) with any test complex is identified and removed from the training set. This step ensures that the ligands and structural motifs present in the test set are not encountered during training [4].
• Reducing Training Set Redundancy: The algorithm also addresses internal redundancies within the training data itself. The analysis found that nearly 50% of training complexes were part of a similarity cluster. Using adapted filtering thresholds, the algorithm iteratively removes complexes to resolve these clusters, resulting in a more diverse and less redundant training set. This step encourages models to learn generalizable patterns rather than relying on memorization [4].

The final output of this protocol is a cleaned training dataset that is strictly separated from the CASF benchmarks, allowing for a genuine evaluation of model generalization.

Impact on Model Performance Evaluation

The true test of the CleanSplit protocol was its impact on the performance of state-of-the-art affinity prediction models. When top-performing models like GenScore and Pafnucy were retrained on the PDBbind CleanSplit dataset, their performance on the CASF benchmark dropped substantially [4]. This performance drop confirmed that the previously reported high accuracy of these models was largely driven by data leakage and memorization, not by a robust understanding of protein-ligand interactions.

In contrast, a new graph neural network model named GEMS (Graph neural network for Efficient Molecular Scoring) maintained high benchmark performance when trained exclusively on CleanSplit [4]. This suggests that its architecture—which leverages a sparse graph model of interactions and transfer learning from language models—is better suited to learning generalizable principles. Furthermore, ablation studies showed that GEMS failed to produce accurate predictions when protein node information was omitted, indicating its predictions are based on a genuine understanding of the protein-ligand interaction rather than ligand memorization [4].

Experimental Validation and Researcher's Toolkit

Key Experimental Workflows

The validation of data leakage and the efficacy of new datasets like CleanSplit rely on specific experimental workflows. The core process for benchmarking a scoring function's true generalization capability involves a strict separation of training and test data, followed by a multi-faceted evaluation. The following diagram outlines this critical benchmarking workflow.

Diagram 2: Workflow for rigorously benchmarking a scoring function's generalization.

This workflow emphasizes two critical steps:

• Training on Leakage-Free Data: Using a curated dataset like CleanSplit as the exclusive training source.
• Comprehensive Evaluation: Going beyond simple scoring power (e.g., Pearson R, RMSE) to include diagnostic tests like ablation studies that probe whether the model is learning meaningful interactions.

The Scientist's Toolkit for Data Curation

To address the data leakage problem, researchers require a set of specialized tools and resources for curating and evaluating their protein-ligand data. The following table details key solutions.

Table 2: Research Reagent Solutions for Mitigating Data Leakage

Tool / Resource	Type	Primary Function in Leakage Mitigation
PDBbind CleanSplit [4]	Curated Dataset	Provides a pre-processed training set with minimized structural similarity to the CASF benchmark.
Multimodal Filtering Algorithm [4]	Algorithm/Methodology	Identifies redundant complexes based on combined protein TM-score, ligand Tanimoto, and binding pose RMSD.
HiQBind-WF [9] [8] [10]	Automated Workflow	An open-source, semi-automated workflow that corrects common structural artifacts in PDB files and creates high-quality datasets.
GEMS Model [4]	Software/Model	An example of a graph neural network architecture demonstrated to generalize well when trained on a leakage-free dataset.
Structure-Based Search Algorithm [4]	Diagnostic Tool	A simple non-learning algorithm that finds similar training complexes to a test query; used to demonstrate the feasibility of data exploitation.

The uncovering of profound train-test data leakage between PDBbind and CASF has served as a necessary corrective for the field of computational affinity prediction. It has demonstrated that the quest for better models must be intrinsically linked to the pursuit of better, more rigorously curated data. The development of solutions like the PDBbind CleanSplit dataset and the HiQBind workflow marks a pivotal shift towards a data-centric approach in the field [4] [9] [8]. These resources provide the foundation for developing models whose benchmark performance genuinely reflects their ability to generalize to novel targets and ligands, which is the ultimate requirement for accelerating drug discovery.

Looking forward, the field is moving beyond a singular focus on static 3D structures. Emerging efforts involve the creation of large-scale, high-quality datasets through initiatives like Target2035, a global consortium aiming to generate standardized protein-ligand binding data for thousands of human proteins [11]. Furthermore, there is a growing emphasis on incorporating molecular dynamics to capture the conformational flexibility of binding, and on using AI-based co-folding models to generate high-quality synthetic data, provided it is filtered with the same rigor advocated by the CleanSplit study [11]. The lesson is clear: future progress in binding affinity prediction depends on a continued synthesis of scale and quality, ensuring that models are trained on a foundation of truth rather than an illusion of performance.

In the field of computational drug design, the accuracy of binding affinity prediction models is paramount for identifying viable therapeutic candidates. However, a pervasive yet often overlooked issue—structural redundancy within training data—severely compromises the real-world performance of these models. Structural redundancy occurs when training and test datasets contain highly similar protein-ligand complexes, leading to a phenomenon known as train-test data leakage. This leakage allows models to perform well on benchmark tests by recognizing structural similarities rather than by genuinely learning the underlying principles of molecular interactions. Consequently, validation metrics become artificially inflated, creating a significant gap between benchmark performance and practical utility in drug discovery applications.

The core of this problem lies in the standard practice of training models on public databases like PDBbind and evaluating them on benchmarks from the Comparative Assessment of Scoring Functions (CASF). A 2025 study by Graber et al. revealed that nearly 49% of CASF test complexes had highly similar counterparts in the PDBbind training set [12]. This extensive overlap means that nearly half of the test complexes do not present novel challenges to the models, enabling performance through memorization rather than generalization. This tutorial explores the mechanisms through which structural redundancy inflates validation metrics, provides detailed protocols for identifying and mitigating this issue, and presents a framework for developing robust, generalizable affinity prediction models.

Quantitative Evidence of Data Leakage Impact

Performance Decay in Cleaned Datasets

Retraining existing state-of-the-art models on a properly filtered dataset provides the most direct evidence of how structural redundancy inflates performance metrics. When models like GenScore and Pafnucy were retrained on the PDBbind CleanSplit dataset—which removes structurally similar training-test pairs—their performance on the CASF-2016 benchmark dropped markedly [12]. This performance decay indicates that their previously reported high accuracy was largely driven by data leakage rather than true predictive capability.

Table 1: Performance Comparison of Models Trained on Standard vs. Cleaned Data

Model	Training Dataset	CASF-2016 RMSE	Performance Change	Generalization Assessment
GenScore	Original PDBbind	1.21	Baseline	Overestimated
GenScore	PDBbind CleanSplit	1.58	+30.6% RMSE increase	Substantially reduced
Pafnucy	Original PDBbind	1.34	Baseline	Overestimated
Pafnucy	PDBbind CleanSplit	1.72	+28.4% RMSE increase	Substantially reduced
GEMS (Novel GNN)	PDBbind CleanSplit	1.24	-	Maintained high performance

Structural Similarity Analysis Between Training and Test Sets

The extent of structural redundancy between standard training and test sets can be quantified using multimodal similarity assessment. Research has demonstrated that approximately 49% of complexes in the CASF benchmark share striking similarities with complexes in the PDBbind training set according to defined thresholds of protein structure, ligand chemistry, and binding conformation [12]. This analysis identified nearly 600 highly similar train-test pairs that enable model memorization.

Table 2: Analysis of Structural Similarity Clusters in Protein-Ligand Data

Similarity Metric	Threshold Value	Percentage of CASF Complexes Affected	Impact on Model Performance
Protein Structure (TM-score)	>0.7	34%	Enables protein-based memorization
Ligand Similarity (Tanimoto)	>0.9	28%	Enables ligand-based memorization
Binding Conformation (pocket-aligned RMSD)	<2.0Å	41%	Enables binding mode memorization
Combined Multimodal Similarity	All above thresholds	49%	Severe data leakage inflation

Methodologies for Identifying Structural Redundancy

Multimodal Structural Clustering Algorithm

Identifying structural redundancy requires a multimodal approach that assesses similarity across multiple dimensions of protein-ligand complexes. The clustering algorithm developed by Graber et al. combines three critical metrics to comprehensively evaluate complex similarity [12]:

Protein Similarity Assessment: Calculated using TM-scores, with values >0.7 indicating significant structural homology that often corresponds to functional similarity. This metric identifies proteins that share similar folds despite potential differences in sequence identity.

Ligand Similarity Assessment: Computed using Tanimoto coefficients based on molecular fingerprints, with values >0.9 indicating nearly identical chemical structures. This prevents models from memorizing affinity values for specific molecular structures.

Binding Conformation Assessment: Measured through pocket-aligned root-mean-square deviation (RMSD) of ligand positions, with values <2.0Å indicating nearly identical binding modes. This ensures that similar interaction geometries between training and test complexes are identified.

The algorithm employs an iterative clustering approach that groups complexes sharing similarities across all three dimensions, then selectively filters representatives to create a non-redundant dataset. This process effectively identifies and eliminates both train-test leakage and internal training set redundancies.

Diagram 1: Multimodal Structural Clustering Workflow (76 characters)

The PDBbind CleanSplit Protocol

The PDBbind CleanSplit protocol represents a standardized methodology for creating training datasets free from structural redundancy. The implementation involves these critical steps [12]:

Step 1: Cross-Dataset Comparison - Compare all CASF test complexes against all PDBbind training complexes using the multimodal similarity algorithm to identify problematic pairs.

Step 2: Train-Test Separation - Remove all training complexes that meet similarity thresholds (TM-score >0.7, Tanimoto >0.9, or RMSD <2.0Å) with any test complex.

Step 3: Internal Redundancy Reduction - Apply adapted thresholds to identify and eliminate the most striking similarity clusters within the training data, removing approximately 7.8% of complexes.

Step 4: Ligand-Based Filtering - Eliminate all training complexes with ligands identical to those in the test set (Tanimoto >0.9) to prevent ligand-based memorization.

This protocol resulted in the removal of 4% of training complexes due to train-test similarity and an additional 7.8% due to internal redundancies, creating a more challenging but realistic training scenario that genuinely tests model generalization.

Experimental Validation Protocols

Robust Validation Strategies for Affinity Prediction Models

Proper validation strategies are essential for obtaining accurate performance estimates free from the confounding effects of structural redundancy. The following protocols should be implemented to ensure reliable model assessment [12] [13]:

Strictly External Test Sets: Completely independent test sets with no structural similarity to training complexes based on the multimodal criteria previously described. Performance on these sets provides the only valid measure of generalization capability.

Nested Cross-Validation: When external test sets are unavailable, implement nested cross-validation where the inner loop performs hyperparameter tuning and the outer loop provides performance estimates. This prevents over-optimization during model selection.

Cluster-Based Cross-Validation: Instead of random splitting, ensure that all complexes within identified similarity clusters remain within the same split (either all in training or all in test) to prevent data leakage.

Ablation Studies: Systematically remove different input modalities (e.g., protein information, ligand information) to verify that predictions rely on genuine protein-ligand interaction understanding rather than memorization of single components.

Diagram 2: Robust Experimental Validation Protocol (76 characters)

Case Study: GEMS Model Architecture and Training

The Graph neural network for Efficient Molecular Scoring (GEMS) represents a case study in developing models resistant to the pitfalls of structural redundancy. The GEMS architecture and training protocol incorporate several features designed to promote genuine generalization [12]:

Sparse Graph Representation: Models protein-ligand interactions as sparse graphs where nodes represent protein residues and ligand atoms, and edges represent interactions within a defined spatial cutoff. This explicit representation of interactions discourages mere pattern matching.

Transfer Learning from Language Models: Incorporates protein language model embeddings to provide evolutionary information, reducing dependence on structural similarities alone.

Multi-Task Training: Combines binding affinity prediction with auxiliary tasks such as binding site prediction and functional classification to encourage learning of generalizable representations.

When trained on the PDBbind CleanSplit dataset, GEMS maintained a high CASF-2016 prediction RMSE of 1.24, in contrast to the significant performance drops observed in other models. Ablation studies confirmed that GEMS fails to produce accurate predictions when protein nodes are omitted, indicating that its predictions are based on genuine understanding of protein-ligand interactions rather than exploiting data leakage.

Research Reagent Solutions

Table 3: Essential Research Tools for Structural Redundancy Analysis

Tool/Resource	Function	Application Context
PDBbind Database	Comprehensive collection of protein-ligand complexes with binding affinity data	Primary source of training data for affinity prediction models
CASF Benchmark	Standardized test sets for scoring function evaluation	Performance benchmarking; requires careful similarity analysis
Foldseek Cluster	Structural alignment-based clustering algorithm	Identifying similar protein structures at scale [14]
TM-align Algorithm	Protein structure comparison tool	Quantifying protein structural similarity (TM-scores)
RDKit	Cheminformatics toolkit	Calculating ligand similarities (Tanimoto coefficients)
PDBbind CleanSplit	Curated training dataset with reduced structural redundancy	Training and evaluation without data leakage [12]
GEMS Implementation	Graph neural network for binding affinity prediction	Reference model with robust generalization capabilities

Structural redundancy in training data represents a critical challenge in developing reliable binding affinity prediction models for drug discovery. The artificial inflation of validation metrics through data leakage gives a false impression of model capability, ultimately hindering the drug development process when these models fail in real-world applications. Through the implementation of rigorous multimodal clustering algorithms, careful dataset curation following protocols like PDBbind CleanSplit, and robust validation strategies that properly separate training and test data, researchers can develop models with genuine generalization capability. The field must move beyond convenient but flawed benchmarking practices and adopt these more stringent standards to accelerate meaningful progress in computational drug design.

In the field of computational drug design, accurately predicting the binding affinity between a protein and a small molecule ligand is a fundamental task crucial for identifying promising therapeutic compounds. Deep-learning-based scoring functions have emerged as powerful tools for this purpose, often demonstrating exceptionally high performance on standard benchmarks. However, a growing body of evidence indicates that these impressive results are frequently inflated by a critical flaw: train-test data leakage. This case study examines how when models are prevented from memorizing test data through a cleaned dataset, their performance substantially drops, revealing their true generalization capabilities and challenging the perceived progress in the field [12] [4].

The core issue lies in the standard practice of training models on the PDBbind database and evaluating them on the Comparative Assessment of Scoring Functions (CASF) benchmark. Studies have shown that these datasets share a high degree of structural similarity, meaning models can perform well by recognizing patterns seen during training rather than by genuinely understanding underlying protein-ligand interactions. This case study analyzes the impact of removing this leakage using the novel PDBbind CleanSplit dataset and explores a model architecture that maintains robust performance under these stricter conditions, providing a framework for building more reliable affinity prediction tools [12] [15].

The Data Leakage Problem in Affinity Prediction

Origins and Mechanisms of Leakage

The data leakage between PDBbind and CASF benchmarks is not merely a statistical oversight but is rooted in the structural similarities between the complexes in these datasets. When models are trained on PDBbind and tested on CASF, nearly half (49%) of the test complexes have exceptionally similar counterparts in the training set [12]. These similarities exist across multiple dimensions:

• Protein similarity: High TM-scores indicating similar protein structures [12]
• Ligand similarity: Tanimoto scores >0.9, reflecting nearly identical ligand molecules [12]
• Binding conformation similarity: Low pocket-aligned ligand root-mean-square deviation (r.m.s.d.), meaning nearly identical binding modes [12]

This multi-dimensional similarity creates a scenario where test data points are virtually identical to training data points, allowing models to achieve high accuracy through pattern recognition and memorization rather than learning fundamental principles of molecular recognition. Alarmingly, some models maintain competitive performance on CASF benchmarks even when critical input features are omitted, such as all protein or all ligand information, confirming that their predictions are not based on a genuine understanding of interactions [12] [4].

Documented Impacts on Model Performance

The inflation of performance metrics due to data leakage has been independently verified across multiple studies. Research from 2023 highlighted that random splitting of protein-ligand data allows similar sequences to be present in both training and test sets, leading to overoptimistic results that do not reflect true generalization ability [15]. The study found that this bias rewards overfitting, as the test set no longer provides a valid indication of how the model will perform on truly novel complexes.

Further investigation revealed that protein-only and ligand-only models could achieve surprisingly high accuracy on standard benchmarks, demonstrating that the predictive signal was coming from memorization of individual components rather than learning their interactions [15]. This finding fundamentally undermines the premise of structure-based affinity prediction and explains why models that excel on benchmarks often fail in real-world virtual screening applications.

The PDBbind CleanSplit Solution

A Novel Filtering Methodology

To address the data leakage problem, researchers developed a structure-based clustering algorithm that systematically identifies and removes similarities between training and test complexes [12] [4]. This algorithm employs a multi-modal approach that compares complexes across three key dimensions simultaneously:

• Protein similarity using TM-scores [12]
• Ligand similarity using Tanimoto scores [12]
• Binding conformation similarity using pocket-aligned ligand root-mean-square deviation (r.m.s.d.) [12]

This comprehensive approach can identify complexes with similar interaction patterns even when the proteins share low sequence identity, overcoming limitations of traditional sequence-based filtering methods [12]. The algorithm applies specific thresholds to determine unacceptable similarity, though the exact numerical thresholds are detailed in the methodology section of the original publication [12].

CleanSplit Dataset Construction

The filtering process to create PDBbind CleanSplit involves two critical phases:

• Reducing train-test leakage: The algorithm excludes all training complexes that closely resemble any CASF test complex based on the multi-modal similarity assessment. Additionally, it removes training complexes with ligands nearly identical to those in the test set (Tanimoto > 0.9). This combined filtering removed 4% of all training complexes [12].

• Minimizing training set redundancy: The algorithm identified that nearly 50% of all training complexes belonged to similarity clusters, meaning random train-validation splits would still inflate performance metrics. Using adapted thresholds, the process iteratively removed complexes until the most striking similarity clusters were resolved, eliminating an additional 7.8% of training complexes [12].

The resulting PDBbind CleanSplit dataset is strictly separated from the CASF benchmarks, transforming them into truly external datasets that enable genuine evaluation of model generalizability [12] [4].

Experimental Workflow for Dataset Filtering

The following diagram illustrates the comprehensive workflow for creating the CleanSplit dataset, from initial analysis to the final filtered dataset:

Performance Comparison on Cleaned Data

Experimental Protocol for Model Evaluation

To quantify the impact of data leakage, researchers designed a rigorous evaluation protocol [12] [4]:

• Model Selection: Multiple state-of-the-art binding affinity prediction models were selected, including GenScore and Pafnucy as representatives of top-performing architectures [12].

• Training Regimen: Each model was trained under two conditions: first on the original PDBbind dataset, then on the PDBbind CleanSplit dataset. All other hyperparameters and architectural details remained identical between conditions.

• Evaluation Benchmark: Model performance was assessed on the standard CASF benchmark, with particular attention to the root-mean-square error (r.m.s.e.) and Pearson correlation coefficient (R) as key metrics [12].

• Baseline Comparison: A simple search algorithm was implemented as a baseline, which predicts affinity by averaging the labels of the five most similar training complexes. This demonstrates the performance achievable through pure memorization [12].

Quantitative Results and Comparison

The table below summarizes the performance changes observed when models were transitioned from the original PDBbind dataset to the CleanSplit version:

Table 1: Performance Comparison on CASF Benchmark Before and After CleanSplit

Model / Method	Training Data	Performance Metric	Impact of Data Leakage
GenScore	Original PDBbind	High benchmark performance	Substantial performance drop on CleanSplit [12]
Pafnucy	Original PDBbind	High benchmark performance	Marked performance decrease on CleanSplit [12]
GEMS (Ours)	PDBbind CleanSplit	Maintains high performance	Genuine generalization to independent test sets [12]
Similarity Search Algorithm	Original PDBbind	Competitive performance (R=0.716)	Demonstrates memorization capability [12]

The performance drops observed in established models confirm that their previously reported high accuracy was largely driven by data leakage rather than true understanding of protein-ligand interactions [12].

GEMS: A Model Designed for Generalization

Architectural Innovations

In response to the generalization challenges revealed by CleanSplit, researchers developed the Graph neural network for Efficient Molecular Scoring (GEMS). This architecture incorporates several key innovations designed to promote robust learning [12]:

• Sparse graph modeling: Represents protein-ligand interactions as sparse graphs, focusing computational resources on relevant interfacial regions rather than processing entire complexes uniformly [12].

• Transfer learning from language models: Leverages pre-trained representations from protein language models, incorporating evolutionary information and structural priors that enhance generalization, especially on limited data [12].

• Interaction-aware conditioning: Utilizes universal patterns of protein-ligand interactions (hydrogen bonds, salt bridges, hydrophobic interactions, π-π stackings) as prior knowledge to guide the model toward physiologically meaningful features [12] [16].

Validation Through Ablation Studies

To verify that GEMS makes predictions based on genuine protein-ligand interactions rather than exploiting biases, researchers conducted critical ablation studies [12]:

• Protein node omission: When protein nodes were removed from the input graph, GEMS failed to produce accurate predictions, confirming that its performance depends on modeling both interaction partners rather than relying on ligand information alone [12].

• Interaction pattern analysis: The model's attention mechanisms were found to align with known interaction hotspots in protein binding sites, demonstrating that it learns biophysically meaningful representations [16].

These experiments confirm that GEMS maintains its performance on CleanSplit by developing a genuine understanding of molecular interactions rather than exploiting dataset-specific biases [12].

Implications for Drug Discovery

Virtual Screening and Lead Optimization

The development of properly validated affinity predictors has significant implications for structure-based drug design. Generative AI models like RFdiffusion and DiffSBDD can create vast libraries of novel protein-ligand complexes, but identifying therapeutically promising candidates requires accurate affinity prediction [12]. Models with genuine generalization capability, validated on strictly independent test sets, can fill this critical gap in the drug discovery pipeline.

For lead optimization, interaction-aware models like GEMS and frameworks like DeepICL can guide molecular modifications that enhance binding affinity while maintaining favorable drug properties [16]. By focusing on universal interaction patterns rather than dataset-specific correlations, these approaches offer more reliable guidance for medicinal chemists.

Future Research Directions

This case study points to several important directions for future research:

• Standardized benchmarking: The field would benefit from adopting cleaned benchmarks like CleanSplit as standard evaluation frameworks to prevent inflated performance claims [12] [15].

• Explicit interaction modeling: Future architectures should explicitly incorporate biophysical constraints and interaction principles to reduce reliance on correlational patterns that may not generalize [16].

• Multi-target generalization: Developing models that maintain accuracy across diverse protein families and binding sites remains an important challenge [15].

The Scientist's Toolkit: Essential Research Reagents

Table 2: Key Experimental Resources for Bias-Free Affinity Prediction

Resource Name	Type	Function / Application
PDBbind CleanSplit	Dataset	Training data with minimized train-test leakage for proper model validation [12]
CASF Benchmark	Benchmark	Standardized test set for comparing scoring functions [12]
Structure-Based Clustering Algorithm	Algorithm	Identifies similar protein-ligand complexes based on structure to detect data leakage [12]
PLIP (Protein-Ligand Interaction Profiler)	Software	Automatically identifies non-covalent interactions from structural data [16]
GEMS Architecture	Model	Graph neural network with transfer learning for generalization [12]
DeepICL	Model	Interaction-aware generative model for ligand design [16]
TM-score	Metric	Quantifies protein structural similarity independent of sequence [12]
Tanimoto Coefficient	Metric	Measures ligand similarity based on molecular fingerprints [12]
Pocket-Aligned Ligand RMSD	Metric	Assesses binding pose similarity [12]

This case study demonstrates that the impressive benchmark performance of many deep-learning-based affinity prediction models is substantially inflated by data leakage between standard training and test datasets. When models are prevented from memorizing test data through the PDBbind CleanSplit protocol, their performance drops markedly, revealing more limited generalization capabilities than previously assumed.

The development of models like GEMS that maintain robust performance on cleaned datasets points the way forward for the field. By employing architectures that explicitly model protein-ligand interactions through sparse graphs and transfer learning, and by validating on strictly independent test sets, researchers can develop more reliable tools for computational drug discovery. Widespread adoption of rigorous data splitting practices and interaction-aware modeling approaches will be essential for building predictive models that translate effectively to real-world drug design applications.

The generalization capability of machine learning models in computational drug design has been significantly overestimated due to pervasive train-test data leakage and inadequate assessment of complex similarity. Conventional benchmarks, which rely on random data splitting or sequence-based identity measures, fail to detect subtle structural similarities that enable models to exploit memorization rather than developing genuine understanding of protein-ligand interactions. This technical guide introduces a multimodal framework for assessing complex similarity that integrates protein structural similarity, ligand chemical similarity, and binding conformation similarity. By implementing the PDBbind CleanSplit methodology and retraining state-of-the-art models on this rigorously filtered dataset, we demonstrate a substantial performance drop in existing models—from Pearson R=0.816 to 0.641 for top performers—while our Graph Neural Network for Efficient Molecular Scoring (GEMS) maintains robust performance (Pearson R=0.779). This work establishes a new paradigm for evaluating and developing affinity prediction models with truly generalizable capabilities, addressing critical data bias issues that have plagued the field for decades.

Accurate prediction of protein-ligand binding affinities stands as a cornerstone of computational drug design, yet the field has been hampered by systematically inflated performance metrics and overestimated generalization capabilities. The root cause lies in inadequate assessment of complex similarity and subsequent data leakage between training and testing datasets. Current state-of-the-art deep learning models for binding affinity prediction typically train on the PDBbind database and evaluate generalization using the Comparative Assessment of Scoring Function (CASF) benchmarks [4]. However, studies reveal that nearly half (49%) of CASF test complexes have exceptionally similar counterparts in the training set, providing nearly identical input data points that enable accurate prediction through simple memorization rather than genuine understanding of protein-ligand interactions [4].

The conventional approach to dataset splitting has relied predominantly on sequence identity, failing to capture the multidimensional nature of molecular recognition. This oversight has created an illusion of progress while models increasingly master the art of pattern matching within biased datasets rather than developing robust predictive capabilities for novel complexes. The consequences extend throughout the drug discovery pipeline, where models that perform exceptionally on benchmarks fail dramatically in real-world applications on truly novel targets [4] [17].

This whitepaper introduces a multimodal framework for assessing complex similarity that transcends sequence-based metrics alone. By simultaneously evaluating protein structure, ligand chemistry, and binding conformation, we establish a rigorous methodology for creating truly independent datasets and evaluating model performance. Within the broader thesis of data bias and generalization in affinity prediction research, this work provides both a critical analysis of current shortcomings and a practical roadmap for developing models with robust, generalizable predictive capabilities.

The Data Leakage Problem in Affinity Prediction

Quantifying Train-Test Similarity

Recent investigations have exposed severe train-test data leakage between the PDBbind database and CASF benchmarks, fundamentally undermining claims of generalization in binding affinity prediction models. When analyzing the relationship between PDBbind training complexes and CASF test complexes, researchers identified approximately 600 similarity pairs sharing not only similar ligand and protein structures but also comparable ligand positioning within protein pockets [4]. Alarmingly, these structurally similar complexes naturally exhibit closely matched affinity labels, creating a direct pathway for models to achieve high benchmark performance through memorization.

The scope of this data leakage is substantial, affecting 49% of all CASF complexes [4]. This means nearly half the test instances do not present novel challenges to models trained on PDBbind, as highly similar examples exist in the training data. This leakage explains the dramatic performance deterioration observed when models transition from benchmark evaluation to real-world deployment on genuinely novel targets.

Limitations of Current Data Splitting Strategies

Current dataset partitioning strategies in affinity prediction research suffer from fundamental limitations that perpetuate the data leakage problem:

• Random splitting produces spuriously high correlations that inflate performance estimates, as structurally similar complexes inevitably appear in both training and testing sets [17].
• Sequence-based splitting (e.g., UniProt-based partitioning) reduces accuracy but fails to address structural similarities that persist despite sequence differences [17].
• Ligand-based splitting overlooks protein structural similarities and binding pose conservation, allowing models to exploit protein-level memorization.

Studies evaluating data partitioning strategies for predicting protein-ligand binding free energy changes demonstrate that while models show high predictive correlations (Pearson coefficients up to 0.70) under random partitioning, their performance significantly declines with more rigorous UniProt-based partitioning [17]. This performance drop reveals the true generalization capability of models absent data leakage.

Multimodal Similarity Assessment Framework

Core Similarity Metrics

Our multimodal similarity assessment framework integrates three complementary metrics that collectively capture the complexity of protein-ligand interactions:

Protein Similarity (TM-score)

• Measurement: Template Modeling score quantifies protein structural similarity
• Scale: 0-1, where >0.5 indicates generally the same fold
• Advantage: Detects structural similarity even with low sequence identity
• Application: Identifies proteins with similar binding pockets despite sequence divergence

Ligand Similarity (Tanimoto Coefficient)

• Measurement: Computed based on molecular fingerprints
• Scale: 0-1, where 1 indicates identical compounds
• Threshold: >0.9 considered highly similar for data splitting
• Application: Prevents ligand-based memorization

Binding Conformation Similarity (Pocket-Aligned Ligand RMSD)

• Measurement: Root-mean-square deviation of ligand atoms after pocket alignment
• Scale: Ångstroms, lower values indicate similar binding modes
• Application: Identifies complexes with similar interaction geometries

Table 1: Multimodal Similarity Assessment Metrics

Metric	Measurement Type	Scale	Threshold for Exclusion	Primary Function
Protein TM-score	Structural alignment	0-1	>0.5	Identify similar binding pockets
Ligand Tanimoto Coefficient	Chemical fingerprint	0-1	>0.9	Prevent ligand memorization
Binding Conformation RMSD	Spatial coordinate comparison	Ångstroms	<2.0Å	Identify similar binding poses

Filtering Algorithm and Workflow

The multimodal filtering algorithm processes protein-ligand complexes through a structured workflow that systematically identifies and removes complexes with unacceptable similarity across multiple dimensions. The algorithm employs iterative comparison and cluster resolution to ensure both train-test independence and reduced internal dataset redundancy.

Diagram Title: Multimodal Filtering Workflow for CleanSplit

Implementation: PDBbind CleanSplit

The application of our multimodal filtering algorithm to the PDBbind database produces PDBbind CleanSplit, a training dataset rigorously separated from CASF benchmark datasets. The filtering process involves two critical phases:

Phase 1: Train-Test Separation

• Removes all training complexes closely resembling any CASF test complex
• Excludes training complexes with ligands identical to CASF test complexes (Tanimoto > 0.9)
• Eliminates 4% of training complexes to ensure test independence
• Results in structurally distinct train-test pairs with clear differences

Phase 2: Internal Redundancy Reduction

• Identifies and resolves similarity clusters within training data
• Iteratively removes complexes until all striking similarity clusters are resolved
• Eliminates 7.8% of training complexes to reduce memorization bias
• Creates a more diverse training basis that encourages generalization

Table 2: PDBbind CleanSplit Filtering Impact

Filtering Phase	Complexes Removed	Similarity Type Addressed	Impact on Model Training
Train-Test Separation	4% of training set	Direct and indirect leakage	Prevents test set memorization
Internal Redundancy Reduction	7.8% of training set	Within-dataset similarities	Reduces memorization tendency
Total Filtering	11.8% overall reduction	Multimodal similarities	Encourages genuine learning

After filtering, the remaining train-test pairs with highest similarity exhibit clear structural differences, confirming the effectiveness of our approach in creating truly independent datasets for model evaluation [4].

Experimental Protocols

Data Preparation and Filtering Methodology

The PDBbind CleanSplit curation process follows a rigorous experimental protocol to ensure comprehensive similarity assessment and filtering:

Step 1: Multimodal Comparison

• Compute all-pairs similarity between training and test complexes
• Calculate TM-scores for all protein pairs using structural alignment
• Compute Tanimoto coefficients for all ligand pairs using extended-connectivity fingerprints
• Calculate pocket-aligned ligand RMSD for complexes with TM-score > 0.4 and Tanimoto > 0.7
• Store similarity metrics in structured database for filtering decisions

Step 2: Train-Test Filtering

• Identify all training complexes with TM-score > 0.5 to any test complex
• Identify all training complexes with Tanimoto > 0.9 to any test ligand
• Identify all training complexes with RMSD < 2.0Å to any test complex
• Remove all identified training complexes from the dataset
• Verify separation by re-computing similarities on filtered set

Step 3: Internal Redundancy Reduction

• Apply adapted thresholds (TM-score > 0.8, Tanimoto > 0.95, RMSD < 1.5Å) for internal filtering
• Identify similarity clusters using graph-based community detection
• Iteratively remove complexes from each cluster, preserving maximal diversity
• Continue until no clusters exceed similarity thresholds
• Balance dataset size against diversity requirements

Model Retraining and Evaluation Protocol

To validate the impact of CleanSplit on model generalization, we implemented a comprehensive retraining and evaluation protocol:

Model Selection and Retraining

• Select state-of-the-art binding affinity prediction models (GenScore, Pafnucy, GEMS)
• Train each model on both standard PDBbind and PDBbind CleanSplit
• Maintain identical hyperparameters and training procedures across datasets
• Implement early stopping based on validation performance
• Save model checkpoints for performance comparison

Evaluation Metrics and Benchmarks

• Evaluate all models on CASF-2016 and CASF-2018 benchmarks
• Calculate standard metrics: Pearson R, RMSE, MAE
• Perform statistical significance testing on performance differences
• Conduct ablation studies to isolate contribution of different filtering phases
• Analyze performance on different similarity subgroups

Ablation Study Design

• Train models with progressively stricter filtering thresholds
• Measure performance impact of individual similarity metrics
• Evaluate model robustness to different types of novelty
• Assess trade-offs between dataset size and diversity

Quantitative Results and Performance Analysis

Impact of CleanSplit on Existing Models

Retraining current top-performing binding affinity prediction models on PDBbind CleanSplit revealed dramatic performance drops, confirming that their benchmark performance was largely driven by data leakage rather than genuine generalization capability.

Table 3: Model Performance Before and After CleanSplit Training

Model	Original PDBbind (Pearson R)	CleanSplit Training (Pearson R)	Performance Drop	Generalization Gap
GenScore	0.816	0.641	21.4%	High
Pafnucy	0.792	0.603	23.9%	High
GEMS (Ours)	0.779	0.754	3.2%	Low

The substantial performance degradation observed in GenScore and Pafnucy when trained on CleanSplit indicates their heavy reliance on data leakage for benchmark performance. In contrast, our GEMS model maintains robust performance, demonstrating genuine generalization capability to strictly independent test datasets [4].

Structural Similarity Search Performance

To further illustrate the impact of data leakage, researchers devised a simple similarity search algorithm that predicts binding affinity by identifying the five most similar training complexes and averaging their affinity labels. This simple non-learning algorithm achieved competitive performance on CASF2016 (Pearson R = 0.716, RMSE = 1.45) compared to some published deep-learning-based scoring functions [4]. This result starkly demonstrates that sophisticated deep learning models may be essentially replicating this simple similarity matching rather than learning fundamental principles of protein-ligand interactions.

The Scientist's Toolkit: Research Reagent Solutions

Table 4: Essential Research Reagents and Resources

Resource	Type	Primary Function	Access Information
PDBbind Database	Data Resource	Comprehensive collection of protein-ligand complexes with binding affinity data	Publicly available at https://www.pdbbind.org.cn/
CASF Benchmark	Evaluation Suite	Standardized benchmark for scoring function assessment	Included with PDBbind distribution
PDBbind CleanSplit	Curated Dataset	Data-leakage-free training dataset for robust model development	Available via publication supplementary materials
GEMS Model	Software Tool	Graph neural network for binding affinity prediction with proven generalization	Python code publicly available
Structure-Based Clustering Algorithm	Software Tool	Multimodal similarity assessment and filtering tool	Available via publication supplementary materials

Discussion and Future Directions

Implications for Model Development

The multimodal similarity assessment framework fundamentally changes how we develop and evaluate affinity prediction models. By addressing the critical issue of data leakage, researchers can now focus on building models with genuine understanding of protein-ligand interactions rather than optimizing for benchmark exploitation. The maintained performance of our GEMS model on CleanSplit demonstrates that robust generalization is achievable through appropriate architectures and training regimens.

The graph neural network architecture of GEMS, which leverages sparse graph modeling of protein-ligand interactions and transfer learning from language models, proves particularly suited for generalization to strictly independent test datasets [4]. Ablation studies confirming that GEMS fails to produce accurate predictions when protein nodes are omitted from the graph provide evidence that its predictions stem from genuine understanding of protein-ligand interactions rather than dataset artifacts.

Applications in Structure-Based Drug Design

The multimodal assessment framework and CleanSplit methodology have profound implications for structure-based drug design (SBDD). Generative models such as RFdiffusion and DiffSBDD can create extensive libraries of novel protein-ligand interactions, but their practical utility has been bottlenecked by the absence of accurate affinity prediction models for these novel complexes [4]. With robust generalization capabilities validated on strictly independent datasets, models like GEMS provide the accurate affinity predictions needed to identify interactions with genuine therapeutic potential.

Future work should focus on extending the multimodal similarity framework to additional dimensions including solvation effects, conformational dynamics, and allosteric mechanisms. Additionally, developing standardized benchmarking protocols that incorporate multimodal similarity assessment will ensure the field continues to advance toward genuinely generalizable models rather than benchmark-specific optimization.

This technical guide has established a comprehensive framework for multimodal assessment of complex similarity that transcends the limitations of sequence-based metrics. By simultaneously evaluating protein structural similarity, ligand chemical similarity, and binding conformation similarity, we can create rigorously independent datasets that enable true evaluation of model generalization capability. The significant performance drops observed in state-of-the-art models when trained on PDBbind CleanSplit expose the pervasive data leakage that has inflated reported performance metrics across the field.

The maintained performance of our GEMS model under these rigorous conditions demonstrates that genuine generalization is achievable through appropriate architectural choices and training methodologies. As the field progresses toward increasingly complex challenges in drug design, adopting rigorous multimodal similarity assessment will be essential for developing models with robust real-world applicability rather than merely impressive benchmark performance.

Building Better Benchmarks: Methodological Solutions for Robust Training

The field of computational drug design relies on accurate scoring functions to predict protein-ligand binding affinities. However, the generalization capability of deep-learning models has been severely overestimated due to train-test data leakage between the PDBbind database and Comparative Assessment of Scoring Functions (CASF) benchmark datasets. This whitepaper introduces PDBbind CleanSplit, a rigorously curated training dataset created through a novel structure-based filtering algorithm that eliminates data leakage and internal redundancies. When state-of-the-art models are retrained on CleanSplit, their benchmark performance drops substantially, revealing that previous high scores were largely driven by data memorization rather than true understanding of protein-ligand interactions. Our findings underscore the critical importance of proper dataset curation for developing binding affinity prediction models with robust generalization capabilities.

The Data Leakage Problem in Affinity Prediction

Structure-based drug design (SBDD) aims to develop small-molecule drugs that bind with high affinity to specific protein targets. While deep neural networks have revolutionized computational drug design, their real-world performance has consistently fallen short of benchmark expectations [12]. The root cause of this discrepancy lies in fundamental flaws in dataset organization and evaluation protocols.

The standard practice of training models on the PDBbind database and evaluating them on CASF benchmarks has created an inflated perception of model performance [12] [4]. Analysis reveals that nearly 49% of all CASF complexes have exceptionally similar counterparts in the PDBbind training set, sharing nearly identical ligand and protein structures, comparable ligand positioning within protein pockets, and closely matched affinity labels [12] [4]. This structural similarity enables accurate prediction of test labels through simple memorization rather than genuine learning of interaction principles.

Alarmingly, some models perform comparably well on CASF datasets even after omitting all protein or ligand information from their input data, suggesting their predictions are not based on understanding protein-ligand interactions [12] [4]. This problem is compounded by significant redundancies within the training dataset itself, where approximately 50% of all training complexes belong to similarity clusters, further encouraging memorization over generalization [12].

The CleanSplit Methodology: A Multi-Modal Filtering Approach

Core Algorithm and Similarity Metrics

The PDBbind CleanSplit protocol employs a sophisticated structure-based clustering algorithm that performs combined assessment across three complementary dimensions of similarity. Unlike traditional sequence-based approaches, this multimodal filtering can identify complexes with similar interaction patterns even when proteins have low sequence identity [12] [4].

Table 1: Similarity Metrics Used in CleanSplit Filtering Protocol

Metric	Calculation Method	Assessment Purpose	Filtering Threshold
Protein Similarity	TM-score	Global protein structure similarity	TM-score > 0.7
Ligand Similarity	Tanimoto coefficient	2D chemical structure similarity	Tanimoto > 0.9
Binding Conformation Similarity	Pocket-aligned ligand RMSD	3D ligand positioning in binding pocket	RMSD < 2.0 Å

The algorithm systematically compares all CASF complexes against all PDBbind complexes, identifying train-test pairs that exceed similarity thresholds across these three metrics. This comprehensive approach ensures that complexes with similar interaction patterns are properly identified and removed, even when they involve proteins with low sequence identity [12].

Filtering Protocol Implementation

The CleanSplit filtering process involves two critical phases that address both external and internal dataset issues:

Phase 1: Train-Test Separation

• Exclusion of all training complexes that closely resemble any CASF test complex based on combined similarity metrics
• Removal of all training complexes with ligands identical to those in CASF test complexes (Tanimoto > 0.9)
• Provides additional safeguard against ligand-based data leakage, addressing research showing that GNNs often rely on ligand memorization [12] [4]

Phase 2: Internal Redundancy Reduction

• Identification and resolution of similarity clusters within the training dataset
• Iterative removal of complexes until all striking similarity clusters are resolved
• Uses adapted filtering thresholds to balance dataset size minimization and diversity maximization [12]

This two-phase approach resulted in the removal of approximately 4% of training complexes due to train-test leakage and an additional 7.8% due to internal redundancies, ultimately producing a more diverse and robust training dataset [12] [4].

Diagram 1: CleanSplit filtering workflow showing the multi-stage process for creating leakage-free datasets.

Experimental Validation and Performance Impact

Quantifying the Data Leakage Effect

To illustrate the profound impact of data leakage on model performance, researchers devised a simple search algorithm that predicts the affinity of each CASF test complex by identifying the five most similar training complexes and averaging their affinity labels [12] [4]. Despite its simplicity, this algorithm achieved competitive CASF2016 prediction performance (Pearson R = 0.716) compared with published deep-learning-based scoring functions, demonstrating that sophisticated models were essentially replicating this nearest-neighbor approach through memorization [12].

The scale of data leakage was quantitatively established through systematic analysis, which identified nearly 600 high-similarity pairs between PDBbind training and CASF complexes [12] [4]. After applying the CleanSplit filtering protocol, the remaining train-test pairs with highest similarity exhibited clear structural differences, confirming the effectiveness of the filtering approach [12].

Model Performance on CleanSplit Versus Standard Splits

Retraining experiments with state-of-the-art binding affinity prediction models revealed dramatic performance differences when evaluated on CleanSplit versus standard dataset splits:

Table 2: Performance Comparison on Standard vs. CleanSplit Datasets

Model	Architecture Type	Performance on Standard Split	Performance on CleanSplit	Performance Change
GenScore [18]	Graph Neural Network	High benchmark performance	Substantially dropped performance	Significant decrease
Pafnucy [4]	Convolutional Neural Network	High benchmark performance	Substantially dropped performance	Significant decrease
GEMS (New Model)	Graph Neural Network with Transfer Learning	Not applicable	Maintained high performance	State-of-the-art

The substantial performance drop observed in existing models when trained on CleanSplit confirms that their previously reported high scores were largely driven by data leakage rather than genuine generalization capability [12] [4]. In contrast, the newly developed GEMS model maintained high benchmark performance when trained on CleanSplit, demonstrating robust generalization to strictly independent test datasets [12].

Diagram 2: Performance comparison of models trained on standard datasets versus CleanSplit, showing decreased performance for existing models but maintained performance for GEMS.

The GEMS Model: Architecture for Generalization

Technical Innovations

To address the generalization shortcomings exposed by CleanSplit, researchers developed the Graph neural network for Efficient Molecular Scoring (GEMS) model, which incorporates several key innovations [12] [4]:

Sparse Graph Modeling: GEMS represents protein-ligand interactions using a sparse graph structure that efficiently captures relevant atomic interactions without unnecessary computational overhead.

Transfer Learning from Language Models: The model leverages knowledge transferred from large language models, enabling it to incorporate broader chemical and biological context.

Ablation-Resistant Design: Experimental ablation studies demonstrated that GEMS fails to produce accurate predictions when protein nodes are omitted from the graph, confirming that its predictions are based on genuine understanding of protein-ligand interactions rather than dataset biases [12].

Integration with Generative AI Workflows

GEMS addresses a critical bottleneck in modern SBDD pipelines. Generative models like RFdiffusion and DiffSBDD can create diverse libraries of new protein-ligand interactions but lack accurate methods to predict binding affinities for these generated complexes [12]. With its robust generalization capabilities validated on strictly independent datasets, GEMS provides the prediction accuracy needed to identify interactions with therapeutic potential from generative model outputs [12] [4].

Implementation and Research Applications

Research Reagent Solutions

Table 3: Essential Research Reagents for CleanSplit Implementation

Resource	Type	Function	Access Information
PDBbind CleanSplit Dataset	Curated training data	Provides leakage-free training dataset for robust model development	Available through Zenodo [19]
Pairwise Similarity Matrices	Precomputed similarity data	Enables quick establishment of leakage-free evaluation setups	Available through Zenodo [19]
GEMS Python Code	Model implementation	Reference implementation of generalization-capable affinity prediction	Publicly available in easy-to-use format [12]
Structure-Based Clustering Algorithm	Filtering algorithm	Identifies and removes structurally similar complexes from datasets	Methodology described in publication [12]

Integration with Existing Workflows

The CleanSplit protocol represents a paradigm shift in how binding affinity prediction models should be trained and evaluated. Researchers can integrate it into existing workflows through several approaches:

Retraining Existing Models: Models like GenScore and Pafnucy can be retrained on CleanSplit to assess their true generalization capabilities and identify architectural limitations [12].

Benchmark Redesign: The CASF benchmarks can now serve as truly external evaluation datasets when models are trained exclusively on CleanSplit, enabling genuine assessment of generalization to unseen protein-ligand complexes [12] [4].

Quality Control for Custom Datasets: The structure-based filtering algorithm can be applied to custom datasets to identify and eliminate similar data leakage issues in proprietary or specialized collections [12].

The PDBbind CleanSplit protocol addresses a fundamental challenge in computational drug design: the inflated performance metrics resulting from data leakage between standard training and testing datasets. By providing a rigorously curated training dataset with minimized redundancy and strict separation from benchmark complexes, CleanSplit enables development of binding affinity prediction models with genuinely generalizable capabilities rather than expertise in dataset memorization.

The substantial performance drop observed in existing models when evaluated on CleanSplit underscores the critical importance of proper dataset curation and the previously overlooked severity of data leakage in this field. Moving forward, CleanSplit sets a new standard for robust training and reliable evaluation in binding affinity prediction, potentially accelerating the development of more effective computational tools for drug discovery.

The field of biomedical machine learning, particularly drug-target affinity (DTA) prediction, faces a critical replication crisis. Models that demonstrate excellent performance during benchmark testing often fail dramatically in real-world applications and independent validations. This discrepancy stems primarily from data leakage and over-optimistic evaluations caused by inappropriate data splitting methodologies [4].

Conventional random splitting of datasets creates test sets dominated by samples with high similarity to the training set. This allows models to achieve inflated performance metrics by exploiting similarity-based shortcuts rather than learning generalizable principles of biomolecular interactions [20]. The consequence is a generalization gap where performance substantially degrades on lower-similarity samples that better represent real-world deployment scenarios [20] [4]. Similarity-Aware Evaluation (SAE) addresses this fundamental flaw by providing a framework for controlled data splitting that systematically minimizes similarity between training and test sets, enabling realistic assessment of model performance on out-of-distribution data.

Theoretical Foundations of Similarity-Aware Evaluation

The Data Leakage Problem in Biomedical Machine Learning

Information leakage occurs when a model inadvertently gains access to information during training that would not be available in real-world inference scenarios. In biomedical contexts, this often manifests as similarity-induced leakage, where test samples share significant structural or sequential similarity with training samples [21].

Recent studies have quantified this problem across multiple domains. In drug-target affinity prediction, performance on standard benchmarks can be misleading because "the canonical randomized split of a test set in conventional evaluation leaves the test set dominated by samples with high similarity to the training set" [20]. For protein-protein interaction prediction, models that perform excellently on random splits often show "performance often becomes close to random when evaluated on protein pairs with low homology to the training data" [21]. Similar issues pervade binding affinity prediction, where "train–test data leakage between the PDBbind database and the Comparative Assessment of Scoring Function benchmark datasets has severely inflated the performance metrics" of deep-learning models [4].

Formal Problem Definition

The core challenge addressed by SAE can be formalized as a constrained optimization problem. For a dataset (\mathcal{M}={(x1,y1),\ldots,(xn,yn)}) of n samples with feature vectors (xi \in X) and labels (yi \in Y), the goal is to split (\mathcal{M}) into training ((\mathcal{M}{train})), validation ((\mathcal{M}{val})), and test ((\mathcal{M}_{test})) sets such that:

• Similarity between samples across different splits is minimized
• Statistical properties (e.g., class distributions) are preserved within each split
• The test set represents the intended out-of-distribution use case [21]

This problem is particularly complex for biomolecular data exhibiting intricate dependency structures. DataSAIL formalizes this as the (k, R, C)-DataSAIL problem, which involves splitting an R-dimensional dataset into k folds while minimizing inter-class similarity and preserving the distribution of C classes across folds [21].

Implementation Frameworks for SAE

DataSAIL: A Combinatorial Optimization Approach

DataSAIL implements SAE through a scalable heuristic based on clustering and integer linear programming (ILP). The framework formulates similarity-aware data splitting as a combinatorial optimization problem and provides practical solutions despite its NP-hard nature [21].

The methodology supports both one-dimensional and two-dimensional datasets:

• One-dimensional data: Each sample (xi, yi) corresponds to one elementary data point (e.g., predicting molecular properties for single compounds)
• Two-dimensional data: Each sample consists of two elementary data points (e.g., drug-target pairs for interaction prediction) [21]

DataSAIL provides multiple splitting strategies categorized by whether they account for similarity and dataset dimensionality, including identity-based (I1, I2) and similarity-based (S1, S2) splitting tasks [21].

Optimization-Based Splitting Methodologies

Alternative implementations frame the splitting problem as direct optimization. Recent work proposes "a formulation of optimization problems which are approximately and efficiently solved by gradient descent" to create splits that adapt to any desired similarity distribution [20].

This approach enables researchers to define custom similarity thresholds and distributions for their test sets, providing flexibility to simulate various real-world scenarios where models encounter data with specific similarity relationships to training examples.

Structural Filtering Algorithms

For structure-based affinity prediction, specialized filtering algorithms have been developed to address data leakage. These methods use multimodal similarity assessment combining:

• Protein similarity (TM scores)
• Ligand similarity (Tanimoto scores)
• Binding conformation similarity (pocket-aligned ligand RMSD) [4]

This comprehensive approach identifies and removes complexes with high structural similarity across splits, ensuring that test complexes present genuinely novel challenges rather than variations of training examples.

Experimental Protocols and Methodologies

Quantitative Similarity Metrics for Biomolecular Data

Table 1: Similarity Metrics for SAE in Drug-Target Affinity Prediction

Entity Type	Similarity Metric	Calculation Method	Application Context
Proteins	TM-score	Template Modeling score for structural alignment	Binding affinity prediction [4]
Protein Sequences	Sequence Identity	Percentage of identical residues in alignment	Protein-protein interaction prediction [21]
Small Molecules	Tanimoto Coefficient	Fingerprint-based similarity calculation	Drug-target interaction [4]
Binding Conformations	RMSD	Root-mean-square deviation of atomic positions	Structure-based affinity prediction [4]
Complex Structures	Multimodal Similarity	Combined protein, ligand, and conformation metrics	Comprehensive leakage prevention [4]

Data Splitting Protocols

Table 2: SAE Splitting Strategies for Different Data Types

Splitting Type	Dataset Dimensionality	Similarity Consideration	Key Applications
Random (R)	1D or 2D	None	Baseline comparison [21]
Identity-based (I1)	1D	Identity of samples	Single-molecule property prediction [21]
Identity-based (I2)	2D	Identity of both entities	Drug-target interaction with no overlap [21]
Similarity-based (S1)	1D	Similarity between samples	Protein function prediction [21]
Similarity-based (S2)	2D	Similarity along both dimensions	Cold-start drug-target affinity [21]

Implementation Workflow

The following diagram illustrates the complete SAE workflow for creating similarity-aware splits:

Impact on Model Performance and Generalization

Quantifying the Generalization Gap

SAE reveals substantial performance gaps between standard and similarity-aware evaluations. Studies retraining state-of-the-art binding affinity prediction models on properly split data show "their performance dropped markedly when trained on PDBbind CleanSplit, confirming that the previous high scores were largely driven by data leakage" [4].

Table 3: Performance Comparison Between Standard and SAE Splits

Model	Dataset	Standard Split CI	SAE Split CI	Performance Drop	Reference
GenScore	PDBbind	0.836 (reported)	0.723 (CleanSplit)	13.5%	[4]
Pafnucy	PDBbind	0.815 (reported)	0.698 (CleanSplit)	14.4%	[4]
DeepDTA	KIBA	0.893 (random)	0.827 (similarity-aware)	7.4%	[20]
GraphDTA	Davis	0.885 (random)	0.812 (similarity-aware)	8.2%	[20]

Case Study: PDBbind CleanSplit

The PDBbind CleanSplit initiative demonstrates the profound impact of proper data splitting. Analysis revealed that "nearly 600 such similarities were detected between PDBbind training and CASF complexes, involving 49% of all CASF complexes" [4]. This extensive leakage meant nearly half the test complexes didn't present novel challenges to trained models.

After filtering using structural similarity thresholds, the retrained models showed significantly reduced but more realistic performance, confirming that "the previous high scores were largely driven by data leakage" [4]. This case highlights how SAE provides more reliable estimates of real-world model performance.

The Scientist's Toolkit: Research Reagent Solutions

Table 4: Essential Tools and Algorithms for Similarity-Aware Evaluation

Tool/Algorithm	Function	Application Context	Implementation
DataSAIL	Similarity-aware data splitting	General biomolecular data	Python package [21]
Structural Clustering Algorithm	Multimodal complex similarity	Structure-based affinity prediction	Custom implementation [4]
Gradient Descent Optimizer	Custom distribution splitting	Drug-target affinity	Framework-specific [20]
FetterGrad Algorithm	Gradient conflict mitigation	Multitask learning for DTA	DeepDTAGen framework [22]
TM-score	Protein structural similarity	Protein-ligand complexes	Standalone tool [4]
Tanimoto Coefficient	Ligand similarity	Small molecule comparison	Standard cheminformatics [4]

Advanced Applications and Future Directions

Integration with Multitask Learning Frameworks

SAE principles are being integrated into next-generation drug discovery pipelines. The DeepDTAGen framework demonstrates how "a multitask deep learning framework for drug-target affinity prediction and target-aware drugs generation" can benefit from proper evaluation methodologies [22]. Such frameworks face additional complexity from "optimization challenges such as conflicting gradients" between tasks, which can be addressed by specialized algorithms like FetterGrad that "keep the gradients of both tasks aligned while learning from a shared feature space" [22].

Toward Standardized Benchmarking Practices

The field is moving toward standardized SAE practices to enable meaningful comparison across studies. This includes:

• Strict separation protocols between training and benchmark datasets
• Similarity threshold definitions for different biomolecular data types
• Automated leakage detection in existing benchmarks
• Domain-specific splitting strategies for particular application contexts

The following diagram illustrates the relationship between different splitting strategies and their impact on model generalization:

Similarity-Aware Evaluation represents a paradigm shift in how we develop and validate machine learning models for biomedical applications. By systematically controlling data splits to minimize similarity-induced leakage, SAE provides realistic performance estimates that truly reflect a model's ability to generalize to novel examples. The framework addresses a critical need in computational drug discovery, where overoptimistic evaluations have led to inflated expectations and failed translations.

As the field progresses, SAE methodologies will likely become standard practice, enabling more reliable model development and accelerating the creation of genuinely predictive tools for drug discovery. The tools and protocols outlined in this guide provide researchers with practical approaches for implementing similarity-aware evaluation in their own work, ultimately contributing to more robust and generalizable biomedical machine learning.

Accurate prediction of binding affinity changes caused by protein mutations is vital for drug design and interpreting drug resistance mechanisms. However, the field of machine learning (ML) and deep learning (DL) for drug discovery faces a significant crisis of generalization. A pervasive issue of train-test data leakage between standard training databases like PDBbind and common benchmark datasets has severely inflated the performance metrics of many published models, creating an overoptimistic impression of their generalization capabilities [4] [5]. When models are evaluated on truly independent data, their performance often drops substantially, revealing that many existing approaches rely on memorizing structural similarities rather than learning fundamental protein-ligand interaction principles [4].

Conventional random data partitioning of protein-ligand interaction datasets often produces spuriously high correlations that misrepresent real-world performance. Studies demonstrate that while models may achieve high predictive correlations (e.g., Pearson coefficients up to 0.70) under random partitioning, their performance declines significantly with more rigorous UniProt-based partitioning that preserves data independence [17]. This performance gap highlights how conventional evaluation methods potentially overestimate model accuracy and fail to predict real-world performance on novel protein targets.

Within this context of addressing data bias, advanced partitioning strategies like the anchor-query framework have emerged as promising solutions. These approaches explicitly structure learning to leverage limited reference data to improve predictive generalization for unknown query states, offering a more robust foundation for mutation studies in computational drug discovery [17].

Anchor-Query Partitioning: Conceptual Framework and Mechanism

Core Theoretical Principles

The anchor-query partitioning framework represents a paradigm shift in how training data is structured for mutation effect prediction. Unlike conventional random splitting, this approach explicitly separates the learning process into anchor states (known reference points) and query states (unknown predictions). The fundamental principle involves using known states as fixed anchor points for predicting unknown query states, creating a relational learning system that mimics how researchers might approach the problem conceptually [17].

This framework functions through a pairwise learning strategy where the model learns relationships between protein states rather than absolute properties. By leveraging a limited set of well-characterized reference mutations as anchors, the model can make predictions about novel mutations by inferring their behavior relative to these established anchors. This approach is particularly valuable for predicting mutation-induced changes in binding free energy, where the relative difference between wild-type and mutant proteins is more meaningful and predictable than absolute energy values [17].

Comparative Advantages Over Conventional Partitioning

Table 1: Comparison of Data Partitioning Strategies for Mutation Studies

Partitioning Strategy	Key Characteristics	Performance on Independent Data	Risk of Data Leakage	Suitable Applications
Random Partitioning	Splits data randomly without considering protein relationships	Often spuriously high but inflates performance estimates [17]	High - similar proteins can appear in both sets [4]	Initial model prototyping, non-generalizable applications
UniProt-Based Partitioning	Ensures no protein overlaps between training and test sets	Reduced performance but more realistic generalization assessment [17]	Low - maintains protein-level independence	Benchmarking true model generalization capabilities
Anchor-Query Framework	Uses known references (anchors) to predict unknown queries (novel mutations)	Enhanced generalization even with limited reference data [17]	Minimal - explicitly designed for novel prediction	Predicting effects of novel mutations, drug resistance studies

The anchor-query framework addresses fundamental limitations of both random and UniProt-based partitioning. While UniProt-based splitting reduces data leakage, it often lacks high prediction accuracy for truly novel targets. The anchor-query approach maintains independence while improving accuracy by structuring the learning problem to explicitly handle the prediction of novel states based on limited references [17].

Experimental validation across three biological systems revealed that even a small amount of carefully selected reference data can significantly enhance prediction accuracy within this framework. This suggests that the strategic selection and use of anchor points allows for more precise interpolation to unknown query states than models trained to make absolute predictions without this relational structure [17].

Implementation Methodologies and Experimental Protocols

Data Preparation and Feature Engineering

Successful implementation of anchor-query frameworks begins with comprehensive data preparation. For mutation studies, this involves compiling a dataset of protein-ligand complexes with experimentally determined binding free energies for both wild-type and mutant variants. The MdrDB database has been used for such studies, providing a foundation for evaluating partitioning strategies [17].

Protein sequences should be embedded using modern protein language models such as ESM-2, which provides contextualized representations of amino acid sequences. These embeddings effectively integrate features of both wild-type and mutant proteins, capturing structural and functional information relevant to binding affinity changes. The embedding process converts protein sequences into numerical representations that preserve evolutionary and structural relationships essential for the anchor-query framework [17].

The critical step in data preparation is the strategic division of available data into anchor and query sets. Anchors should represent diverse structural and functional contexts while maintaining relevance to the query mutations. This selection can be guided by clustering techniques based on protein similarity, functional classification, or structural properties to ensure anchor diversity and relevance.

Model Architecture and Training Protocols

Table 2: Experimental Components for Anchor-Query Framework Implementation

Component Category	Specific Tools/Methods	Function in Experiment	Key Parameters
Protein Representation	ESM-2 Protein Language Model	Converts protein sequences into numerical embeddings that capture structural and evolutionary information [17]	Embedding dimensions, layer selection, pooling strategy
Machine Learning Frameworks	Scikit-learn, PyTorch, TensorFlow	Provides implementations of ML/DL models for the prediction task [17]	Varies by specific algorithm
Similarity Assessment	TM-score, Tanimoto coefficients, RMSD	Quantifies structural and chemical similarities between complexes for filtering and analysis [4]	Threshold settings for similarity definitions
Data Filtering	Structure-based clustering algorithm	Identifies and removes overly similar complexes to prevent data leakage [4]	Similarity thresholds, iterative removal parameters
Evaluation Metrics	Pearson correlation, RMSE, Concordance Index	Quantifies prediction accuracy and model performance [17] [22]	Statistical significance testing

Six distinct ML/DL models have been evaluated in anchor-query frameworks, ranging from traditional machine learning algorithms to sophisticated deep learning architectures. The pairwise learning approach is implemented by structuring the input data to represent relationships between anchor-query pairs rather than individual samples [17].

Training involves minimizing a loss function that measures the discrepancy between predicted and actual differences in binding free energy between query and anchor states. The training protocol should include rigorous validation using cross-validation strategies that maintain the anchor-query separation to properly assess generalization performance [17].

Workflow Visualization

Anchor-Query Workflow: The end-to-end process for implementing anchor-query partitioning in mutation studies.

Quantitative Performance Assessment

Comparative Performance Across Partitioning Strategies

Table 3: Performance Comparison of Partitioning Strategies on Protein Mutation Data

Evaluation Metric	Random Partitioning	UniProt-Based Partitioning	Anchor-Query Framework	Notes on Significance
Pearson Correlation	Up to 0.70 [17]	Significant decline compared to random [17]	Improved generalization over UniProt-based [17]	Anchor-query provides better balance of performance and generalization
Root Mean Square Error (RMSE)	Not reported in sources	Not reported in sources	Significantly enhanced with reference data [17]	Even small reference data improvements were substantial
Generalization Gap	Large (overestimation) [17]	Reduced but with accuracy trade-off	Minimized while maintaining accuracy [17]	Most important advantage for real-world applications
Dependence on Data Leakage	High performance depends on leakage [4]	Low - minimal dependence	Very low - explicitly designed for independence	Retraining models on clean data shows anchor-query robustness

Empirical evaluations demonstrate that the anchor-query framework achieves a superior balance between prediction accuracy and generalization capability. While models trained with random partitioning show deceptively high performance (Pearson coefficients up to 0.70), this performance substantially declines under proper independent evaluation [17]. In contrast, the anchor-query approach maintains more stable performance across different evaluation scenarios, particularly for predicting mutation-induced changes in binding free energy.

The performance advantage of anchor-query frameworks becomes particularly evident in challenging prediction scenarios such as drug resistance mutations, where the model must extrapolate to novel mutational patterns not present in the training data. The relational learning approach enables more robust prediction for these novel variants by leveraging similarities to characterized anchor mutations [17].

Integration with Broader Data Bias Mitigation Strategies

Complementary Relationship with Data Cleaning Methods

The anchor-query framework does not operate in isolation but complements other data bias mitigation strategies. A significant advancement in addressing data leakage is the PDBbind CleanSplit dataset, curated using a novel structure-based filtering algorithm that eliminates train-test data leakage as well as redundancies within the training set [4]. This approach uses a combined assessment of protein similarity (TM-scores), ligand similarity (Tanimoto scores), and binding conformation similarity (pocket-aligned ligand RMSD) to identify and remove overly similar complexes [4].

When state-of-the-art binding affinity prediction models like GenScore and Pafnucy were retrained on the PDBbind CleanSplit dataset, their benchmark performance dropped substantially, confirming that their previous high performance was largely driven by data leakage rather than genuine understanding of protein-ligand interactions [4]. This underscores the critical importance of proper dataset partitioning and bias mitigation as a foundation for reliable model development.

Synergy with Advanced Model Architectures

The anchor-query framework shows particular promise when combined with modern neural network architectures designed for robust generalization. Graph neural networks (GNNs) that leverage sparse graph modeling of protein-ligand interactions and transfer learning from language models have demonstrated maintained high benchmark performance even when trained on properly cleaned datasets [4].

These architectures appear naturally compatible with the anchor-query approach, as both emphasize learning fundamental interaction principles rather than memorizing specific complex structures. The integration of these technologies—properly partitioned data, bias-aware model architectures, and structured learning frameworks like anchor-query—represents the most promising path toward developing binding affinity prediction models that maintain accuracy in real-world drug discovery applications [17] [4].

Research Reagent Solutions for Implementation

Table 4: Essential Research Reagents and Computational Tools

Reagent/Tool	Type	Primary Function	Application Notes
ESM-2 Protein Language Model	Computational	Generates contextualized protein sequence embeddings [17]	Pre-trained models available; fine-tuning possible for specific domains
PDBbind Database	Data Resource	Provides curated protein-ligand complexes with binding affinity data [4]	General version suffers from data leakage; CleanSplit version recommended
MdrDB Database	Data Resource	Specialized database for mutation-induced binding free energy changes [17]	Used in original anchor-query framework validation
Structure-Based Filtering Algorithm	Computational Method	Identifies and removes overly similar complexes to prevent data leakage [4]	Uses TM-score, Tanimoto, and RMSD metrics for comprehensive similarity assessment
Graph Neural Network (GNN) Architectures	Computational Model	Models protein-ligand interactions as sparse graphs for improved generalization [4]	Particularly effective when combined with anchor-query approaches

The development and validation of advanced partitioning strategies like the anchor-query framework represent a crucial step toward addressing the pervasive problem of data bias and generalization in affinity prediction models. By explicitly structuring the learning process to leverage limited reference data for predicting novel queries, this approach provides a more robust foundation for mutation studies in drug discovery.

The integration of anchor-query frameworks with complementary advances in data cleaning methods like PDBbind CleanSplit and specialized model architectures like graph neural networks creates a powerful toolkit for developing predictive models that maintain accuracy in real-world scenarios. As these methodologies continue to mature and see broader adoption, they hold significant promise for improving the efficiency and success rates of computational drug discovery, particularly for addressing challenges like drug resistance mutations and polypharmacology.

Future research directions should focus on optimizing anchor selection strategies, developing specialized model architectures explicitly designed for pairwise anchor-query learning, and extending the framework to predict additional molecular properties beyond binding affinity. As the field moves toward these more rigorous evaluation and training paradigms, we can anticipate substantial improvements in the real-world applicability of computational models for drug discovery.

Accurate prediction of protein-ligand binding affinity is a cornerstone of computational drug design. However, the field faces a significant reproducibility crisis, where models demonstrating exceptional benchmark performance fail to generalize to truly novel targets. Recent research has revealed that this discrepancy stems primarily from train-test data leakage and dataset redundancies that severely inflate performance metrics [4].

The core issue lies in the standard practice of training models on the PDBbind database and evaluating them on the Comparative Assessment of Scoring Functions (CASF) benchmark. Studies have found a high degree of structural similarity between these datasets, allowing models to perform well through memorization rather than genuine understanding of protein-ligand interactions. Alarmingly, some models maintain competitive benchmark performance even when critical protein or ligand information is omitted from their inputs [4]. This indicates that the reported performance of many existing models is artificially inflated, creating an over-optimistic view of their generalization capabilities and ultimately hindering progress in structure-based drug design (SBDD) [4] [5].

This whitepaper provides a technical guide for implementing a robust, structure-based multimodal filtering algorithm designed to resolve these data bias issues. By creating rigorously independent training and test splits, researchers can build and evaluate affinity prediction models with truly reliable generalization capabilities.

The Foundation: Key Similarity Metrics for Multimodal Comparison

Effective multimodal filtering requires a combined assessment of similarity across three distinct structural dimensions: the protein, the ligand, and their binding conformation. Relying on a single metric, such as sequence identity, is insufficient to identify complexes with similar interaction patterns.

Table 1: Core Similarity Metrics for Multimodal Filtering

Modality	Metric	Technical Description	Interpretation
Protein Structure	Template Modeling Score (TM-score) [4]	Measures protein structural similarity, ranging from 0 to 1.	A score > 0.5 indicates generally the same protein fold. Less sensitive to local variations than RMSD.
Ligand Chemistry	Tanimoto Coefficient (TC) [4] [23]	Calculates chemical similarity based on molecular fingerprints (e.g., 1024-bit fingerprints via OpenBabel).	Ranges from 0 (no similarity) to 1 (identical fingerprints). A threshold of >0.9 often indicates near-identical ligands [4].
Binding Conformation	Root-Mean-Square Deviation (RMSD) [4] [23]	Standard measure of the average distance between atoms in superimposed ligand structures.	Ligand-size dependent. Lower values indicate higher conformational similarity (e.g., <2 Å is considered a successful pose prediction).
Binding Conformation	Contact Mode Score (CMS) [23] [24]	Assesses similarity based on intermolecular protein-ligand contacts rather than Cartesian coordinates.	Less dependent on ligand size than RMSD. Better captures biologically meaningful binding features.

The Contact Mode Score (CMS) is a particularly valuable alternative to RMSD. Whereas RMSD is purely geometric and ligand-size dependent, CMS compares the sets of interatomic contacts formed by a ligand and its receptor. This provides a more biologically relevant assessment of whether two binding modes engage the protein pocket in a similar way [23] [24]. For comparing complexes involving different proteins and non-identical ligands, the eXtended Contact Mode Score (XCMS) provides a template-based method for effective comparison [23] [24].

A Protocol for Implementing the Multimodal Filtering Algorithm

The following section details a step-by-step protocol for implementing the multimodal filtering algorithm, culminating in the creation of a rigorously curated dataset like PDBbind CleanSplit [4].

Algorithm Workflow and Logic

The diagram below illustrates the logical workflow and decision process of the filtering algorithm.

Step-by-Step Experimental Protocol

• Data Preparation: Begin with the comprehensive PDBbind database (e.g., the general set) and your chosen benchmark set (e.g., CASF-2016).
• All-vs-All Comparison: Perform a pairwise comparison between every complex in the training set (PDBbind) and every complex in the test set (CASF). This computationally intensive step is essential for identifying all potential leakages.
• Apply Multimodal Thresholds: For each train-test pair, calculate the three similarity metrics and apply the following filtering logic, as visualized in the workflow above:
  • Protein Similarity: Compute the TM-score between the two protein structures. A threshold of TM-score > 0.5 is used to identify proteins that share the same overall fold [4].
  • Ligand Similarity: For pairs passing the protein filter, compute the Tanimoto coefficient based on molecular fingerprints. A threshold of Tanimoto > 0.9 identifies nearly identical ligands, preventing ligand-based memorization [4].
  • Conformation Similarity: For pairs passing both previous filters, structurally align the protein binding pockets and calculate the pocket-aligned ligand RMSD. A threshold of RMSD < 2.0 Å identifies ligands binding in nearly identical poses [4].
  • Any training complex that meets all three criteria against a test complex is flagged for removal due to an unacceptably high risk of data leakage.
• Remove Redundant Training Data: After addressing train-test leakage, apply a similar clustering approach within the training set itself to reduce internal redundancy. Using adapted thresholds (e.g., slightly less stringent), iteratively identify and remove complexes from similarity clusters until all major clusters are resolved. This forces models to learn generalizable rules rather than relying on memorization of similar complexes [4].
• Generate the Final Dataset: The remaining training complexes, stripped of both test look-alikes and major internal redundancies, constitute your final filtered dataset (e.g., PDDBind CleanSplit). This dataset provides a robust foundation for training models whose benchmark performance will reflect true generalization ability.

Quantitative Impact: Validation and Benchmark Results

The effect of implementing multimodal filtering is dramatic and quantifiable. Retraining existing state-of-the-art models on a properly filtered dataset provides a definitive test of their true generalization capability.

Table 2: Performance Impact of Training on a Filtered Dataset (PDBbind CleanSplit)

Model / Benchmark	Performance on CASF-2016 (Trained on Standard PDBbind)	Performance on CASF-2016 (Trained on PDBbind CleanSplit)	Implied Generalization Capability
GenScore [4]	High Benchmark Performance (e.g., Low RMSE, High Pearson R)	Substantial Performance Drop	Previously reported performance was largely driven by data leakage.
Pafnucy [4]	High Benchmark Performance (e.g., Low RMSE, High Pearson R)	Substantial Performance Drop	Previously reported performance was largely driven by data leakage.
GEMS (Graph Neural Network) [4]	Not Applicable	Maintains High Benchmark Performance	Demonstrates genuine generalization to unseen complexes, as performance is not based on exploiting leakage.

The data in Table 2 underscores a critical point: the high performance of many published models on common benchmarks is a mirage created by data leakage. When this leakage is removed via multimodal filtering, their performance drops markedly [4]. This validates the filtering algorithm's effectiveness in creating a more meaningful evaluation benchmark.

To further illustrate the extent of data leakage, a simple search algorithm that predicts test affinity by averaging the labels of the five most similar training complexes can achieve a competitive Pearson R of 0.716 on the CASF-2016 benchmark, performing comparably to some deep-learning scoring functions [4]. After applying the multimodal filter, the most similar remaining train-test pairs exhibit clear structural differences, confirming the elimination of problematic similarities [4].

Table 3: Key Research Reagents and Computational Tools for Implementation

Item / Resource	Function / Purpose	Example Sources / Implementation
PDBbind Database	A comprehensive database of protein-ligand complexes with experimentally measured binding affinities. Serves as the primary source for training data.	http://www.pdbbind.org.cn/ [4]
CASF Benchmark	The Comparative Assessment of Scoring Functions benchmark, used for evaluating the generalization capability of trained models.	Distributed with PDBbind [4]
US-align / TM-align	Open-source algorithms for calculating the TM-score, used for protein structure comparison.	https://zhanggroup.org/US-align/ [4]
OpenBabel	A chemical toolbox used for handling chemical data, including the calculation of molecular fingerprints (e.g., for Tanimoto coefficients).	http://openbabel.org/ [23]
Contact Mode Score (CMS)	A tool for calculating the CMS and XCMS scores, providing an alternative, biologically meaningful measure of binding conformation similarity.	http://brylinski.cct.lsu.edu/content/contact-mode-score [23] [24]
Graph Neural Network (GNN) Model	A deep learning architecture capable of learning robust representations of protein-ligand interactions, leading to better generalization on filtered data.	e.g., GEMS model [4]

The implementation of rigorous, structure-based multimodal filtering is no longer an optional refinement but a necessary step for ensuring the validity and generalizability of binding affinity prediction models. By systematically eliminating data leakage and reducing dataset redundancy, researchers can build models that genuinely understand protein-ligand interactions rather than merely memorizing training examples.

The PDBbind CleanSplit dataset, generated through the methodology described in this guide, provides a new foundation for model development and evaluation in computational drug design [4]. The application of this filtering principle is also crucial for validating the next generation of generative AI models in SBDD, such as RFdiffusion and DiffSBDD, which create novel protein-ligand interactions but require accurate scoring functions to identify high-affinity complexes [4]. Adopting these stringent data curation practices is essential for bridging the gap between impressive benchmark metrics and real-world utility in drug discovery.

The field of computational drug design relies on accurate scoring functions to predict protein-ligand binding affinities. However, a fundamental challenge has undermined the real-world applicability of many models: data bias. Recent research has exposed a "data leakage crisis" wherein models achieve inflated benchmark performance not by learning generalizable principles, but by exploiting structural redundancies between training and test sets [11]. This leakage, combined with inherent dataset imbalances, leads to models that fail to generalize to novel protein-ligand complexes, creating significant barriers to reliable drug discovery [12].

This guide addresses two complementary frameworks for combating these issues. The CleanSplit methodology provides a rigorous, structure-based approach to dataset splitting that eliminates data leakage and ensures meaningful evaluation [12]. Meanwhile, Sparse Autoencoders (SAEs) offer a pathway to more interpretable and robust feature representations, enabling researchers to understand and control what their models are truly learning [25]. When applied together, these techniques form a powerful foundation for building more generalizable and trustworthy affinity prediction models.

Understanding and Implementing CleanSplit

The Data Leakage Problem

Traditional random splitting of protein-ligand datasets often fails to separate structurally similar complexes, creating an illusion of high performance through memorization rather than genuine learning. One groundbreaking analysis revealed that nearly 600 structural similarities existed between the standard PDBbind training set and the Comparative Assessment of Scoring Functions (CASF) benchmark complexes, affecting 49% of all test complexes [12]. This meant nearly half the test set presented no new challenges to trained models.

Table 1: Quantitative Analysis of Data Leakage in PDBbind-CASF

Metric	Before CleanSplit	After CleanSplit
Similar train-test pairs	~600	Minimal structural similarities
CASF complexes affected	49%	True external evaluation
Training complexes removed	N/A	4% due to test similarity + 7.8% due to internal redundancy

The CleanSplit Methodology

The CleanSplit algorithm addresses data leakage through a multi-modal filtering approach that assesses complexes across three dimensions: protein similarity, ligand similarity, and binding conformation similarity [12]. The algorithm employs specific similarity metrics and thresholds to ensure comprehensive filtering:

Table 2: CleanSplit Similarity Metrics and Thresholds

Dimension	Similarity Metric	Threshold for Exclusion
Protein similarity	TM-score	> 0.7
Ligand similarity	Tanimoto coefficient	> 0.9
Binding conformation	Pocket-aligned ligand RMSD	< 2.0 Å

The implementation involves a structured, iterative process that can be adapted to any protein-ligand dataset:

Step-by-Step Protocol:

• Multi-modal Clustering: Compute all pairwise similarities using:

  • TM-score for protein structure similarity (threshold > 0.7 indicates similar folds)
  • Tanimoto coefficient for ligand similarity (threshold > 0.9 indicates nearly identical compounds)
  • Pocket-aligned ligand RMSD for binding mode similarity (threshold < 2.0 Å indicates similar binding conformations)

• Train-Test Separation: Identify and remove all training complexes that exceed similarity thresholds with any test complex. This step typically removes approximately 4% of training data but is crucial for eliminating leakage [12].

• Internal Redundancy Reduction: Apply adapted thresholds to identify and resolve similarity clusters within the training data itself. This iterative process typically removes an additional 7.8% of complexes that enable "shortcut learning" through memorization [12].

• Validation: Verify the final split by confirming that the most similar train-test pairs now exhibit clear structural differences in both protein folds and ligand positioning.

Research Reagent Solutions

Table 3: Essential Tools for CleanSplit Implementation

Tool/Resource	Function	Application Notes
PDBbind Database	Source of experimental structures and affinities	General set (~20k complexes) provides foundation for curation
CASF Benchmark	Standardized test sets	Use 2016 or later versions; apply CleanSplit to prevent leakage
TM-align Algorithm	Protein structure comparison	Calculate TM-scores for all protein pairs
RDKit	Cheminformatics toolkit	Compute Tanimoto coefficients and ligand descriptors
MDTraj	Molecular dynamics trajectory analysis	Calculate RMSD with optimal alignment
Custom Python Scripts	Multi-modal filtering implementation	Combine metrics for comprehensive similarity assessment

Sparse Autoencoders for Interpretable Features

SAE Fundamentals and Biological Relevance

Sparse Autoencoders (SAEs) are neural network architectures designed to learn compressed, interpretable representations of input data by enforcing sparsity constraints on the latent space. In protein structure prediction, SAEs transform dense, nonlinear representations from models like ESM2-3B into sparse, linear features that can be causally linked to biological concepts [25].

The mathematical objective of an SAE can be summarized as:

• Input: Token embeddings ( x \in \mathbb{R}^d ) from a pre-trained protein language model
• Encoding: Sparse, higher-dimensional latent representation ( z \in \mathbb{R}^n ) where ( d << n )
• Decoding: Reconstruction ( \hat{x} ) minimizing L2 loss ( \mathcal{L} = |x - \hat{x}|^2_2 )
• Sparsity: Enforcement through L1 regularization or TopK activation constraints

Matryoshka SAEs for Hierarchical Protein Features

Proteins exhibit inherent hierarchical organization—from local amino acid patterns to domain-level motifs and full tertiary structures. Standard SAEs often struggle to capture this multi-scale nature, which led to the development of Matryoshka SAEs that learn nested hierarchical representations through embedded feature groups of increasing dimensionality [25].

Implementation Protocol for Protein SAEs:

• Model Setup:

  • Source embeddings from intermediate layers of ESM2-3B (layers 18 and 36 are most informative for structure)
  • Normalize embeddings to enable hyperparameter transfer between layers
  • Use dictionary sizes of 20,480-65,536 features for sufficient biological concept coverage

• Architecture Selection:

  • Matryoshka SAEs: Ideal for capturing protein hierarchy; divide latent dictionary into 3-5 nested groups
  • TopK SAEs: Alternative for fixed sparsity; force exactly K active features per sample
  • L1-SAEs: Traditional approach; use L1 penalty to encourage sparsity

• Training Configuration:

  • Data: 10M sequences randomly selected from UniRef50 (≈2.5B tokens)
  • Batch size: 4096 sequences
  • Learning rate: 4×10⁻⁴ with warmup and cosine decay
  • Sparsity target: L0 < 32 active features for efficient structure prediction

SAE Evaluation and Validation

Table 4: SAE Performance on Downstream Tasks

Evaluation Metric	Original ESM2-3B	SAE (Layer 36)	Performance Preservation
Language Modeling (ΔCE)	Baseline	+0.2-0.5	High
Structure Prediction (RMSD Å)	3.1 ± 2.5	3.2 ± 2.6	96.8%
Contact Map Precision	P@L/2 = 0.75	P@L/2 = 0.72	96%
Biological Concepts (F1 > 0.5)	N/A	233 concepts	48.9% coverage

Biological Concept Discovery Protocol:

• Feature Activation: Compute activations across Swiss-Prot annotated sequences (30+ million amino acid tokens)
• Concept Alignment: Identify features with high F1 scores (> 0.5) for specific biological concepts (e.g., active sites, binding pockets)
• Cross-model Comparison: ESM2-3B SAEs identify 233 concepts (48.9% coverage) versus only 72 concepts (15.1%) for ESM2-8M SAEs, demonstrating the importance of model scale [25]

Integrated Workflow: Combining CleanSplit and SAE

End-to-End Implementation Pipeline

The true power of CleanSplit and SAEs emerges when they are combined into a cohesive workflow for developing generalizable, interpretable affinity prediction models.

Validation Framework

Robust validation is essential when combining these techniques. The integrated framework includes multiple validation checkpoints:

• Data-Level Validation:

  • Confirm CleanSplit effectiveness through similarity analysis of nearest train-test neighbors
  • Verify dataset stratification maintains representation of key protein families and ligand chemotypes

• Representation-Level Validation:

  • Assess SAE feature sparsity (L0 < 32 for efficiency)
  • Evaluate biological concept discovery against Swiss-Prot annotations
  • Measure reconstruction fidelity on downstream tasks (RMSD < 3.5Å for structure prediction)

• Model-Level Validation:

  • Compare performance on CleanSplit test set versus standard benchmarks
  • Conduct feature ablation studies to confirm SAE features capture causal determinants of binding
  • Perform cross-dataset generalization tests on entirely external benchmarks

Research Reagent Solutions for Integrated Pipeline

Table 5: Comprehensive Toolkit for CleanSplit + SAE Implementation

Category	Tool/Resource	Application in Integrated Pipeline
Data Curation	PDBbind CleanSplit	Pre-processed leakage-free dataset
Protein Language Models	ESM2-3B, ESMFold	Source embeddings for SAE training
SAE Implementation	Matryoshka SAE Code	Customizable architecture for hierarchical features
Similarity Metrics	TM-align, RDKit	Multi-modal clustering for CleanSplit
Visualization	SAE Visualizer	Biological concept interpretation
Benchmarking	CASF, PL-REX	External validation with leakage prevention

The integration of CleanSplit methodology and Sparse Autoencoders represents a paradigm shift from model-centric to data-centric and interpretability-aware approaches in affinity prediction. By rigorously addressing data leakage through structure-aware dataset splitting and enabling mechanistic interpretation through sparse, biologically-grounded features, researchers can develop models that genuinely generalize to novel targets and compounds.

The field is rapidly evolving toward even more sophisticated approaches. The Target2035 initiative aims to create massive, high-quality, standardized protein-ligand binding datasets that inherently incorporate these principles [11]. Meanwhile, advances in synthetic data generation with rigorous quality filtering offer pathways to scale without sacrificing generalization. By adopting the practices outlined in this guide—rigorous data splitting, interpretable feature learning, and integrated validation—researchers can contribute to this evolving landscape and build more reliable, trustworthy models for drug discovery.

The era where benchmark performance alone validated models is ending. The future belongs to models that demonstrate both technical proficiency and genuine biological understanding—a future built on the foundations of CleanSplit and interpretable AI.

Optimizing Model Architecture and Training for True Generalization

The field of computational drug design stands at a critical juncture. While deep learning has revolutionized protein-ligand interaction prediction, a pervasive challenge threatens to undermine its progress: the overestimation of model generalization capabilities due to dataset biases and train-test data leakage. Recent research has revealed that the performance metrics of currently available deep-learning-based binding affinity prediction models have been severely inflated by data leakage between the PDBbind database and the Comparative Assessment of Scoring Function (CASF) benchmark datasets [4]. This leakage has led to an overestimation of their true generalization capabilities, creating a significant gap between benchmark performance and real-world applicability. Within this context, architectural innovations—particularly sparse graph neural networks (GNNs)—emerge as a promising pathway toward robust, generalizable affinity prediction models that genuinely understand protein-ligand interactions rather than merely memorizing training data patterns.

The Data Bias Problem: Quantifying Benchmark Inflation

Systematic Analysis of Train-Test Leakage

A rigorous investigation into the structural similarities between PDBbind and CASF benchmarks has uncovered a substantial level of train-test data leakage. Through a novel structure-based clustering algorithm that assesses protein similarity (TM-scores), ligand similarity (Tanimoto scores), and binding conformation similarity (pocket-aligned ligand RMSD), researchers identified nearly 600 significant similarities between PDBbind training and CASF complexes, affecting 49% of all CASF test complexes [4]. These similarities enable models to accurately predict test labels through simple memorization rather than genuine understanding of interaction principles.

The table below summarizes the key findings from the data leakage analysis:

Table 1: Quantified Data Leakage Between PDBbind and CASF Benchmarks

Metric	Before Filtering	After CleanSplit Filtering
Similar train-test pairs	~600	Structurally distinct
CASF complexes affected	49%	0%
Training complexes removed	N/A	4% for test separation + 7.8% for redundancy
Highest similarity after filtering	TM-score > 0.9, Tanimoto > 0.9	Clear structural differences

The PDBbind CleanSplit Solution

To address this fundamental flaw in benchmark evaluation, researchers developed PDBbind CleanSplit, a refined training dataset curated through a structure-based filtering algorithm that eliminates both train-test data leakage and internal training set redundancies [4]. The filtering process employs multimodal criteria to identify and remove complexes that share significant structural similarities with test cases, ensuring that models face genuinely novel challenges during evaluation.

The following DOT language script visualizes the CleanSplit creation workflow:

Sparse Graph Architecture for Protein-Ligand Modeling

GEMS: Graph Neural Network for Efficient Molecular Scoring

The Graph Neural Network for Efficient Molecular Scoring (GEMS) represents a architectural innovation specifically designed to address generalization challenges in binding affinity prediction. GEMS employs a sparse graph modeling approach that represents protein-ligand complexes as heterogeneous graphs with focused interaction edges, avoiding the computational overhead of dense representations while capturing physically meaningful interactions [4].

The core architectural principles of GEMS include:

• Sparse Interaction Modeling: Rather than connecting all protein and ligand atoms within a cutoff distance, GEMS implements a selective edge creation process that prioritizes chemically relevant interactions
• Transfer Learning Integration: The architecture incorporates pre-trained representations from protein and molecular language models (such as ProtBERT and ChemBERTa) to bootstrap feature learning with biophysical and chemical knowledge [2]
• Geometric Aware Message Passing: The message-passing mechanism explicitly accounts for spatial relationships and directional information critical for molecular recognition

Ablation Studies and Interpretability

Critical ablation studies demonstrate that GEMS achieves its performance through genuine understanding of protein-ligand interactions rather than exploiting dataset biases. When protein nodes are omitted from the input graph, the model fails to produce accurate predictions, confirming that its predictions are based on integrated structural information rather than ligand memorization [4]. This represents a significant advancement over previous models that could achieve competitive benchmark performance even when protein information was excluded—a clear indicator of label leakage exploitation.

Experimental Framework and Benchmarking

Retraining Existing Models on CleanSplit

To quantify the impact of data leakage on reported model performance, researchers retrained state-of-the-art binding affinity prediction models (GenScore and Pafnucy) on the PDBbind CleanSplit dataset. The results demonstrated a substantial performance drop for these models when evaluated without data leakage, confirming that their previously reported high performance was largely driven by benchmark contamination rather than genuine generalization capability [4].

The table below compares model performance before and after addressing data leakage:

Table 2: Performance Comparison on CASF Benchmark With and Without Data Leakage

Model	Training Dataset	CASF Performance	Generalization Assessment
GenScore	Original PDBbind	High (Inflated)	Overestimated due to data leakage
GenScore	PDBbind CleanSplit	Substantially reduced	True performance lower than reported
Pafnucy	Original PDBbind	High (Inflated)	Overestimated due to data leakage
Pafnucy	PDBbind CleanSplit	Substantially reduced	True performance lower than reported
GEMS	PDBbind CleanSplit	Maintains high performance	Genuine generalization to unseen complexes

Target Identification Benchmark

Beyond traditional affinity prediction metrics, researchers have developed more demanding benchmarks to assess real-world applicability. The target identification benchmark based on LIT-PCBA evaluates whether models can identify the correct protein target for active molecules—a critical task in drug discovery that requires robust generalization across different binding pockets [26].

Even advanced models like Boltz-2 struggle with this benchmark, indicating that while they may show promising results on traditional affinity prediction tasks, their ability to generalize across diverse protein targets remains limited. This highlights the need for architectural innovations like sparse GNNs that can capture transferable interaction principles.

Implementation Protocols

Structure-Based Filtering Methodology

The algorithmic protocol for creating leakage-free datasets involves:

• Multimodal Similarity Calculation:

  • Compute protein structural similarity using TM-score
  • Calculate ligand chemical similarity using Tanimoto coefficients on extended connectivity fingerprints
  • Determine binding mode similarity using pocket-aligned ligand RMSD

• Iterative Filtering Process:

  • Identify all training complexes with TM-score > 0.9 AND Tanimoto > 0.9 to any test complex
  • Remove these training complexes to eliminate direct test leakage
  • Apply cluster-based filtering within training set: for similarity clusters (TM-score > 0.8 AND Tanimoto > 0.85), retain only representative complexes

• Validation of Separation:

  • Verify maximum similarity between training and test sets falls below thresholds
  • Ensure chemical diversity of ligands across splits

GEMS Training Protocol

The experimental protocol for training the sparse graph neural network includes:

• Graph Construction:

  • Protein residues represented as nodes with features from ProtBERT embeddings
  • Ligand atoms represented as nodes with features from ChemBERTa embeddings
  • Sparse edges created for: covalent bonds, hydrogen bonds (< 3.5Å), hydrophobic contacts (< 4.5Å), and ionic interactions (< 5.0Å)

• Model Configuration:

  • 6-layer graph attention network with multi-head attention (8 heads)
  • Residual connections between layers
  • Geometric attention mechanism incorporating spatial distances
  • Combined loss function: MSE for affinity + auxiliary contrastive loss for interaction classification

• Training Regimen:

  • Two-phase training: initial feature alignment using language model embeddings, followed by end-to-end fine-tuning
  • Learning rate: 1e-4 with cosine decay schedule
  • Batch size: 16 complexes
  • Early stopping based on validation loss with patience of 20 epochs

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Research Reagents and Computational Tools for Protein-Ligand Affinity Prediction

Resource	Type	Function	Access
PDBbind CleanSplit	Dataset	Leakage-free training data for robust model evaluation	Publicly available
CASF 2016/2019	Benchmark	Standardized test sets for scoring function comparison	Publicly available
PLA15 Benchmark	Dataset	Fragment-based interaction energy evaluation at DLPNO-CCSD(T) level	Publicly available
GEMS Implementation	Software	Sparse graph neural network for binding affinity prediction	Open source code
Boltz-2	Model	Foundation model for protein-ligand interaction prediction	Limited access
DAVIS Complete	Dataset	Modification-aware benchmark with protein variants	Publicly available
g-xTB	Software	Semiempirical quantum method for interaction energy calculation	Publicly available
LIT-PCBA Target ID Benchmark	Dataset	Evaluation set for target identification capability	Publicly available

Performance Analysis and Validation

Quantitative Benchmark Results

When evaluated under rigorous data separation conditions, GEMS demonstrates state-of-the-art performance on the CASF benchmark while maintaining robust generalization. The model achieves this through its sparse graph architecture that effectively captures physical interactions without relying on dataset biases.

The following DOT language script illustrates the message-passing mechanism within the sparse graph architecture:

Generalization to Independent Test Sets

The true validation of GEMS comes from its performance on strictly independent test datasets that share no significant similarities with the training data. Unlike previous models that showed drastic performance drops when evaluated on truly novel complexes, GEMS maintains predictive accuracy, demonstrating its ability to learn transferable principles of molecular recognition [4].

This robust generalization makes GEMS particularly valuable for screening protein-ligand interactions generated by generative AI models such as RFdiffusion and DiffSBDD, which can create novel complexes unlike those in existing structural databases.

The development of sparse graph neural networks for protein-ligand interaction prediction represents a significant architectural innovation addressing the critical challenge of generalization in computational drug discovery. By combining sparse graph modeling with rigorous dataset curation through PDBbind CleanSplit, researchers have established a new paradigm for developing and evaluating affinity prediction models that genuinely understand molecular interactions rather than exploiting dataset biases.

Future research directions include extending sparse graph architectures to model protein dynamics and allostery, incorporating explicit solvation effects, and developing multi-scale representations that combine atomic-level precision with residue-level efficiency. As the field moves toward these challenges, the principles of architectural sparsity and rigorous benchmark design established by this work will remain essential for building predictive models that translate successfully to real-world drug discovery applications.

The convergence of artificial intelligence (AI) and computational biology is reshaping the landscape of drug discovery and protein engineering. Central to this transformation are protein language models (PLMs) and chemical language models (CLMs), which reconceptualize molecular structures as a formal 'language' amenable to advanced computational techniques [27]. These models, pre-trained on vast corpora of biological and chemical data, learn the intricate "grammar" and "syntax" governing protein sequences and small molecules. However, the true potential of these models emerges not through standalone application, but through strategic integration via transfer learning paradigms.

This technical guide examines the framework for integrating protein and chemical language models, with particular emphasis on addressing critical challenges of data bias and generalization in affinity prediction research. Recent studies have revealed that performance metrics of many deep-learning-based binding affinity models are severely inflated due to train-test data leakage between standard benchmarks like the PDBbind database and the Comparative Assessment of Scoring Functions (CASF) datasets [4] [5]. One analysis found that nearly half of all CASF complexes had exceptionally similar counterparts in the training data, enabling models to achieve high benchmark performance through memorization rather than genuine understanding of protein-ligand interactions [4]. This context makes the development of robust, generalizable models through advanced transfer learning techniques not merely an optimization strategy but a fundamental requirement for credible computational drug design.

Molecular Representation Learning: From Sequence to Function

Protein Language Models (PLMs)

Protein language models learn meaningful representations of protein sequences through self-supervised training on evolutionary-scale datasets. These models typically employ transformer architectures to capture complex patterns and dependencies within amino acid sequences.

Table 1: Key Protein Language Models and Their Characteristics

Model	Architecture	Training Data	Parameters	Key Features
ESM-2 [28]	Transformer Encoder	UniRef50 (60M+ sequences)	8M to 15B	Masked language modeling, evolutionary scale
ProtT5 [28]	Encoder-Decoder	BFD100 (2.1B sequences)	Not specified	Text-to-Text Transfer Transformer framework
METL [29]	Transformer	Synthetic biophysical data	Not specified	Incorporates biophysical simulation data
ProteinBERT [28]	Transformer	UniRef90	Not specified	Joint learning of sequences and functions
ProtAlbert/ProtXLNet [28]	Transformer variants	UniRef100	Not specified	Improved architectures for protein modeling

Chemical Language Models (CLMs)

Chemical language models operate on string-based molecular representations such as SMILES (Simplified Molecular Input Line Entry System) or SELFIES (Self-referencing Embedded Strings), which translate molecular graphs into linear sequences [27] [30]. These models learn to generate syntactically and semantically valid molecular structures, enabling exploration of chemical space. Recent advancements demonstrate that CLMs can scale to generate entire biomolecules atom-by-atom, including proteins and protein-drug conjugates [30].

Transfer Learning Frameworks and Methodologies

Transfer learning with PLMs and CLMs typically follows two primary paradigms: embedding-based transfer and parameter fine-tuning. The selection between these approaches depends on available data, computational resources, and the specific downstream task.

Embedding-Based Transfer Learning

This approach uses pre-trained models as fixed feature extractors. The generated embeddings serve as input features for training separate, task-specific classifiers or regressors.

Table 2: Performance of PLM Embeddings with Different Classifiers for AMP Classification

PLM Embedding Source	Classifier	Key Performance Metrics	Dataset
ESM-2 [28]	Logistic Regression	State-of-the-art results	AMP classification
ProtT5 [28]	Support Vector Machines	Consistent improvement with model scale	AMP classification
ESM-1b [28]	XGBoost	Minimal effort implementation	AMP classification

Experimental Protocol: Embedding-Based AMP Classification

• Tokenization: Protein sequences are tokenized using the model's native tokenizer.
• Embedding Generation: Tokenized sequences are passed through the pre-trained PLM to generate token-level embeddings.
• Pooling: Mean pooling is applied across the sequence length to create fixed-size representations.
• Classification: Pooled embeddings with corresponding labels train shallow classifiers (LogReg, SVM, XGBoost).
• Evaluation: Moderate hyperparameter tuning is performed for SVM and XGBoost classifiers [28].

Parameter Fine-Tuning

This approach adapts a pre-trained model's weights to a specific downstream task through additional training on task-specific data. Efficient fine-tuning techniques have been shown to further enhance performance beyond embedding-based approaches [28].

Experimental Protocol: METL Framework for Protein Engineering The METL framework exemplifies a sophisticated transfer learning approach that incorporates biophysical knowledge:

• Synthetic Data Generation: Generate synthetic pretraining data via molecular modeling with Rosetta to model structures of millions of protein sequence variants. Extract 55+ biophysical attributes including molecular surface areas, solvation energies, and hydrogen bonding [29].
• Synthetic Data Pretraining: Pretrain a transformer encoder with structure-based relative positional embeddings to learn relationships between amino acid sequences and biophysical attributes.
• Experimental Data Fine-tuning: Fine-tune the pretrained transformer on experimental sequence-function data to produce models that integrate prior biophysical knowledge with experimental observations [29].

The METL framework demonstrates exceptional performance in challenging protein engineering tasks, particularly when generalizing from small training sets (as few as 64 examples) and in position extrapolation scenarios [29].

Diagram 1: METL Transfer Learning Framework (77 characters)

Addressing Data Bias and Generalization Challenges

The issue of data bias represents a critical challenge in computational drug design. Recent research has exposed widespread train-test data leakage between the PDBbind database and CASF benchmarks, severely inflating performance metrics of deep-learning-based binding affinity models [4] [5]. One study found that nearly 50% of CASF complexes had exceptionally similar counterparts in the training data, with some models performing comparably well even after omitting protein or ligand information from inputs [4].

The PDBbind CleanSplit Solution

To address data bias, researchers have developed PDBbind CleanSplit, a training dataset curated by a novel structure-based filtering algorithm that eliminates train-test data leakage and internal redundancies [4].

Methodology: Structure-Based Filtering Algorithm

• Multi-Modal Similarity Assessment: Compute similarity between protein-ligand complexes using:
  • Protein similarity (TM-scores)
  • Ligand similarity (Tanimoto scores)
  • Binding conformation similarity (pocket-aligned ligand RMSD)
• Train-Test Leakage Reduction: Exclude all training complexes closely resembling any CASF test complex.
• Redundancy Minimization: Iteratively remove complexes from training dataset to resolve similarity clusters.
• Ligand-Based Filtering: Remove training complexes with ligands identical to those in test complexes (Tanimoto > 0.9) [4].

When state-of-the-art models were retrained on CleanSplit, their performance dropped substantially, confirming that previous high scores were largely driven by data leakage rather than genuine generalization capability [4].

Generalized Affinity Prediction with GEMS

The Graph neural network for Efficient Molecular Scoring (GEMS) demonstrates robust generalization when trained on CleanSplit. Key innovations include:

• Sparse Graph Modeling: Representing protein-ligand interactions as sparse graphs
• Transfer Learning Integration: Leveraging knowledge from language models
• Multi-Modal Architecture: Combining structural and chemical information [4]

GEMS maintains high benchmark performance when trained on the rigorously filtered CleanSplit dataset, demonstrating genuine generalization to strictly independent test complexes rather than exploiting data leakage [4].

Diagram 2: Data Bias Resolution Workflow (78 characters)

Integrated Architectures for Molecular Design

Unified Protein and Chemical Modeling

The integration of protein and chemical language models enables simultaneous exploration of protein space and chemical space. Recent research demonstrates that chemical language models can generate atom-level representations of substantially larger molecules—scaling to entire proteins and protein-drug conjugates [30].

Experimental Protocol: Atom-by-Atom Biomolecule Generation

• Dataset Construction: Collect small proteins (50-150 residues) from PDB and single-domain antibodies from structural databases.
• Representation Parsing: Convert atom-level graph representations to linear string representations (SMILES/SELFIES).
• Model Training: Train chemical language models using masked or next-token prediction objectives.
• Generation and Validation: Generate novel samples and validate using:
  • Primary sequence analysis
  • AlphaFold structure prediction with pLDDT confidence scores
  • Amino acid distribution comparison to training data [30]

In one study, approximately 68.2% of generated samples represented valid proteins with unique, novel primary sequences that folded into structured conformations with high pLDDT scores (70-90), significantly outperforming random amino acid sequences [30].

Agentic AI Systems for Scientific Discovery

Beyond static models, agentic AI systems represent a emerging frontier where LLMs coordinate multiple tools and data sources to execute complex research workflows. Systems like Coscientist demonstrate how LLMs can transition from "passive" question-answering to "active" experimentation, where they:

• Plan and design experiments
• Translate natural language descriptions to executable code
• Control laboratory instrumentation
• Integrate computational and experimental workflows [31]

This active environment approach grounds model outputs in reality through interaction with specialized tools and databases, mitigating hallucination risks while accelerating discovery cycles.

The Scientist's Toolkit: Research Reagent Solutions

Table 3: Essential Resources for Protein and Chemical Language Model Research

Resource	Type	Function	Application Context
UniProt [28]	Database	Protein sequences and functional annotation	PLM pre-training and validation
PDBbind [4]	Database	Protein-ligand complexes with binding affinities	Training binding affinity prediction models
CleanSplit [4]	Curated Dataset	Bias-minimized training data	Robust model evaluation and training
Rosetta [29]	Software Suite	Molecular structure modeling and design	Biophysical simulation for pretraining
ESM-2 [28]	Pre-trained Model	General protein sequence representation	Transfer learning for diverse protein tasks
ProtT5 [28]	Pre-trained Model	Protein sequence understanding	Embedding generation and fine-tuning
METL [29]	Framework	Biophysics-informed protein engineering	Protein design with limited experimental data
AlphaFold [30]	Tool	Protein structure prediction	Validation of generated protein sequences
SELFIES/SMILES [30]	Representation	String-based molecular encoding	Chemical language model training and generation

The strategic integration of protein and chemical language models through transfer learning represents a paradigm shift in computational biology and drug discovery. By leveraging pre-trained models and adapting them to specific tasks, researchers can achieve state-of-the-art performance even with limited labeled data. However, the field must confront critical challenges of data bias and generalization, as exemplified by the PDBbind CleanSplit initiative, to build models that genuinely understand biological mechanisms rather than exploiting dataset artifacts.

The future trajectory points toward increasingly integrated and active AI systems that unite protein and small molecule design, incorporate biophysical principles, and interact directly with experimental instrumentation. These advancements will accelerate the transition from observational biology to programmable molecular design, ultimately enabling the creation of novel therapeutics and molecular solutions to address pressing challenges in human health and disease.

The process of drug discovery is traditionally characterized by high costs, extensive timelines, and significant attrition rates. In recent years, multitask learning (MTL) has emerged as a transformative paradigm that simultaneously addresses multiple predictive and generative tasks within a unified computational framework. Unlike single-task models that operate in isolation, MTL frameworks leverage shared representations and knowledge across related tasks, leading to improved generalization, streamlined model architectures, and more efficient learning, particularly for tasks with limited data [32]. Within computational drug discovery, this approach has created powerful new capabilities for integrating drug-target affinity (DTA) prediction with the generation of novel drug candidates, two tasks that are intrinsically interconnected in pharmacological research [22].

The integration of these capabilities addresses a critical bottleneck in therapeutic development. While predictive models identify potential interactions and generative models propose novel molecular structures, MTL frameworks combine these strengths to create a closed-loop discovery system. These systems predict binding affinities while simultaneously generating target-aware drug variants optimized for those same affinity characteristics [22]. However, this integration introduces significant computational challenges, particularly concerning gradient conflicts between tasks and data bias in affinity prediction benchmarks that can severely limit real-world generalization [33] [4]. This technical guide examines the architecture, optimization strategies, and validation methodologies for MTL frameworks that successfully balance affinity prediction with drug generation, while addressing the critical issue of generalization in predictive models.

The DeepDTAGen Framework: A Case Study in Unified MTL

The DeepDTAGen framework represents a state-of-the-art implementation of MTL for drug discovery, specifically designed to predict drug-target binding affinities while simultaneously generating novel target-aware drug molecules [22]. This framework employs a shared feature space for both tasks, allowing knowledge of ligand-receptor interactions learned during affinity prediction to directly inform the drug generation process. The architectural design consists of several integrated components:

• Shared Encoder Modules: Process both drug molecules (represented as SMILES strings or molecular graphs) and target proteins (represented as amino acid sequences) to extract latent features that capture structural properties and conformational dynamics.
• Affinity Prediction Head: A regression module that takes the shared representations and predicts quantitative binding affinity values, typically using fully connected layers.
• Target-Aware Generator: A transformer-based decoder that generates novel drug SMILES strings conditioned on the target protein features and interaction information from the shared encoder.

This unified approach ensures that the generated molecules are not merely chemically valid but are specifically optimized for binding to the target of interest, significantly increasing their potential for clinical success [22].

The FetterGrad Optimization Algorithm

A fundamental challenge in MTL arises when gradients from different tasks conflict, potentially slowing convergence and reducing final performance—a phenomenon known as negative transfer [33]. DeepDTAGen introduces the FetterGrad algorithm to specifically address this optimization challenge. The algorithm operates by:

• Monitoring Gradient Directions: Continuously tracking the gradient vectors for both affinity prediction and drug generation tasks during training.
• Quantifying Gradient Interference: Calculating the cosine similarity between task gradients to identify conflicting update directions.
• Aligning Gradient Updates: Actively minimizing the Euclidean distance between task gradients when conflicts are detected, ensuring more harmonious parameter updates [22].

This approach mitigates the optimization challenges associated with multitask learning, particularly those caused by gradient conflicts between distinct tasks, leading to more stable training and improved performance on both objectives [22].

Table 1: DeepDTAGen Performance on Benchmark Datasets for Affinity Prediction

Dataset	MSE	Concordance Index	R²m	AUPR
KIBA	0.146	0.897	0.765	-
Davis	0.214	0.890	0.705	-
BindingDB	0.458	0.876	0.760	-

Diagram 1: DeepDTAGen Framework Architecture showing shared encoder and dual task heads with FetterGrad optimization.

Data Bias and Generalization Challenges in Affinity Prediction

The Data Leakage Problem in Benchmark Datasets

A critical challenge in developing robust affinity prediction models is the pervasive issue of data bias and train-test leakage in commonly used benchmarks. Recent research has revealed that the performance metrics of many deep-learning-based binding affinity prediction models have been severely inflated due to data leakage between the PDBbind database and the Comparative Assessment of Scoring Function (CASF) benchmark datasets [4].

This leakage occurs when training and test datasets share highly similar protein-ligand complexes, enabling models to achieve high benchmark performance through memorization and exploitation of structural similarities rather than genuine understanding of protein-ligand interactions. Alarmingly, some models maintain competitive performance even when critical input information (such as protein or ligand data) is omitted, indicating they are not learning the underlying interaction mechanics [4].

Addressing Bias Through Curated Data Splits

To combat this issue, researchers have developed PDBbind CleanSplit, a training dataset curated by a structure-based filtering algorithm that eliminates train-test data leakage and reduces redundancies within the training set [4]. The filtering approach employs a multimodal strategy that assesses:

• Protein Similarity: Using TM-scores to quantify structural similarity between proteins.
• Ligand Similarity: Calculating Tanimoto scores based on molecular fingerprints.
• Binding Conformation Similarity: Measuring pocket-aligned ligand root-mean-square deviation (RMSD).

This comprehensive filtering identified that nearly 50% of CASF complexes had highly similar counterparts in the training data, creating substantial data leakage. When state-of-the-art models are retrained on CleanSplit, their performance typically drops substantially, confirming that previous high scores were largely driven by data leakage rather than true generalization capability [4].

Table 2: Impact of Data Bias on Model Generalization Performance

Model	Performance on Standard Split	Performance on CleanSplit	Performance Drop
GenScore	High (Reported SOTA)	Substantially Reduced	Significant
Pafnucy	High (Reported SOTA)	Substantially Reduced	Significant
GEMS	-	Maintains High Performance	Minimal

Advanced MTL Optimization Strategies

Gradient Conflict Resolution

Beyond FetterGrad, several advanced optimization strategies have been developed to address gradient conflicts in MTL environments. The SON-GOKU scheduler represents an alternative approach that:

• Computes gradient interference between tasks
• Constructs an interference graph based on cosine similarity of gradients
• Applies greedy graph coloring to partition tasks into compatible groups
• Activates only one group of tasks per training step [33]

This method ensures that each mini-batch contains only tasks that pull the model in compatible directions, reducing gradient variance and conflicting updates. Empirical results across six datasets show that this interference-aware graph coloring approach consistently outperforms baselines and can be combined with existing MTL optimizers like PCGrad, AdaTask, and GradNorm for additional improvements [33].

Task-Specific Neurons in Large Language Models

Recent research on large language models (LLMs) has revealed that task-specific neurons play a crucial role in MTL generalization and specialization. Through gradient attribution analysis, researchers have identified that:

• Different tasks activate distinct subsets of neurons within shared model architectures
• The overlap between task-specific neurons correlates strongly with generalization capabilities
• In certain model layers, high parameter similarity between task-specific neurons predicts better generalization performance [34]

These insights have led to neuron-level continuous fine-tuning methods that selectively update only task-relevant neurons during continuous learning, reducing catastrophic forgetting while maintaining performance on previous tasks [34].

Diagram 2: SON-GOKU Task Grouping and Scheduling based on gradient conflict analysis.

Experimental Protocols and Methodologies

Model Training and Evaluation

Comprehensive evaluation of MTL frameworks for drug discovery requires rigorous experimental protocols across both predictive and generative tasks:

Affinity Prediction Evaluation:

• Datasets: KIBA, Davis, and BindingDB are standard benchmarks, though CleanSplit should be used for generalization assessment [22] [4]
• Evaluation Metrics:
  • Mean Squared Error (MSE) for regression accuracy
  • Concordance Index (CI) for ranking performance
  • R²m for model goodness-of-fit
  • Area Under Precision-Recall Curve (AUPR) for interaction prediction

Drug Generation Evaluation:

• Validity: Proportion of chemically valid molecules among generated structures
• Novelty: Percentage of valid molecules not present in training data
• Uniqueness: Proportion of unique molecules among valid generations
• Target-Specific Binding: Assessment of generated molecules' ability to bind intended targets [22]

Chemical Property Analysis

For generated molecules, comprehensive chemical analyses should include:

• Solubility: Prediction of aqueous solubility for developability assessment
• Drug-likeness: Compliance with established rules (Lipinski's Rule of Five)
• Synthesizability: Estimation of synthetic feasibility and complexity
• Structural Analysis: Counts of atom types, bond types, and ring structures [22]

Research Reagent Solutions

Table 3: Essential Research Tools for MTL in Drug Discovery

Resource	Type	Primary Function	Application in MTL
PDBbind CleanSplit	Dataset	Curated protein-ligand complexes	Generalization evaluation for affinity prediction
CASF Benchmark	Dataset	Standardized test complexes	Performance comparison (with leakage awareness)
DeepDTAGen Framework	Software	Multitask affinity prediction and drug generation	Unified MTL implementation reference
FetterGrad Algorithm	Algorithm	Gradient conflict mitigation	MTL optimization
SON-GOKU Scheduler	Algorithm	Task grouping via graph coloring	Interference-aware MTL training
GEMS Model	Model	Graph neural network for scoring	Robust affinity prediction on clean splits

The integration of affinity prediction with drug generation in multitask learning frameworks represents a paradigm shift in computational drug discovery. These approaches leverage shared representations to create synergistic effects between predictive and generative tasks, potentially accelerating the entire drug discovery pipeline. However, addressing data bias and ensuring genuine generalization remain critical challenges that must be confronted through rigorous benchmarking and specialized optimization techniques.

Future research directions should focus on:

• Developing more sophisticated optimization algorithms that dynamically balance task priorities during training
• Creating更大规模, rigorously curated datasets that minimize bias while maximizing chemical and target diversity
• Exploring transformer-based architectures that can seamlessly integrate protein language modeling with molecular generation
• Implementing explainable AI techniques to interpret model decisions and build trust in generated molecules

As these technologies mature, MTL frameworks that balance affinity prediction with drug generation have the potential to significantly reduce the time and cost of therapeutic development while increasing the success rate of candidate molecules in preclinical and clinical testing.

In computational drug discovery, the application of multitask deep learning models for predicting drug-target interactions and generating novel compounds presents significant optimization challenges. Conflicting gradients arising from distinct learning objectives can impede model convergence and degrade performance. This technical guide examines the core algorithms and experimental methodologies for resolving these conflicts, with a specific focus on their critical role in mitigating data bias and enhancing the generalization capabilities of affinity prediction models. We provide an in-depth analysis of gradient descent optimization techniques, including the novel FetterGrad algorithm, and present structured experimental protocols to validate their efficacy in producing robust, generalizable models for structure-based drug design.

The integration of multitask learning (MTL) in computational drug discovery represents a paradigm shift, enabling simultaneous prediction of drug-target binding affinity (DTA) and generation of target-aware drug variants. However, these models are prone to optimization challenges, particularly conflicting gradients between distinct tasks, which can lead to biased parameter updates, unstable training, and poor generalization [22]. The issue of generalization is further exacerbated by underlying data biases in public benchmarks. Recent studies have revealed that train-test data leakage between the PDBbind database and the Comparative Assessment of Scoring Function (CASF) benchmarks has severely inflated performance metrics of deep-learning-based scoring functions, leading to overestimation of their true capabilities [4] [5]. When models are trained on datasets with such redundancies and leakage, they often settle for a local minimum in the loss landscape by exploiting structural similarities rather than learning genuine protein-ligand interactions [4]. Therefore, addressing conflicting gradients is not merely an optimization concern but a fundamental prerequisite for developing models that generalize reliably to novel, unseen protein-ligand complexes in real-world drug development scenarios.

Core Gradient Descent Algorithms for Multi-Objective Optimization

The foundation for resolving conflicting learning objectives lies in advanced variants of the gradient descent algorithm. These methods modulate the direction and magnitude of parameter updates by incorporating historical gradient information.

Table 1: Core Gradient Descent Optimization Algorithms

Algorithm	Key Mechanism	Advantages in MTL Context	Hyperparameters
Momentum	Accumulates an exponentially decaying average of past gradients (first moment) [35] [36].	Prevents stalling in local minima/plateaus; maintains directionality [35] [37].	Decay rate (β₁, ~0.9), Learning Rate (η)
RMSProp	Maintains an exponentially decaying average of squared gradients (second moment) [35] [37].	Adapts learning rate per parameter; handles sparse features well [35].	Decay rate (β₂, ~0.999), Learning Rate (η)
Adam	Combines Momentum and RMSProp, using bias-corrected estimates of both first and second moments [35] [36] [37].	Provides smooth, scaled updates; generally robust and well-suited for non-stationary objectives [35] [38].	β₁ (~0.9), β₂ (~0.999), η, ε (e.g., 1e-8)

The Adam optimizer is particularly noteworthy as it empirically performs well on a wide range of deep learning problems [35]. It calculates updates by combining the first moment estimate (mean of gradients), which provides momentum, and the second moment estimate (uncentered variance of gradients), which adapts the learning rate for each parameter [36] [37]. This allows it to navigate the complex loss landscapes common in multitask learning for drug discovery with consistent and stable updates [35].

Visualizing Gradient Descent Dynamics

The following diagram illustrates the distinct paths taken by different optimization algorithms through a simplified loss landscape, highlighting how momentum and adaptive scaling influence the convergence behavior.

Specialized Algorithms for Gradient Conflict Resolution

While general-purpose optimizers like Adam are powerful, multitask learning with competing objectives often requires more specialized techniques.

The FetterGrad Algorithm

The FetterGrad algorithm was developed specifically to address gradient conflicts in the DeepDTAGen framework, a multitask model that predicts drug-target affinity and generates novel drugs using a shared feature space [22]. Its primary innovation lies in actively aligning the gradients of different tasks during training.

The core objective of FetterGrad is to mitigate gradient conflicts and biased learning by minimizing the Euclidean distance (ED) between the gradients of distinct tasks [22]. This ensures that the updates for one task do not undermine the learning progress of another, leading to more stable and effective convergence on both objectives simultaneously.

Table 2: Comparison of Gradient Conflict Resolution Strategies

Strategy	Primary Approach	Application Context
FetterGrad	Minimizes Euclidean Distance between task gradients [22].	Multitask Learning for DTA Prediction & Drug Generation.
Gradient Surgery	Projects conflicting components of task gradients [22].	General Computer Vision and NLP Multitask Problems.
Uncertainty Weighting	Adaptively weights task losses based on uncertainty [22].	Multi-loss Regression and Classification Problems.

Experimental Protocols for Evaluating Optimization and Generalization

Validating the effectiveness of optimization techniques requires rigorous experimentation focused on both performance metrics and generalization capability.

Protocol: Benchmarking Optimization Algorithms

Objective: Compare the performance of SGD, Momentum, Adam, and FetterGrad on a defined multitask problem.

• Model Architecture: Implement a standard multitask architecture (e.g., shared encoder with task-specific heads).
• Dataset: Use a benchmark dataset with known bias issues, such as PDBbind, employing a rigorously cleaned split like PDBbind CleanSplit to ensure a valid assessment of generalization [4].
• Training: Train the model using each optimizer with their optimally tuned hyperparameters.
• Evaluation Metrics:
  • Task Performance: For DTA prediction, use Mean Squared Error (MSE) and Concordance Index (CI) [22].
  • Optimization Efficiency: Track training loss convergence and wall-clock time to a specific performance threshold.
  • Generalization Gap: Measure the difference in performance between training and a strictly independent test set [4].

Protocol: Assessing Generalization with PDBbind CleanSplit

Objective: Quantify the true generalization of a model by eliminating data leakage.

• Dataset Curation: Apply a structure-based filtering algorithm to the PDBbind database to create the PDBbind CleanSplit [4]. This algorithm uses combined assessments of protein similarity (TM-scores), ligand similarity (Tanimoto scores), and binding conformation similarity (pocket-aligned ligand RMSD) to identify and remove training complexes that are overly similar to those in the test sets [4].
• Baseline Establishment: Retrain existing state-of-the-art models (e.g., GenScore, Pafnucy) on the CleanSplit training set and benchmark their performance on the independent CASF test set. Studies show a marked performance drop, confirming prior performance was inflated by data leakage [4].
• Model Evaluation: Train the new model (e.g., GEMS - Graph neural network for Efficient Molecular Scoring) on the CleanSplit training data and evaluate it on the strictly independent test set. High performance under these conditions indicates robust generalization [4].

The workflow below outlines the key steps in creating and using a rigorously filtered dataset to assess model generalization, a critical process for overcoming data bias.

The Scientist's Toolkit: Essential Research Reagents and Materials

The following table details key computational tools and data resources essential for experimental work in this field.

Table 3: Key Research Reagents and Computational Tools

Item Name	Function / Description	Application in Research
PDBbind Database	A comprehensive database of protein-ligand complexes with binding affinity data [4] [5].	Primary source of training data for structure-based binding affinity prediction models.
CASF Benchmark	The Comparative Assessment of Scoring Functions benchmark datasets [4].	Standard benchmark for evaluating the generalization capability of scoring functions.
PDBbind CleanSplit	A curated version of PDBbind with minimized train-test leakage and internal redundancy [4].	Enables genuine evaluation of model generalization on strictly independent test complexes.
FetterGrad Optimizer	A gradient optimization algorithm that minimizes Euclidean distance between task gradients [22].	Resolves gradient conflicts in multitask learning models (e.g., DeepDTAGen).
Graph Neural Network (GNN)	A neural network architecture that operates on graph-structured data, modeling nodes and edges [4].	Represents protein-ligand complexes as sparse graphs to capture key interaction features.
Language Model Embeddings	Pre-trained embeddings from large language models (e.g., ProtBERT for proteins) [4] [2].	Provides transfer learning of semantic and structural features for proteins and ligands.

Resolving conflicting learning objectives through advanced gradient optimization is a cornerstone for building robust and generalizable models in computational drug discovery. Techniques ranging from the widely-used Adam optimizer to specialized algorithms like FetterGrad are essential for training complex multitask architectures effectively. However, algorithmic advances alone are insufficient without a concerted effort to address underlying data biases. The use of rigorously curated datasets, such as PDBbind CleanSplit, is critical for moving beyond inflated benchmark metrics and achieving genuine generalization. The future of affinity prediction lies in the continued co-development of unbiased data resources and optimization techniques that ensure models learn the true principles of biomolecular interaction, ultimately accelerating the discovery of novel therapeutics.

In computational drug design, the cold-start problem presents a fundamental challenge for developing accurate predictive models, particularly in the critical task of binding affinity prediction. This problem manifests when models face new protein-ligand complexes with structural characteristics or interaction patterns that significantly differ from those present in the training data, creating a low-similarity scenario where predictive accuracy substantially degrades. The core issue stems from the data bias and generalization crisis currently affecting the field, where train-test data leakage between standard benchmarking datasets has severely inflated performance metrics and led to overestimation of model capabilities [4] [5]. This leakage creates a false impression of model robustness, masking fundamental weaknesses that become apparent only when models encounter truly novel complexes in real-world drug discovery applications.

The cold-start problem is particularly acute in structure-based drug design (SBDD), where accurate scoring functions are essential for predicting protein-ligand binding affinities. Classical scoring functions implemented in docking tools like AutoDock Vina and GOLD demonstrate limited accuracy in binding affinity prediction, while deep-learning approaches have failed to deliver expected performance gains on independent test datasets [4]. This performance gap directly impacts the drug development pipeline, where unreliable affinity predictions for novel targets can lead to costly late-stage failures and missed therapeutic opportunities. Addressing this challenge requires both methodological innovations in model architecture and fundamental improvements in dataset construction and evaluation protocols to ensure models can generalize beyond their training distributions.

The Data Leakage Crisis: Quantifying Benchmark Inflation

Recent research has revealed systematic flaws in the standard evaluation paradigms for binding affinity prediction, with significant implications for cold-start performance. A critical analysis of the relationship between the PDBbind database and Comparative Assessment of Scoring Function (CASF) benchmarks has exposed widespread train-test data leakage, fundamentally compromising the validity of reported generalization capabilities [4].

Structural Similarity Analysis Between Training and Test Complexes

To quantify the extent of this data leakage, researchers developed a structure-based clustering algorithm that assesses similarity across three dimensions: protein similarity (TM scores), ligand similarity (Tanimoto scores), and binding conformation similarity (pocket-aligned ligand root-mean-square deviation) [4]. This multimodal approach can identify complexes with similar interaction patterns even when proteins share low sequence identity, providing a robust framework for detecting functionally equivalent complexes.

Table 1: Quantified Data Leakage Between PDBbind and CASF Benchmarks

Similarity Metric	Threshold Value	Number of Similar Complex Pairs	Percentage of CASF Complexes Affected
Combined similarity (protein + ligand + conformation)	Structure-based filtering algorithm	Nearly 600 pairs identified	49% of all CASF complexes
Ligand similarity only	Tanimoto > 0.9	Not specified	Affected complexes removed in CleanSplit
Protein similarity	TM score threshold	Not specified	Contributing factor to combined similarity

The analysis revealed nearly 600 highly similar pairs between PDBbind training and CASF complexes, affecting approximately 49% of all CASF test complexes [4]. These structurally similar pairs share not only comparable ligand and protein structures but also nearly identical ligand positioning within protein pockets, and consequently, closely matched affinity labels. This enables models to achieve misleadingly high benchmark performance through simple memorization rather than genuine understanding of protein-ligand interactions, creating a false confidence in their ability to handle true cold-start scenarios.

Performance Impact of Data Leakage

The practical consequence of this data leakage becomes evident when comparing model performance before and after its removal. When state-of-the-art binding affinity prediction models like GenScore and Pafnucy were retrained on the cleaned PDBbind CleanSplit dataset—which eliminates both train-test leakage and internal redundancies—their benchmark performance dropped substantially [4]. This performance degradation confirms that previously reported high accuracy metrics were largely driven by data leakage rather than true generalization capability, highlighting the vulnerability of these models to cold-start conditions.

Alarmingly, some models maintained competitive performance on CASF benchmarks even after omitting all protein or ligand information from their input data, suggesting they were exploiting dataset-specific biases rather than learning fundamental principles of molecular recognition [4]. This finding has profound implications for real-world drug discovery, where models must predict affinities for genuinely novel complexes that share minimal structural similarity with previously characterized interactions.

Methodological Framework: PDBbind CleanSplit Protocol

To address the data leakage crisis and establish a more rigorous foundation for cold-start research, researchers developed the PDBbind CleanSplit protocol—a systematically filtered training dataset that eliminates train-test data leakage and reduces internal redundancies [4]. This methodology provides a robust framework for training and evaluating models intended for low-similarity scenarios.

Multimodal Filtering Algorithm

The core innovation of the CleanSplit protocol is a structure-based clustering algorithm that performs multimodal filtering based on three complementary similarity metrics. The algorithm executes the following sequential filtering steps:

• Protein Structure Similarity Assessment: Computes TM-scores between all protein pairs to identify structurally similar proteins regardless of sequence identity [4].

• Ligand Chemical Similarity Evaluation: Calculates Tanimoto scores between all ligand pairs to identify chemically similar compounds [4].

• Binding Conformation Comparison: Measures pocket-aligned ligand root-mean-square deviation (r.m.s.d.) to identify complexes with similar binding modes [4].

The algorithm applies conservative thresholds across all three dimensions to identify and remove training complexes that resemble any CASF test complex. Additionally, it eliminates all training complexes with ligands identical to those in the CASF test set (Tanimoto > 0.9), providing an additional safeguard against ligand-based data leakage [4]. This comprehensive approach ensures that models evaluated on CASF benchmarks face genuinely novel challenges rather than variations of previously encountered complexes.

Redundancy Reduction in Training Data

Beyond addressing train-test leakage, the CleanSplit protocol systematically reduces internal redundancies within the training dataset. The original PDBbind database contained numerous similarity clusters, with nearly 50% of all training complexes belonging to such clusters [4]. These redundancies enable models to settle for easily attainable local minima in the loss landscape through structure-matching rather than developing robust feature representations.

The filtering algorithm uses adapted thresholds to identify and iteratively eliminate the most significant similarity clusters until all are resolved, ultimately removing 7.8% of training complexes [4]. This redundancy reduction encourages models to learn generalizable principles of molecular recognition rather than memorizing specific structural patterns, directly enhancing their capability to handle cold-start scenarios with low-similarity complexes.

Experimental Strategies for Cold-Start Scenarios

Graph Neural Network for Efficient Molecular Scoring (GEMS)

To address the cold-start challenge in strict low-similarity environments, researchers developed GEMS (Graph Neural Network for Efficient Molecular Scoring)—a novel architecture that maintains high benchmark performance even when trained on the rigorously filtered PDBbind CleanSplit dataset [4]. The model incorporates several key innovations specifically designed to enhance generalization capability:

• Sparse Graph Modeling: Represents protein-ligand interactions using a sparse graph structure that efficiently captures essential interaction patterns while reducing noise and redundancy [4].

• Transfer Learning from Language Models: Leverages knowledge transferred from pre-trained protein and chemical language models to bootstrap understanding of structural and functional relationships, providing a foundational representation that generalizes to novel complexes [4].

• Multi-Scale Feature Integration: Combines atomic-level interaction features with residue-level and molecular-level contextual information to create a hierarchical representation of binding interactions.

When evaluated on strictly independent test datasets after training on CleanSplit, GEMS maintained state-of-the-art prediction accuracy while ablations studies confirmed that the model fails to produce accurate predictions when protein nodes are omitted from the graph [4]. This demonstrates that GEMS predictions are based on genuine understanding of protein-ligand interactions rather than exploiting dataset biases or memorization strategies.

Heuristics and Wizard-of-Oz Approaches for Early-Stage Validation

Beyond architectural innovations, strategic methodological approaches can help mitigate cold-start challenges during initial model development and validation:

Heuristics-First Implementation: Before deploying complex machine learning models, researchers recommend solving the problem with statistical methods or heuristics to establish performance baselines and develop intimate familiarity with the problem domain [39]. As former GitHub Staff ML engineer Hamel Hussain notes: "Solve the problem manually, or with heuristics. This will force you to become intimately familiar with the problem and the data, which is the most important first step" [39]. In binding affinity prediction, this might involve implementing classical scoring functions or knowledge-based potentials to establish baseline performance before introducing deep learning approaches.

Wizard-of-Oz Prototyping: For high-stakes applications where model inaccuracies could have significant consequences, incorporating human validation for edge cases provides a crucial safety mechanism during early deployment phases [39]. This approach, exemplified by Amazon's Just Walk Out technology that employs humans to validate edge cases where computer vision algorithms fail, allows for real-world validation while acknowledging current model limitations [39]. In drug discovery contexts, this might involve expert medicinal chemists reviewing and validating predictions for novel target classes.

Table 2: Strategic Approaches for Cold-Start Scenarios in Drug Discovery

Approach	Methodology	Application Context	Benefits
Heuristics-First Implementation	Statistical methods and rule-based systems	Early-stage model development	Provides reliable baseline; facilitates problem understanding
Wizard-of-Oz Prototyping	Human-in-the-loop validation for edge cases	High-stakes validation phases	Enables real-world testing; provides safety mechanism
Synthetic Data Generation	Artificially generating training data	Data-scarce domains and novel targets	Addresses data scarcity; privacy preservation
Public Dataset Utilization	Curated open data repositories	Initial model prototyping	Rapid experimentation; benchmark establishment

Synthetic Data and Public Dataset Utilization

For particularly challenging cold-start scenarios involving novel target classes or rare structural motifs, supplemental data strategies can provide additional leverage:

Synthetic Data Generation: Artificially generating training data addresses fundamental data scarcity challenges, particularly for novel target classes with limited structural characterization [39]. In computational drug discovery, this might involve generating synthetic protein-ligand complexes through molecular dynamics simulations or computational docking of diverse compound libraries against target structures.

Public Dataset Curation: While public datasets like PDBbind provide valuable starting points, their static nature and potential quality issues limit their utility for production systems [39]. As Eric Ma, Principal Data Scientist at Moderna Therapeutics, recommends: "Reach for public datasets only as a testbed to prototype a model" rather than as a complete solution to scientific problems [39]. Successful examples include Google's use of public datasets with synthetic 3D molecular structures to train models predicting small-molecule drug affinity [39].

Experimental Protocols and Validation Frameworks

Structural Similarity Assessment Protocol

To ensure rigorous evaluation of model performance in genuine cold-start scenarios, researchers must implement comprehensive structural similarity assessment between training and test complexes. The following experimental protocol provides a standardized approach:

• Protein Structure Alignment: For all protein pairs between training and test sets, compute TM-scores using structural alignment algorithms. Record all pairs exceeding a conservative similarity threshold (e.g., TM-score > 0.7) [4].

• Ligand Similarity Calculation: For all ligand pairs, compute Tanimoto coefficients based on molecular fingerprints. Identify pairs with high chemical similarity (Tanimoto > 0.9) for exclusion [4].

• Binding Mode Comparison: For protein-ligand pairs passing initial similarity filters, perform binding site alignment and calculate pocket-aligned ligand RMSD to identify complexes with similar interaction geometries [4].

• Composite Filter Application: Apply conservative thresholds across all three similarity dimensions to identify and exclude complexes with potential data leakage.

This protocol should be implemented before any model training to ensure clean dataset splits, and should be repeated for any new test sets introduced during model evaluation.

Cross-Validation Under Low-Similarity Conditions

Traditional random cross-validation approaches can significantly overestimate model performance in cold-start scenarios due to undetected structural similarities between training and validation splits. To address this limitation, researchers should implement similarity-aware cross-validation:

This validation approach ensures that models are evaluated on truly novel structural motifs rather than variations of training examples, providing a realistic assessment of cold-start performance.

Performance Metrics and Benchmarking

When evaluating models for cold-start scenarios, standard performance metrics must be supplemented with similarity-aware analyses:

Table 3: Performance Metrics for Cold-Start Evaluation

Metric	Calculation Method	Interpretation in Cold-Start Context
Similarity-Stratified RMSE	RMSE calculated separately for high, medium, and low similarity test cases	Reveals performance degradation with decreasing similarity
Novel Target Prediction Accuracy	Accuracy specifically on targets with <30% sequence identity to training set	Directly measures cold-start capability
Structural Motif Transfer Score	Performance on novel structural motifs not present in training	Assesses generalization beyond training distribution
Affinity Rank Correlation	Spearman correlation between predicted and experimental affinities	Measures utility for virtual screening applications

Research Reagent Solutions

Table 4: Essential Research Reagents for Cold-Start Experimentation

Reagent/Solution	Function	Application Context
PDBbind Database	Comprehensive collection of protein-ligand complexes with binding affinity data	Primary source of training data for binding affinity prediction models
CASF Benchmark Sets	Curated test sets for scoring function evaluation	Standardized performance assessment; requires careful similarity filtering
CleanSplit Filtering Algorithm	Structure-based clustering to eliminate data leakage	Creation of rigorously separated training and test sets
TM-score Algorithm	Protein structure similarity quantification	Detection of structurally similar complexes despite low sequence identity
Tanimoto Coefficient Calculator	Ligand chemical similarity assessment	Identification of chemically related compounds in training and test sets
GEMS Architecture Reference Implementation	Graph neural network for binding affinity prediction	Baseline model with demonstrated generalization capability
Molecular Graph Construction Toolkit	Protein-ligand complex representation as sparse graphs	Input data preparation for graph-based learning approaches

The cold-start problem in binding affinity prediction represents a significant bottleneck in computational drug discovery, particularly as the field increasingly targets novel protein classes with limited structural characterization. Addressing this challenge requires a multifaceted approach that combines rigorous dataset curation, specialized model architectures, and comprehensive evaluation protocols. The PDBbind CleanSplit methodology provides a foundational framework for eliminating data leakage and establishing meaningful performance benchmarks, while approaches like GEMS demonstrate that architectural innovations can deliver genuine generalization to novel complexes.

Future progress will likely depend on increased integration of transfer learning from protein language models, development of more sophisticated data augmentation strategies for structural data, and establishment of community standards for cold-start evaluation. As the field moves toward targeting increasingly novel biological systems, overcoming the cold-start challenge will be essential for realizing the full potential of computational approaches in accelerating therapeutic development.

Rigorous Validation: Comparing Model Performance Across Strict Benchmarks

The accurate prediction of protein-ligand binding affinity is a cornerstone of computer-aided drug discovery. For years, the field has relied on benchmark performances trained on the PDBbind database and evaluated on the Comparative Assessment of Scoring Functions (CASF) benchmark to gauge progress [40] [12]. However, recent research has exposed a critical flaw in this paradigm: widespread train-test data leakage has severely inflated performance metrics, leading to an overestimation of model true generalization capabilities [40] [41] [42]. This leakage occurs because the standard and core sets of PDBBind are cross-contaminated with proteins and ligands of high similarity, meaning models are often tested on data that closely resembles their training set [42]. One analysis found that nearly 600 similarities existed between PDBbind training complexes and the CASF test set, affecting 49% of all CASF complexes [40]. This means nearly half of the standard test cases do not represent novel challenges, allowing models to perform well through memorization rather than a genuine understanding of protein-ligand interactions [40] [43].

The introduction of rigorously curated datasets, most notably PDBbind CleanSplit, aims to resolve this issue by creating a strict separation between training and test data [40]. This whitepaper provides a technical guide and performance comparison, framing the discussion within the broader thesis that resolving data bias is fundamental to achieving true generalization in affinity prediction models. We summarize quantitative data from retraining experiments, detail the methodologies for creating clean splits, and provide the scientific community with tools to advance robust model development.

Experimental Protocols for Clean Data Splitting

The PDBbind CleanSplit Methodology

The creation of PDBbind CleanSplit involves a structure-based clustering algorithm designed to eliminate data leakage and reduce internal redundancy [40]. The protocol is as follows:

• Multimodal Similarity Assessment: The algorithm computes a combined similarity score between two protein-ligand complexes using three distinct metrics:

  • Protein Similarity: Calculated using the TM-score, a metric for measuring the structural similarity of protein structures [40] [12].
  • Ligand Similarity: Calculated using the Tanimoto coefficient, a standard measure for comparing molecular fingerprints [40] [12].
  • Binding Conformation Similarity: Calculated using the pocket-aligned ligand root-mean-square deviation (RMSD) to assess the similarity of ligand positioning within the protein binding pocket [40] [12].

• Train-Test Leakage Reduction: The algorithm identifies and excludes all training complexes in PDBbind that closely resemble any complex in the CASF test sets based on the above similarity thresholds. Furthermore, it removes all training complexes with ligands that are nearly identical (Tanimoto > 0.9) to those in the CASF test set [40]. This step addresses findings that graph neural networks (GNNs) often rely on ligand memorization for affinity predictions [40].

• Internal Redundancy Reduction: The algorithm identified that nearly 50% of all training complexes were part of a similarity cluster [40]. Using adapted filtering thresholds, the algorithm iteratively removed complexes from the training dataset to resolve the most striking similarity clusters, eliminating an additional 7.8% of training complexes [40]. This encourages models to learn generalizable patterns instead of settling for a local minimum in the loss landscape via memorization.

The LP-PDBBind Methodology

An independent approach, Leak Proof PDBBind (LP-PDBBind), follows a similar philosophy but with a different splitting strategy [42]:

• Data Cleaning: The protocol first cleans the PDBBind data by eliminating covalently bound ligand-protein complexes, focusing only on non-covalent binders. It also removes ligands with very low-frequency atomic elements and structures with obvious steric clashes [42].
• Similarity-Control Splitting: The dataset is reorganized into training, validation, and test splits by minimizing sequence and chemical similarity of both proteins and ligands between the splits. This provides control over protein-ligand structural interaction patterns across all data splits, an improvement over protein-family-only splits [42].
• Independent Benchmark Creation: The methodology also involves the creation of a new independent evaluation set, BDB2020+, compiled from BindingDB entries deposited after 2020 and filtered with the same similarity control criteria [42]. This provides a true blind test for retrained models.

The following diagram illustrates the logical workflow for creating a cleaned dataset suitable for benchmarking generalization.

Quantitative Performance Comparison

Retraining existing state-of-the-art models on the cleaned datasets revealed a dramatic drop in their benchmark performance, exposing their previous reliance on data leakage.

Table 1: Performance Comparison of Models on Standard vs. Cleaned Data Splits

Model	Training Data	Test Benchmark	Reported Performance (Pearson R)	Performance after Retraining (Pearson R)	Change	Source/Study
GenScore	Original PDBBind	CASF	High (Original Benchmark)	Marked Drop	Substantial	[40]
Pafnucy	Original PDBBind	CASF	High (Original Benchmark)	Marked Drop	Substantial	[40]
GEMS	PDBBind CleanSplit	CASF	N/A	Maintained High Performance	State-of-the-Art	[40]
Multiple SFs (Vina, RF-Score, IGN, DeepDTA)	Original PDBBind	LP-PDBBind Test Set	High (on standard core set)	Better performance due to controlled leakage	Inflated on standard split	[42]
Multiple SFs (Vina, RF-Score, IGN, DeepDTA)	LP-PDBBind	Independent BDB2020+	N/A	Consistently Better	Improved Generalization	[42]

The performance drop for models like GenScore and Pafnucy indicates that their high scores on the original benchmark were largely driven by data memorization [40]. In contrast, the GEMS (Graph neural network for Efficient Molecular Scoring) model, which leverages a sparse graph architecture and transfer learning from language models, maintained high performance when trained and evaluated on the cleaned data, demonstrating genuine generalization capability [40] [12]. Similarly, models retrained on LP-PDBBind showed consistently better performance on the truly independent BDB2020+ dataset [42].

Table 2: Ablation Study Results for the GEMS Model

Model Variant	Input Data	Prediction Performance on CASF	Interpretation
GEMS (Full Model)	Protein and Ligand Structures	High	Predictions are based on genuine understanding of protein-ligand interactions.
GEMS (Ablated)	Ligand Information Only	Failed to produce accurate predictions	Confirms model does not rely solely on ligand memorization.
Search-by-Similarity Algorithm	Training Set Affinity Labels	Competitive with some published models (R=0.716)	Demonstrates that data leakage alone can achieve deceptively good results.

The ablation study for GEMS confirms that its predictive power collapses when critical protein information is omitted, suggesting its performance is based on a genuine understanding of interactions rather than exploiting dataset biases [40].

The Scientist's Toolkit: Key Research Reagents

To facilitate the adoption of robust benchmarking practices, the following table details essential datasets, models, and tools discussed in this paper.

Table 3: Essential Research Reagents for Robust Affinity Model Development

Reagent / Resource	Type	Primary Function	Key Characteristic / Application
PDBbind CleanSplit [40]	Curated Dataset	Training and evaluation with minimized data leakage.	Structure-based filtering removes complexes similar to CASF test set and internal redundancies.
LP-PDBBind [42]	Curated Dataset	Training and evaluation with minimized data leakage.	Similarity-controlled splits for proteins and ligands; includes non-covalent binders only.
CASF Benchmark [40]	Benchmark Suite	Standard test for scoring power.	Requires use with clean training splits (like CleanSplit) for valid generalization assessment.
BDB2020+ [42]	Independent Test Set	True external validation for trained models.	Comprised of BindingDB entries post-2020, filtered for similarity to training data.
GEMS Model [40]	Graph Neural Network	Binding affinity prediction.	Sparse graph modeling with transfer learning; demonstrates high generalization on clean data.
CORDIAL Model [44]	Deep Learning Framework	Generalizable affinity ranking via interaction-only features.	Uses distance-dependent physicochemical interaction signatures, avoiding structure parameterization.
BASE Web Service [41]	Web Tool	Provides bias-reduced affinity prediction datasets.	Allows users to download datasets split by customizable protein/ligand similarity cutoffs.

The benchmarking experiments conducted on CleanSplit versus standard splits deliver a clear and critical message: larger models will not fix biased benchmarks [43]. The performance inflation observed in many state-of-the-art models is a direct artifact of data leakage, not superior learning of underlying biophysics. The adoption of rigorously cleaned datasets, such as PDBbind CleanSplit and LP-PDBBind, along with more stringent validation protocols like leave-superfamily-out (LSO) [44], is essential for accurately measuring progress and developing models that generalize to novel targets in real-world drug discovery. For the field to move forward, structure-level filtering, leakage-aware splits, and independent validation must become standard practice [43]. The tools and methodologies outlined in this whitepaper provide a pathway to reset the baseline for what constitutes true generalization in binding affinity prediction.

The field of computational drug design relies on accurate scoring functions to predict the binding affinity of protein-ligand interactions. However, a pervasive issue of train-test data leakage has severely inflated the performance metrics of deep-learning models, leading to an overestimation of their generalization capabilities [4]. This case study examines how the Graph Neural Network for Efficient Molecular Scoring (GEMS) model maintains state-of-the-art performance when trained on PDBbind CleanSplit, a rigorously curated dataset that eliminates data leakage and internal redundancies. When existing top-performing models were retrained on CleanSplit, their performance dropped substantially, revealing that their previously reported high scores were largely driven by memorization rather than genuine understanding of protein-ligand interactions [4]. In contrast, GEMS demonstrates robust generalization to strictly independent test datasets, establishing a new standard for reliable binding affinity prediction in structure-based drug design.

Accurate prediction of protein-ligand binding affinities is crucial for structure-based drug design (SBDD). While deep learning models have shown promising results in benchmark studies, their real-world performance has been disappointing. This performance gap has been attributed to train-test data leakage between the PDBbind database (used for training) and the Comparative Assessment of Scoring Functions (CASF) benchmark datasets (used for evaluation) [4].

Alarmingly, studies have shown that some models perform comparably well on CASF benchmarks even after omitting all protein or ligand information from their input data, suggesting they exploit dataset biases rather than learning genuine protein-ligand interactions [4]. This memorandum effect has obscured the true generalization capabilities of affinity prediction models, creating a critical need for better dataset curation and more robust model architectures.

Methodology: Addressing Data Bias with PDBbind CleanSplit

Structure-Based Filtering Algorithm

To address the data leakage problem, researchers developed a novel structure-based clustering algorithm that identifies and removes similarities between training and test datasets [4]. This algorithm employs a multimodal approach to assess complex similarity:

• Protein similarity: Calculated using TM-scores [4]
• Ligand similarity: Calculated using Tanimoto scores [4]
• Binding conformation similarity: Calculated using pocket-aligned ligand root-mean-square deviation (r.m.s.d.) [4]

This comprehensive approach can identify complexes with similar interaction patterns even when proteins have low sequence identity, addressing limitations of traditional sequence-based filtering methods.

PDBbind CleanSplit Creation

The filtering process involved two critical steps to ensure dataset integrity:

• Eliminating train-test leakage: All training complexes closely resembling any CASF test complex were removed, including those with ligands identical to those in the test set (Tanimoto > 0.9). This step excluded 4% of training complexes and ensured test ligands were never encountered during training [4].
• Reducing internal redundancy: Similarity clusters within the training dataset itself were identified and resolved through iterative removal of redundant complexes, eliminating an additional 7.8% of training complexes [4].

The resulting PDBbind CleanSplit dataset provides a foundation for robust model training and reliable evaluation of generalization capabilities.

Experimental Validation Protocol

To validate the effectiveness of CleanSplit, researchers implemented a rigorous experimental protocol:

• Model retraining: State-of-the-art models (GenScore and Pafnucy) were retrained on both the original PDBbind dataset and the PDBbind CleanSplit [4].
• Performance benchmarking: Models were evaluated on the CASF-2016 benchmark using standard metrics, including Pearson R correlation and root-mean-square error (r.m.s.e.) [4].
• Ablation studies: The GEMS model was tested with protein nodes omitted to verify that predictions were based on genuine protein-ligand interactions rather than ligand memorization [4].

Table: PDBbind CleanSplit Filtering Impact

Filtering Criteria	Complexes Removed	Impact on Dataset
Train-test similarity	4% of training set	Eliminates direct memorization path
Internal redundancies	7.8% of training set	Reduces overfitting potential
Total reduction	~11.8% of training set	Creates more diverse training basis

The GEMS Model Architecture

Sparse Graph Representation

GEMS utilizes a sparse graph modeling approach to represent protein-ligand interactions. This architecture efficiently captures the essential features of molecular complexes while maintaining computational efficiency. The sparse graph structure focuses on relevant atomic interactions rather than processing entire molecular structures uniformly, enabling the model to learn meaningful physicochemical relationships rather than superficial patterns.

Transfer Learning Integration

A key innovation in GEMS is the incorporation of transfer learning from language models. This approach leverages pre-trained representations from protein language models, allowing GEMS to benefit from evolutionary information and sequence patterns learned from vast biological databases. This transfer learning component enhances the model's ability to generalize to novel protein-ligand complexes not seen during training.

GEMS Model Architecture: Integrating Sparse Graph and Language Models

Experimental Results and Performance Analysis

Impact of CleanSplit on Existing Models

Retraining existing models on PDBbind CleanSplit revealed the substantial impact of data leakage on previously reported performance metrics:

• Performance degradation: Both GenScore and Pafnucy showed marked performance drops when trained on CleanSplit compared to the original PDBbind dataset [4]
• Memorization effect: The performance decline confirmed that these models had relied on data leakage and structural similarities rather than learning fundamental principles of molecular interactions [4]

GEMS Performance on CleanSplit

In contrast to existing models, GEMS maintained high prediction accuracy when trained on PDBbind CleanSplit:

• State-of-the-art performance: GEMS achieved competitive results on the CASF-2016 benchmark despite the reduced data leakage [4]
• Robust generalization: The model demonstrated consistent performance on strictly independent test datasets, confirming its genuine understanding of protein-ligand interactions [4]
• Ablation study validation: When protein nodes were omitted from the graph, GEMS failed to produce accurate predictions, confirming that its performance derives from analyzing protein-ligand interactions rather than memorizing ligand properties [4]

Table: Comparative Model Performance on CASF-2016 Benchmark

Model	Training Dataset	Pearson R	r.m.s.e.	Generalization Assessment
GenScore	Original PDBbind	High (reported)	Low (reported)	Overestimated due to data leakage
GenScore	PDBbind CleanSplit	Substantially lower	Substantially higher	True performance revealed
Pafnucy	Original PDBbind	High (reported)	Low (reported)	Overestimated due to data leakage
Pafnucy	PDBbind CleanSplit	Substantially lower	Substantially higher	True performance revealed
GEMS	PDBbind CleanSplit	High (maintained)	Low (maintained)	Genuine generalization capability

Implications for Structure-Based Drug Design

The development of GEMS and the PDBbind CleanSplit dataset has significant implications for computational drug discovery:

Enabling Generative AI Applications

Generative models like RFdiffusion and DiffSBDD can create novel protein-ligand interactions but lack accurate affinity prediction to identify therapeutically promising candidates [4]. GEMS addresses this critical bottleneck by providing reliable binding affinity predictions for generated complexes, enabling more effective virtual screening of generative AI outputs.

New Standards for Model Evaluation

The data leakage issues identified in this research necessitate a reevaluation of benchmarking practices in computational drug design. PDBbind CleanSplit establishes a new standard for training and evaluation that prevents inflated performance metrics and ensures more realistic assessment of model generalization.

PDBbind CleanSplit Creation Workflow

Research Reagent Solutions

Table: Essential Research Materials and Computational Tools

Resource	Type	Function in Research
PDBbind Database	Data Resource	Primary source of protein-ligand complexes with experimental binding affinity data [4]
CASF Benchmark	Evaluation Framework	Standard benchmark sets for comparative assessment of scoring functions [4]
CleanSplit Algorithm	Software Tool	Structure-based filtering algorithm to detect and remove dataset similarities and redundancies [4]
Graph Neural Network Framework	Modeling Architecture	Deep learning framework for sparse graph representation of protein-ligand complexes [4]
Protein Language Models	Pre-trained Models	Source of transfer learning for evolutionary and sequence pattern information [4]
Escher	Visualization Tool	Software for creating metabolic network maps and pathway visualizations [45]

The GEMS case study demonstrates that resolving data bias through rigorous dataset curation is essential for developing truly generalizable binding affinity prediction models. By addressing the critical issue of train-test data leakage with PDBbind CleanSplit and implementing a robust graph neural network architecture with transfer learning, GEMS sets a new standard for reliable performance assessment in computational drug design. This approach provides a more realistic foundation for developing scoring functions that can genuinely advance structure-based drug design, particularly as generative AI models create increasingly novel protein-ligand complexes. The maintained performance of GEMS when data leakage is eliminated represents a significant step toward more trustworthy and effective computational tools for drug discovery.

The generalization capability of computational models is paramount in data-driven fields such as structure-based drug design. However, standard benchmarking approaches often overestimate real-world performance due to undetected similarities between training and test datasets, a phenomenon known as data leakage [4]. This whitepaper introduces Similarity-Stratified Analysis, a methodological framework designed to quantify and address this vulnerability by systematically evaluating model performance across carefully defined similarity strata.

The urgency of this approach is underscored by recent research revealing that nearly 49% of complexes in widely used Comparative Assessment of Scoring Function (CASF) benchmarks share striking similarities with complexes in the PDBbind training set [4]. This substantial data leakage has led to inflated performance metrics and overoptimistic assessments of model generalization. Similarity-Stratified Analysis provides the technical foundation for a more rigorous, transparent, and realistic evaluation paradigm essential for deploying reliable affinity prediction models in real-world drug discovery applications.

The Data Leakage Problem in Affinity Prediction

Data leakage occurs when information from outside the training dataset inadvertently influences the model, creating an overoptimistic assessment of its predictive capabilities. In structural bioinformatics, this manifests primarily through structural similarities between protein-ligand complexes in training and test sets.

Quantifying the Data Leakage

Recent investigations have revealed extensive data leakage in standard benchmarks. A structure-based clustering analysis identified concerning similarities between the PDBbind training set and CASF benchmark complexes [4]:

Similarity Metric	Threshold Value	Percentage of CASF Complexes Affected
Protein Similarity (TM-score)	> 0.7	49%
Ligand Similarity (Tanimoto)	> 0.9	Significant portion
Binding Conformation (pocket-aligned RMSD)	Low values	49%

This analysis identified nearly 600 high-similarity pairs between PDBbind training and CASF complexes, meaning nearly half of the test complexes did not present genuinely novel challenges to trained models [4]. Alarmingly, some models achieved competitive benchmark performance even when critical input information was omitted, suggesting they relied on memorization and exploitation of structural similarities rather than learning fundamental protein-ligand interactions [4].

Consequences for Model Generalization

The practical consequences of this data leakage are substantial. When top-performing affinity prediction models were retrained on a cleaned dataset (PDBbind CleanSplit) with reduced data leakage, their performance dropped markedly [4]:

Model Type	Performance on Standard Benchmark	Performance on CleanSplit	Performance Drop
GenScore	Excellent	Substantially reduced	Marked
Pafnucy	Excellent	Substantially reduced	Marked
Simple Search Algorithm	Competitive with published models	N/A	Demonstrates benchmark vulnerability

This performance degradation reveals that previously reported impressive results were largely driven by data leakage rather than genuine learning of protein-ligand interactions [4].

Similarity-Stratified Analysis Methodology

Similarity-Stratified Analysis provides a systematic framework to address data leakage by grouping test cases into similarity bins based on their relationship to the training data.

Multimodal Similarity Assessment

Effective stratification requires a combined assessment across multiple structural dimensions. The following multimodal approach has demonstrated effectiveness in identifying data leakage [4]:

Figure 1: Multimodal similarity assessment workflow for stratifying protein-ligand complexes.

Experimental Protocol for Similarity Stratification

The following table outlines the complete experimental protocol for implementing Similarity-Stratified Analysis:

Protocol Step	Technical Specification	Implementation Details
Dataset Preparation	Apply structure-based filtering	Use algorithms like PDBbind CleanSplit to remove redundant complexes and ensure strict train-test separation [4]
Similarity Calculation	Compute multimodal similarity metrics	Calculate TM-score (protein), Tanimoto coefficient (ligand), and pocket-aligned RMSD (binding conformation) for all train-test pairs [4]
Threshold Definition	Establish similarity boundaries	Set thresholds for high (>0.7 TM-score, >0.9 Tanimoto), medium, and low similarity bins based on distribution analysis
Stratification	Assign test cases to similarity bins	Group each test case into appropriate bin based on its maximum similarity to any training complex
Performance Evaluation	Calculate bin-specific metrics	Evaluate model performance (RMSD, R², etc.) separately within each similarity bin
Analysis	Compare cross-bin performance	Identify performance degradation patterns across similarity strata

This protocol specifically addresses the limitations of sequence-based analysis by incorporating structural metrics that can identify complexes with similar interaction patterns even when proteins have low sequence identity [4].

Visualization of Performance Across Strata

The results of Similarity-Stratified Analysis can be visualized to immediately communicate model generalization capabilities:

Figure 2: Interpretation of model performance across similarity strata.

Case Study: Implementation in Binding Affinity Prediction

A recent study on binding affinity prediction provides a compelling case study for Similarity-Stratified Analysis [4]. The researchers developed a graph neural network for efficient molecular scoring (GEMS) and rigorously evaluated its generalization using similarity-aware methodology.

Experimental Workflow

The implementation followed a structured approach to ensure robust evaluation:

Figure 3: Case study workflow for rigorous generalization assessment.

Quantitative Results

The GEMS model maintained high performance on CASF benchmarks even when trained on the cleaned dataset, in contrast to other models that showed significant performance drops [4]:

Model	Training Dataset	CASF2016 Benchmark Performance	Performance on Novel Complexes
GenScore	Original PDBbind	Excellent	Not reported
GenScore	PDBbind CleanSplit	Substantially reduced	Significant performance drop
Pafnucy	Original PDBbind	Excellent	Not reported
Pafnucy	PDBbind CleanSplit	Substantially reduced	Significant performance drop
GEMS	PDBbind CleanSplit	State-of-the-art	Maintained high performance

Crucially, ablation studies demonstrated that GEMS failed to produce accurate predictions when protein nodes were omitted from the graph, suggesting its predictions were based on genuine understanding of protein-ligand interactions rather than exploiting data leakage [4].

Research Reagent Solutions

Implementing Similarity-Stratified Analysis requires specific computational tools and resources. The following table details essential research reagents for proper implementation:

Research Reagent	Function/Significance	Implementation Notes
PDBbind Database	Comprehensive collection of protein-ligand complexes with binding affinity data	Foundation for training and benchmarking; requires filtering [4]
CASF Benchmark	Standardized benchmark for scoring function evaluation	Contains known data leakage issues; requires stratification [4]
Structure-Based Filtering Algorithm	Identifies and removes similar complexes using multimodal metrics	Essential for creating CleanSplit datasets; uses TM-score, Tanimoto, and RMSD [4]
TM-score Algorithm	Measures protein structural similarity independent of length	More reliable than sequence alignment for identifying similar binding sites [4]
Tanimoto Coefficient	Calculates 2D molecular similarity between ligands	Identifies cases where similar ligands appear in both training and test sets [4]
Pocket-Aligned RMSD	Quantifies similarity of ligand binding conformation	Captures similar binding modes despite protein sequence differences [4]
Graph Neural Networks (GNNs)	Advanced architecture for modeling protein-ligand interactions	Can leverage sparse graph representations for improved generalization [4]
Language Model Embeddings	Transfer learning from protein and molecular language models	Enhances model understanding of structural and functional relationships [4]

These reagents collectively enable the development and rigorous evaluation of affinity prediction models with genuinely validated generalization capabilities.

Implications for Drug Discovery

Similarity-Stratified Analysis has profound implications for computational drug discovery. By providing a more realistic assessment of model capabilities, it addresses critical bottlenecks in structure-based drug design.

Generative AI models like RFdiffusion and DiffSBDD can create vast libraries of novel protein-ligand interactions, but their potential has been limited by the absence of accurate affinity prediction models that generalize to these novel structures [4]. Similarity-Stratified Analysis enables the development of reliably evaluated scoring functions that can identify therapeutically promising interactions from generated libraries.

Furthermore, this approach addresses broader cognitive biases in pharmaceutical R&D, particularly confirmation bias - the tendency to overweight evidence consistent with favored beliefs [46]. By objectively quantifying performance across similarity strata, Similarity-Stratified Analysis provides evidence-based guardrails against overoptimism about model capabilities, potentially increasing R&D efficiency and contributing to more equitable healthcare through more reliably predicted drug-target interactions.

Similarity-Stratified Analysis represents a methodological advancement in the evaluation of computational models, particularly for affinity prediction in drug discovery. By systematically accounting for structural similarities between training and test data, this approach addresses pervasive data leakage problems that have inflated performance metrics and hampered real-world application.

The framework provides technical guidance for implementing multimodal similarity assessment, creating properly filtered datasets, and interpreting performance across similarity strata. As the field progresses toward more complex modeling approaches, including generative AI for drug design, rigorous evaluation methodologies like Similarity-Stratified Analysis will be essential for translating computational advances into genuine therapeutic breakthroughs.

Adopting this analytical approach will enable researchers, scientists, and drug development professionals to make more informed decisions about model selection and application, ultimately accelerating the development of effective treatments through more reliable computational predictions.

The accurate prediction of molecular binding affinity is a cornerstone of computational drug design. While deep learning models, including Convolutional Neural Networks (CNNs), Graph Neural Networks (GNNs), and attention-based mechanisms, have shown promising results, their generalization capabilities are often compromised by inherent data biases. This technical review provides a comparative analysis of these architectures, framed within the critical context of data bias and generalization in affinity prediction. We systematically evaluate architectural strengths, quantitative performance, and sensitivity to dataset construction, highlighting how advanced GNNs and hybrid models address bias mitigation through sophisticated data splitting and integrative learning. The analysis underscores that model selection is profoundly influenced by the data curation strategy, with recent benchmarks revealing significant performance inflation in existing literature due to train-test leakage.

In structure-based drug design (SBDD), the primary goal is to identify small molecules that bind with high affinity and specificity to protein targets. Classical scoring functions, often based on force-fields or empirical data, are computationally intensive and exhibit limited accuracy [4]. Deep learning offers a transformative alternative, with CNNs, GNNs, and attention-based architectures emerging as leading approaches for predicting protein-ligand interactions.

However, a critical challenge persists: the reported high performance of these models often masks poor generalization to truly independent test sets. This gap is frequently driven by data biases, such as train-test leakage and dataset redundancies, which inflate benchmark metrics [4] [15]. For instance, models trained on the common PDBbind database and evaluated on the Comparative Assessment of Scoring Functions (CASF) benchmark often encounter nearly identical complexes in both sets, enabling prediction via memorization rather than genuine learning of interactions [4]. This review dissects how different neural architectures perform when these biases are rigorously controlled, providing a realistic comparison of their capabilities in affinity prediction.

Convolutional Neural Networks (CNNs)

CNNs process data structured on a grid, making them suitable for interpreting 3D structures of protein-ligand complexes represented as volumetric voxels.

• Core Principle: CNNs apply convolutional filters to extract hierarchical, translation-invariant local features from input data [47]. In affinity prediction, the input is often a 3D grid representing the protein binding pocket, with channels encoding atomic properties or chemical features.
• Strengths: CNNs excel at capturing local spatial patterns and are highly efficient for processing structured 3D data. Models like 3D-CNNs and Pafnucy have demonstrated strong performance on binding affinity tasks [15] [47].
• Limitations and Biases: CNNs are highly sensitive to variations in input data, such as spatial rotations and intensity changes. Their performance can degrade significantly when test data differs from the training distribution in terms of scanner type, resolution, or spatial alignment [47]. This indicates a bias towards the specific acquisition parameters of the training set. Data augmentation (e.g., rotation, scaling, intensity manipulation) can improve robustness, but it is often limited to a predefined parameter space and may not fully address the underlying generalization problem [47].

Graph Neural Networks (GNNs)

GNNs operate on graph-structured data, offering a natural representation for molecules where atoms are nodes and bonds are edges.

• Core Principle: GNNs learn node representations by iteratively aggregating information from a node's neighbors. In Graph Attention Networks (GATs), a key variant, an attention mechanism assigns varying importance to different neighbors during aggregation [48]. This is computed as:
  • Feature transformation: ( \mathbf{h}i' = \mathbf{W}\mathbf{h}i )
  • Unnormalized attention score between node ( i ) and neighbor ( j ): ( e{ij} = \text{LeakyReLU}(\mathbf{a}^T (\mathbf{h}i' + \mathbf{h}_j')) )
  • Score normalization: ( \alpha{ij} = \frac{\exp(e{ij})}{\sum{k \in \mathcal{N}(i)} \exp(e{ik})} )
  • Output features: ( \mathbf{h}i'' = \sigma\left(\sum{j \in \mathcal{N}(i)} \alpha{ij} \mathbf{h}j'\right) ) Multi-head attention is often used to stabilize learning and capture diverse relational aspects [48].
• Strengths: GNNs directly model the relational structure of molecules, which is innately graph-like. They are less sensitive to the global spatial pose of the molecule compared to CNNs and can more effectively learn the topological rules of molecular interactions.
• Addressing Bias: Advanced GNNs like GEMS (Graph neural network for Efficient Molecular Scoring) leverage sparse graph modeling and transfer learning from language models to achieve robust generalization on strictly independent test sets, such as those defined by the PDBbind CleanSplit protocol [4]. Their architecture reduces reliance on superficial statistical cues.

Attention-Based and Hybrid Architectures

Attention mechanisms enable models to dynamically focus on the most relevant parts of the input for a given task.

• Core Principle in Transformers: The self-attention mechanism computes a weighted sum of values, where weights are determined by the compatibility of a query with corresponding keys. In a multi-head setup, multiple such operations run in parallel, allowing the model to attend to information from different representation subspaces [49]. Hybrid models, such as CNN-Transformer architectures, combine CNN-based local feature extraction with the superior temporal or relational modeling of attention [49] [50].
• Strengths: Attention provides significant interpretability by revealing which input elements (e.g., specific protein residues or ligand atoms) the model deems important [50]. In hybrid models like AttentionMGT-DTA, attention is used to integrate multi-modal information (e.g., molecular graph and protein pocket graph) and to model the interaction strength between drug atoms and protein residues [50].
• Limitations and Bias Context: The flexibility of attention comes with increased computational complexity and a higher number of parameters, raising the risk of overfitting, especially on biased datasets [48] [49]. Furthermore, analyzing attention scores in LLMs has shown that bias can be localized to specific layers, requiring targeted interventions like attention scaling for mitigation [51].

Quantitative Performance Analysis in Affinity Prediction

The performance of these architectures must be evaluated under bias-controlled conditions. The creation of PDBbind CleanSplit—a dataset curated to eliminate train-test leakage and internal redundancies—provides a rigorous benchmark [4] [5]. Retraining models on CleanSplit reveals their true generalization capability.

Table 1: Comparative Model Performance on Standard vs. CleanSplit PDBbind Data

Model Architecture	Representative Model	Reported Performance (Standard Split)	Performance (CleanSplit)	Key Metric
3D CNN	Pafnucy [15]	High (Overestimated)	Substantial Drop	Binding Affinity RMSE
GNN	GenScore [4]	High (Overestimated)	Substantial Drop	Binding Affinity RMSE
Advanced GNN	GEMS [4]	-	Maintains High Performance	Binding Affinity RMSE
Hybrid (GNN + Attention)	AttentionMGT-DTA [50]	Outperformed Baselines	-	Affinity Prediction Accuracy

Table 2: Computational Efficiency of Attention Variants (Non-Domain Specific)

Attention Mechanism	Top-1 Accuracy	Inference Time (Relative)	Key Characteristic
Baseline Multi-Head	85.05%	1.0x (Baseline)	Bidirectional context [49]
Causal Attention	>84%	0.17x (83% reduction)	Enforces temporal causality [49]
Sparse Attention	>84%	0.25x (75% reduction)	Local windowing for efficiency [49]

The data in Table 1 demonstrates that the previously high performance of many CNN and GNN models was largely driven by data leakage. When this bias is removed via CleanSplit, their performance drops markedly. In contrast, architectures like GEMS, which are designed for generalization, maintain robustness. This underscores that the choice of model is secondary to the rigor of the data split in mitigating bias. Furthermore, as shown in Table 2, different attention mechanisms offer trade-offs between accuracy and computational efficiency, which is a key consideration for large-scale virtual screening.

Experimental Protocols for Bias-Aware Model Evaluation

To ensure reliable and generalizable affinity prediction, experimental protocols must explicitly address data bias. The following methodology outlines a robust pipeline for model training and evaluation.

Experimental Workflow for Bias Mitigation

Data Curation: The PDBbind CleanSplit Protocol

The foundational step is creating a training dataset free of data leakage, following the PDBbind CleanSplit protocol [4] [5].

• Structure-Based Filtering Algorithm: This algorithm identifies and removes structurally similar complexes between the training set (PDBbind) and the test benchmark (CASF) using a multi-modal similarity assessment:
  • Protein Similarity: Calculated using TM-scores [4].
  • Ligand Similarity: Calculated using Tanimoto scores based on molecular fingerprints [4].
  • Binding Conformation Similarity: Calculated using pocket-aligned ligand root-mean-square deviation (R.M.S.D.) [4].
• Application of Thresholds: Training complexes that exceed predefined similarity thresholds with any CASF test complex are removed. This process eliminated nearly 600 similar pairs, involving 49% of all CASF complexes, in the original data [4].
• Redundancy Reduction: The algorithm further identifies and resolves similarity clusters within the training dataset itself, removing ~7.8% of complexes to discourage memorization and encourage genuine learning of interactions [4].

Model Training and Evaluation

After obtaining a rigorously split dataset, the standard training and evaluation cycle proceeds.

• Input Representation:
  • For GNNs: Represent the protein-ligand complex as a graph. Nodes are protein residues and ligand atoms, edged by connectivity or spatial proximity [50] [4].
  • For CNNs: Represent the binding pocket as a 3D voxel grid, with channels encoding atom types or chemical features [15].
  • For Hybrid Models: Use multi-modal input, e.g., a molecular graph for the drug and a separate graph for the protein binding pocket [50].
• Training Regime: Models are trained on the filtered training set from CleanSplit. Techniques like early stopping and dropout are essential to prevent overfitting, especially for larger models like GATs and Transformers [48].
• Evaluation: The trained model is evaluated on the strictly independent test set (e.g., the CASF benchmark). Key metrics include Root-Mean-Square Error (RMSE) for affinity prediction and Pearson correlation coefficient.

The Scientist's Toolkit: Essential Research Reagents

Table 3: Key Resources for Bias-Aware Affinity Prediction Research

Resource Name	Type	Function in Research
PDBbind Database [4] [15]	Data	Primary source of experimental protein-ligand structures and binding affinities for training.
CASF Benchmark [4] [15]	Data	Standard benchmark set for evaluating scoring functions; must be used with a clean split.
PDBbind CleanSplit [4] [5]	Data/Protocol	A curated training dataset and splitting method that eliminates data leakage with CASF.
Graph Neural Network (GNN)	Model Architecture	Learns directly from graph-structured molecular data.
Graph Attention Network (GAT) [48]	Model Architecture	A GNN variant that uses attention to weight neighbor importance, improving interpretability.
ATLAS [51]	Algorithm	A technique to localize and mitigate bias in model layers via attention score analysis.
NeuBM [52]	Algorithm	Mitigates model bias in GNNs through neutral input calibration, helpful for class imbalance.

Analysis of Bias Localization and Mitigation Strategies

Understanding where bias manifests within models is crucial for developing effective mitigation strategies.

Bias Localization and Mitigation

• Bias Localization in Models: Studies on Large Language Models (LLMs) have shown that bias often concentrates in specific layers, typically the last third of the network. Techniques like ATLAS (Attention-based Targeted Layer Analysis and Scaling) can localize bias to these layers by analyzing attention scores and then mitigate it by scaling attention in the identified layers [51].
• Bias from Class Imbalance: In GNNs, class imbalance can lead to model bias against minority classes. Methods like NeuBM (Neutral Bias Mitigation) address this by using a dynamically updated neutral graph to estimate and correct the model's inherent biases, recalibrating predictions without altering the core architecture [52].
• The Primacy of Data Curation: While algorithmic interventions are valuable, the most effective strategy for mitigating bias in affinity prediction is rigorous data curation. The profound performance drop observed when models are trained on CleanSplit confirms that resolving data bias is the most critical step for improving generalization [4] [15].

The comparative analysis of GNNs, CNNs, and attention-based approaches reveals that architectural choice is a secondary factor to data bias management in building generalizable affinity prediction models. CNNs, while powerful for spatial feature extraction, are sensitive to input variations. GNNs offer a more natural representation for molecules, and attention mechanisms provide valuable interpretability and flexible integration of multi-modal data.

However, the recent establishment of bias-aware benchmarks like PDBbind CleanSplit has fundamentally shifted the evaluation landscape. It has demonstrated that the previously reported high performance of many models was significantly inflated. The path forward for the field lies in the adoption of such rigorous data splitting protocols, combined with architectures designed for generalization, such as sparse GNNs utilizing transfer learning. Future work must continue to intertwine advanced model design with uncompromising data curation to deliver reliable tools for computational drug discovery.

The application of artificial intelligence and machine learning in drug discovery has created a paradigm shift, offering the potential to rapidly identify hit compounds and optimize lead candidates. However, a significant challenge persists: models that demonstrate exceptional performance on standardized benchmarks often fail unpredictably when applied to novel, real-world drug discovery scenarios [53]. This generalization gap represents a critical roadblock in the transition from benchmark performance to prospective applications, largely driven by pervasive data biases and inadequate validation methodologies that fail to capture the complexity of real-world biological systems.

Recent analyses have revealed that the underlying issue stems from fundamental flaws in how models are trained and evaluated. Data leakage—where information from the test set inadvertently influences the training process—has been identified as a primary culprit, creating an illusion of competence that evaporates when models face truly novel chemical spaces or protein families [4]. For instance, when models are trained on the PDBbind database and evaluated on the Comparative Assessment of Scoring Function (CASF) benchmark, nearly half of the test complexes have highly similar counterparts in the training data, enabling prediction through memorization rather than genuine understanding of protein-ligand interactions [4] [5].

This whitepaper examines the sources of this validation crisis, presents rigorous frameworks for real-world model assessment, and provides experimental protocols to bridge the gap between benchmark performance and successful prospective application in drug discovery pipelines.

The Data Bias Crisis in Affinity Prediction

Documented Cases of Benchmark Overestimation

The extent of the data bias problem has been quantitatively demonstrated through recent studies that implemented rigorous data separation protocols. When models were retrained on carefully curated datasets that eliminated train-test leakage, performance metrics dropped substantially, revealing that previously reported achievements were largely artifacts of biased evaluation practices.

Table 1: Impact of Data Leakage on Model Performance

Model	Reported Performance (Original Benchmark)	Performance (CleanSplit)	Performance Drop	Key Finding
GenScore	Excellent CASF performance	Substantially reduced	Marked	Previous performance driven by data leakage
Pafnucy	High benchmark accuracy	Significantly lower	Significant	Inability to generalize to novel complexes
Search Algorithm (5-nearest neighbors)	Competitive (R=0.716)	Not applicable	Benchmark flaw	Simple similarity matching achieves competitive results

The search algorithm experiment provides particularly compelling evidence of the benchmark contamination problem. When researchers devised a simple algorithm that predicts binding affinity by identifying the five most similar training complexes and averaging their affinity labels, it achieved competitive performance compared to published deep-learning scoring functions (Pearson R = 0.716, r.m.s.e. comparable to established models) [4]. This indicates that the CASF benchmark can be gamed through structural similarity matching rather than genuine understanding of binding principles.

Root Causes of Bias in Drug Discovery Datasets

The inflation of benchmark performance stems from several structural issues in dataset construction and utilization:

• Train-Test Data Leakage: The PDBbind database and CASF benchmark datasets share a high degree of structural similarity, with nearly 600 detected similarities between training and test complexes, affecting 49% of all CASF complexes [4]. This enables models to perform well through memorization of similar structures rather than learning fundamental binding principles.

• Dataset Redundancy: Within the training data itself, approximately 50% of all training complexes belong to similarity clusters, creating internal redundancies that enable models to settle for easily attainable local minima in the loss landscape through structure-matching rather than developing robust generalization capabilities [4].

• Assay Type Confusion: Real-world compound activity data exhibits two distinct patterns—virtual screening (VS) assays with diverse compound libraries and lead optimization (LO) assays with congeneric compound series [54]. Benchmark datasets that fail to distinguish between these scenarios produce misleading performance estimates, as models may perform well on one task type while failing on the other.

Frameworks for Real-World Benchmarking

The CARA Benchmark: Accounting for Real-World Data Characteristics

The Compound Activity benchmark for Real-world Applications (CARA) addresses critical limitations in existing benchmarks by incorporating the actual characteristics and distribution patterns of real-world compound activity data [54]. This framework introduces several key innovations:

Table 2: CARA Benchmark Design Principles

Design Principle	Implementation	Addresses
Assay Type Distinction	Separate Virtual Screening (VS) and Lead Optimization (LO) assays	Different compound distribution patterns in real-world screening vs optimization
Realistic Data Splitting	Scheme designed to avoid overestimation of model performance	Biased distribution of current real-world compound activity data
Few-Shot & Zero-Shot Evaluation	Scenarios with limited or no task-related training data	Practical application settings where historical data is scarce
Multiple Evaluation Metrics	Beyond simple binary classification to include ranking importance	Real-world prioritization needs in drug discovery

The CARA framework recognizes that compounds from different assays exhibit distinct distribution patterns: VS assays show diffused, widespread compound distributions reflecting diverse screening libraries, while LO assays demonstrate aggregated, concentrated patterns resulting from congeneric compound series designed around shared scaffolds [54]. This distinction is critical because models may perform differently on these fundamentally different prediction tasks.

PDBbind CleanSplit: Eliminating Data Leakage

The PDBbind CleanSplit dataset introduces a rigorous structure-based filtering algorithm to address the critical issue of train-test data leakage [4] [5]. The filtering approach employs a multimodal assessment of complex similarity:

• Protein Similarity: Measured using TM scores to identify structurally similar proteins [4]
• Ligand Similarity: Calculated using Tanimoto scores to detect similar compounds [4]
• Binding Conformation Similarity: Assessed through pocket-aligned ligand root-mean-square deviation (r.m.s.d.) [4]

The CleanSplit protocol applies conservative thresholds to exclude training complexes that remotely resemble any CASF test complex, ensuring that benchmark performance reflects genuine generalization capability rather than exploitation of structural similarities. This filtering removed 4% of training complexes due to high similarity with test complexes and an additional 7.8% to resolve internal redundancies [4].

Real-World Performance Validation Protocol

Brown's evaluation protocol for structure-based affinity prediction models establishes a rigorous framework that simulates real-world scenarios [53]. The key innovation is the exclusion of entire protein superfamilies and all associated chemical data from the training set, creating a challenging test of the model's ability to generalize to truly novel protein families. This approach answers the critical question: "If a novel protein family were discovered tomorrow, would our model be able to make effective predictions for it?" [53]

CleanSplit Creation and Validation Workflow

Experimental Protocols for Real-World Validation

Virtual Screening vs. Lead Optimization Validation

The CARA benchmark provides distinct validation protocols for Virtual Screening (VS) and Lead Optimization (LO) tasks, reflecting their different roles in the drug discovery pipeline [54]:

Virtual Screening Validation Protocol:

• Objective: Identify active compounds from large, diverse chemical libraries
• Data Characteristics: Diffused compound distribution with low pairwise similarities
• Evaluation Metric: Enrichment of active compounds in top-ranked predictions
• Training Strategy: Meta-learning and multi-task learning improve performance for VS tasks

Lead Optimization Validation Protocol:

• Objective: Rank congeneric compound series by activity
• Data Characteristics: Aggregated compound distribution with high pairwise similarities
• Evaluation Metric: Ranking accuracy and structure-activity relationship detection
• Training Strategy: Separate QSAR models per assay achieve decent performance

Generalizability-Focused Model Architecture

Brown's generalizable deep learning framework for structure-based protein-ligand affinity ranking introduces a task-specific architecture that addresses the generalization gap by constraining what the model can learn [53]. Instead of learning from the entire 3D structure of a protein and drug molecule, the model is restricted to learn only from a representation of their interaction space, which captures the distance-dependent physicochemical interactions between atom pairs [53].

Generalizable Model Architecture Approach

This constrained approach forces the model to learn transferable principles of molecular binding rather than structural shortcuts present in the training data that fail to generalize to new molecules [53]. The architecture provides an "inductive bias" that guides the model toward learning fundamental binding principles.

Prospective Validation Protocol

Rigorous prospective validation requires protocols that simulate real-world application scenarios:

Protein-Family-Level Splitting:

• Exclude entire protein superfamilies from training data
• Evaluate performance on held-out protein families
• Assesses model capability for novel target discovery

Temporal Splitting:

• Train on data available before a specific date
• Test on compounds discovered after that date
• Simulates real-world deployment scenarios

Chemical Space Coverage Assessment:

• Quantify chemical diversity of training and test sets
• Ensure representative coverage of relevant chemical space
• Identify potential blind spots in model capability

Implementation: The Researcher's Toolkit

Research Reagent Solutions

Table 3: Essential Resources for Real-World Validation

Resource	Type	Function in Validation	Key Features
CARA Benchmark	Dataset	Evaluate compound activity prediction	Distinguishes VS vs LO assays; realistic data splitting [54]
PDBbind CleanSplit	Curated Dataset	Eliminate train-test data leakage	Structure-based filtering; reduced redundancy [4] [5]
GEMS (Graph Neural Network)	Model Architecture	Generalizable affinity prediction	Sparse graph modeling; transfer learning from language models [4]
ChEMBL Database	Compound Activity Data	Source of real-world activity patterns	Millions of activity records; organized by assay type [54]
BindingDB	Binding Affinity Data	Experimental binding data	Ki, Kd, IC50 values; protein-ligand complexes [2]

Validation Workflow Implementation

Implementing robust real-world validation requires a systematic workflow that incorporates bias detection and mitigation:

Comprehensive Validation Workflow

The transition from impressive benchmark performance to genuine real-world utility in drug discovery requires a fundamental shift in validation methodologies. The research community must move beyond convenient but flawed benchmarking practices and adopt the rigorous frameworks outlined in this whitepaper. By implementing assay-distinguished benchmarks like CARA, eliminating data leakage through approaches like CleanSplit, designing generalizable model architectures focused on interaction principles, and employing rigorous evaluation protocols that simulate real-world scenarios, we can begin to close the generalization gap.

The path forward requires increased emphasis on prospective validation—testing models on truly novel targets and compound series that represent the actual challenges faced in drug discovery pipelines. Only through such rigorous and realistic validation can we build trustworthy AI systems that reliably accelerate the discovery of novel therapeutics and fulfill the promise of computational drug design.

Conclusion

The journey toward truly generalizable affinity prediction models requires a fundamental shift from relying on potentially flawed benchmarks to implementing rigorous, bias-aware methodologies. The synthesis of findings reveals that addressing data bias through protocols like PDBbind CleanSplit and similarity-aware evaluation is not merely an optimization but a necessity for realistic performance assessment. When combined with architecturally advanced models like GNNs that leverage transfer learning and sophisticated training techniques, the field can overcome its current generalization challenges. Future directions must focus on developing even more sophisticated data splitting protocols, creating larger and more diverse datasets that better represent real-world chemical space, and establishing standardized evaluation frameworks that explicitly account for similarity distribution. For biomedical research, these advances promise more reliable in silico screening, accelerating the identification of novel therapeutic candidates while reducing costly late-stage failures in drug development. The era of benchmarking on memorization is ending, making way for models that genuinely understand the structural principles of molecular recognition.

References

• 1. Binding Affinity Prediction: From Conventional to Machine ... [https://arxiv.org/html/2410.00709v2]
• 2. A survey on deep learning for drug-target binding prediction [https://pmc.ncbi.nlm.nih.gov/articles/PMC12451107/]
• 3. HPDAF: A Practical Tool for Predicting Drug-Target Binding ... [https://www.sciencedirect.com/science/article/abs/pii/S0223523425009183]
• 4. Resolving data bias improves generalization in binding ... [https://www.nature.com/articles/s42256-025-01124-5]
• 5. Resolving data bias improves generalization in binding ... [https://pubmed.ncbi.nlm.nih.gov/41143208/]
• 6. Drug-target binding affinity prediction with docking pose ... [https://research.ibm.com/publications/drug-target-binding-affinity-prediction-with-docking-pose-physics]
• 7. Machine Learning-Driven Methods for Nanobody Affinity ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC11618429/]
• 8. A workflow to create a high-quality protein–ligand binding ... [https://pubs.rsc.org/en/content/articlehtml/2025/dd/d4dd00357h]
• 9. A workflow to create a high-quality protein–ligand binding ... [https://pubs.rsc.org/en/content/articlelanding/2025/dd/d4dd00357h]
• 10. A workflow to create a high-quality protein–ligand binding ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC11967698/]
• 11. The State-of-the-Art in Data for Protein-Ligand Binding ... [https://www.linkedin.com/pulse/state-of-the-art-data-protein-ligand-binding-affinity-serhii-vakal-a0zrf]
• 12. Resolving data bias improves generalization in binding ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12552120/]
• 13. The importance of choosing a proper validation strategy in ... [https://www.sciencedirect.com/science/article/pii/S0003267025012322]
• 14. Clustering predicted structures at the scale of the known ... [https://www.nature.com/articles/s41586-023-06510-w]
• 15. Latent Biases in Machine Learning Models for Predicting ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC9850481/]
• 16. 3D molecular generative framework for interaction-guided ... [https://www.nature.com/articles/s41467-024-47011-2]
• 17. Evaluating Data Partitioning Strategies for Accurate ... [https://www.sciencedirect.com/science/article/pii/S2001037025004222]
• 18. CENsible: Interpretable Insights into Small-Molecule Binding ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC11200255/]
• 19. Resolving Data Bias Improves Generalization in Binding ... [https://zenodo.org/records/17190227]
• 20. Rethinking the generalization of drug target affinity ... [https://arxiv.org/abs/2504.09481]
• 21. Data splitting to avoid information leakage with DataSAIL [https://www.nature.com/articles/s41467-025-58606-8]
• 22. DeepDTAGen: a multitask deep learning framework for ... [https://www.nature.com/articles/s41467-025-59917-6]
• 23. Assessing the similarity of ligand binding conformations with ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC5159245/]
• 24. Similarity measures for ligand-binding conformations [https://www.brylinski.org/similarity-measures-ligand-binding-conformations]
• 25. Towards Interpretable Protein Structure Prediction with ... [https://arxiv.org/html/2503.08764v1]
• 26. A new benchmark for deep learning based affinity prediction [https://chemrxiv.org/engage/chemrxiv/article-details/68c47cf49008f1a46721c407]
• 27. A survey on large language models in biology and chemistry [https://www.nature.com/articles/s12276-025-01583-1]
• 28. Transfer learning on protein language models improves ... [https://www.nature.com/articles/s41598-025-21223-y]
• 29. Biophysics-based protein language models for ... [https://www.nature.com/articles/s41592-025-02776-2]
• 30. Chemical language models can generate biomolecules atom ... [https://pubs.rsc.org/en/content/articlehtml/2025/dd/d5dd00107b]
• 31. A roadmap for large language models in chemical research [https://engineering.cmu.edu/news-events/news/2025/06/24-roadmap-for-llms.html]
• 32. Multitask Learning 1997–2024: Part I Fundamentals [https://hdsr.mitpress.mit.edu/pub/7fcc3jhv]
• 33. Graph Coloring for Multi-Task Learning [https://arxiv.org/html/2509.16959v2]
• 34. Towards Understanding Multi-Task Learning ... [https://aclanthology.org/2025.coling-main.200/]
• 35. A Visual Explanation of Gradient Descent Methods ... [https://towardsdatascience.com/a-visual-explanation-of-gradient-descent-methods-momentum-adagrad-rmsprop-adam-f898b102325c/]
• 36. Explaining Adam & Momentum for Gradient Descent ... [https://medium.com/@paigecaskey/explaining-adam-momentum-for-gradient-descent-optimization-45c2dc6a9798]
• 37. Gradient Descent, Momentum and Adaptive Learning Rate [https://www.parasdahal.com/sgd-momentum-adaptive]
• 38. Exploring Gradient Descent Optimization Algorithms for ... [https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5082092]
• 39. Best practices for solving the cold start problem in AI projects [https://xenoss.io/blog/cold-start-problem-ai-projects]
• 40. Resolving data bias improves generalization in binding affinity ... [https://www.nature.com/articles/s42256-025-01124-5?error=cookies_not_supported&code=c0351dec-4fbf-4550-a998-911ba1b96498]
• 41. BASE: a web service for providing compound-protein binding affinity ... [https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-024-05968-3]
• 42. Leak Proof PDBBind: A Reorganized Dataset of Protein- ... [https://arxiv.org/html/2308.09639v2]
• 43. Grant Armstrong's Post [https://www.linkedin.com/posts/grantaarmstrong_larger-models-wont-fix-biased-benchmarks-activity-7396691574585466880-OJyB]
• 44. A generalizable deep learning framework for structure ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC12557521/]
• 45. Visualizing metabolic network dynamics through time-series ... [https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-020-3415-z]
• 46. How debunking biases in research and development ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC11252826/]
• 47. Investigation of biases in convolutional neural networks for ... [https://pmc.ncbi.nlm.nih.gov/articles/PMC9948839/]
• 48. Graph Neural Networks Part 2. Graph Attention ... [https://towardsdatascience.com/graph-neural-networks-part-2-graph-attention-networks-vs-gcns-029efd7a1d92/]
• 49. Comparative Analysis of Attention Mechanisms for ... [https://arxiv.org/html/2508.09996v1]
• 50. AttentionMGT-DTA: A multi-modal drug-target affinity ... [https://www.sciencedirect.com/science/article/pii/S089360802300641X]
• 51. Localizing and Mitigating Bias in Language Models [https://arxiv.org/abs/2410.22517]
• 52. NeuBM: Mitigating Model Bias in Graph Neural Networks ... [https://www.ijcai.org/proceedings/2025/315]
• 53. Vanderbilt scientist tackles key roadblock for AI in drug discovery [https://medschool.vanderbilt.edu/basic-sciences/2025/10/16/vanderbilt-scientist-tackles-key-roadblock-for-ai-in-drug-discovery/]
• 54. Benchmarking compound activity prediction for real-world ... [https://www.nature.com/articles/s42004-024-01204-4]