In our study, we utilize the benchmark dataset released by Le et al. [20], which contains 1324 electron transport proteins and 4569 general transport proteins. The data were initially taken from a previous study [17], and data from UniProt release-2018_05 (on 23-May-2018) [23] and Gene Ontology (GO) release-2018-05-01 [24] were also collected. Then, in order to avoid model overfitting, the data were removed the redundant sequences with similarities of more than 30%. To solve this binary classification problem, the dataset is randomly divided into cross-validation dataset and independent dataset in a ratio of 0.85:0.15. Table 1 shows the details of the datasets.

Using appropriate methods to extract protein characteristics is an important step to complete the task of classification. PSSM, which retains the evolutionary information of proteins in a matrix of L rows and 20 columns, is employed as the feature extraction approach in this work. PSSM was first introduced by Jones [25], which is a commonly used method in the field of bioinformatics. It has been used in a number of bioinformatics studies with positive results [26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37]. The PSSM profiles is derived from multiple sequence alignment and contains the evolutionary information of each residue in the protein sequence. Electron transport proteins belong to a class of proteins with a specific function. Compared with other protein families, the key evolutionary information of proteins can be captured by using PSSM matrix and related feature extraction method, and then the classifier can be used to effectively identify electron transport proteins.

FASTA sequences are searched against the Uniprot database to compile position specific scoring matrices (PSSMs) for two iterations using Position Specific Iterative BLAST (PSI-BLAST [38]). The options for using BLAST+ [39] are as follows:

$$\mathit{\text{-num\_iterations}}2\mathit{\text{-db uniprot}}$$

In our study, Pseudo-PSSM (PsePSSM) is employed to retain information in PSSM, and its basic idea is to consider the pseudo-amino-acid composition in PSSM [40]. This operation aims to get vectors that satisfy the algorithms and involves two steps.

The first step is the process of standardizing the PSSM. The formula is shown below:

(1)$$p_{i,j}^{\prime }=\frac{p_{i,j}-\frac{1}{20}{\sum }_{m=1}^{20}{p}_{i,m}}{\sqrt{\frac{1}{20}{\sum }_{n=1}^{20}{\left(p_{i,n}-\frac{1}{20}{\sum }_{m=1}^{20}{p}_{i,m}\right)}^2}}$$

The second step is to use the standardized matrix to generate Pseudo-PSSM (PsePSSM).

(2)$$\begin{array}{c}\hfill F_{PsePssm}=\{\begin{array}{c}\hfill \frac{1}{N}{\sum }_{i=1}^N{p}_{i,j}^{\prime };j=1,\mathrm{\cdots },20\hfill \\ \hfill \frac{1}{N-lag}{\sum }_{i=1}^{N-lag}{\left(p_{i,j}^{\prime }-p_{i+lag,j}^{\prime }\right)}^2;\hfill \\ \hfill lag=1,\mathrm{\cdots },8,j=1,\mathrm{\cdots },20\hfill \end{array}\end{array}$$

where lag denotes the distance between one residue and its neighbors.

Finally, the standardized PSSM matrix is transformed into a 20 + 20 $\times$ 8 = 180 dimensions vector containing protein characteristics.

The evaluation of the result is shown in four standard measurements, Accuracy (ACC), Matthew’s correlation coefficient (MCC) [43, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92], Sensitivity (SN), and Specificity (SP). Their formulas are as follows:

(3)$$SN=\frac{TP}{FN+TP}$$

(4)$$SP=\frac{TN}{FP+TN}$$

(5)$$\mathrm{MCC}=\frac{\mathrm{TP}\times \mathrm{TN}-\mathrm{FP}\times \mathrm{FN}}{\sqrt{(\mathrm{TP}+\mathrm{FN})\times (\mathrm{TP}+\mathrm{FP})\times (\mathrm{TN}+\mathrm{FP})\times (\mathrm{TN}+\mathrm{FN})}}$$

(6)$$ACC=\frac{TP+TN}{TP+TN+FP+FN}$$

where TP, TN, FN, and FP denote the number of true positive, true negative, false negative, and false positive, respectively.

We conduct two types of experiments to test model stacking frameworks. Firstly, predictors of four single classifiers are built by RF, XGBoost, KNN, and SVM, respectively. We compare their performance on the same independent dataset, and the results are shown in Table 2. Then, three classifiers using different combinations of base models are constructed: MSF (SVM, XGBoost), MSF (SVM, XGBoost, KNN), and MSF (SVM, XGBoost, KNN, RF). They employ SVM + XGBoost, SVM + XGBoost + KNN, SVM + XGBoost + KNN + RF as the base classifiers, respectively. Next, LR is employed as the last layer of the model and the results of these base classifiers are fed into the final prediction. The results are shown in Table 3. The ROC curve with AUC values and PR curve are shown in Fig. 2.

Fig. 2.

The ROC and PR curves of different models. (A) The ROC curves of single classifiers. (B) The PR curves of single classifiers. (C) The ROC curves of using different base classifiers. (D) The PR curves of using different base classifiers.

Table 2 shows that the predictor that uses SVM, which achieves the best performance of MCC = 0.8599, ACC = 95.14, SP = 96.66, and SN = 89.79%. The RF performs the worst results of MCC = 0.7558, ACC = 0.9186%, SP = 91.58, and SN = 93.20. The gaps of MCC, ACC, SP, and SN between the worst and best ones are 3.28%, 0.1041, –3.41% and 5.08%, respectively. Fig. 2 also shows that SVM has the best performance and highest AUC values.

Table 3 shows that the fourth predictor is ahead of the others on four measures. In all evaluative measurements, ensemble model is better than a single predictor. In Fig. 2, comparing the AUC values of MSF and single classifiers, we also find that MSF has better performance. We believe that ensemble learning performs better than single classifiers because ensemble learning can combine different classifiers flexibly and make better use of the advantages of classifiers. In addition, different base classifiers may have different feature representations for the same data, resulting in the effect of mutual error correction, thus obtaining better performance. Compared to the first predictor, the second predictor obtain better performance by adding KNN which performs second-best among the four single classifiers. RF reduces the overall prediction performance of ensemble model. The results show that a weak base classifier may lead to the effect of model stacking.

To further demonstrate the statistical significance of MSF, we perform t-test analysis on MSF and single classifiers including RF, XGBoost, KNN, and SVM.

The p-value of the t-test are shown in Table 4. In addition, log (0.05/p-value) visually shows the difference between p-value and 0.05. The larger the value is, the more significant the difference is.

The results show that the p-values of all algorithms are less than 0.05, indicating that the effects of stacked model framework and single classifier are significantly different and statistically significant. In addition, RF is the algorithm with the most significant difference from the stack model.

In order to further verify that model stacking is an appropriate integration strategy for protein classification problems, the majority voting-based method using SVM, XGBoost, and KNN is tested on the independent dataset. It contains two types of hard voting and soft voting. The results can be found in Table 5.

The gaps of MCC, ACC, SP, and SN between the Hard voting and Soft voting are 0.12%, –0.97%, –0.003%, and –0.12%, respectively. On the same independent dataset, MSF ranks first except for the SN measurements.

When the model is applied, the results of the system may fluctuate due to changes in the data. In order to judge the severity of the model changes, we conduct stability tests. We conduct five times of 5-fold cross-validation to test the stability of the algorithm, and calculate the mean and standard deviation of four metrics including SN, SP, ACC and MCC, as shown in Table 6.

The results show that MSF achieves the best mean performance except that the SN value is worse than that of RF. In stability, MSF’s four metrics rank 2,5,3, and 3 respectively among all classifiers.

To further test our method’s sophisticated and superior performance, we compare other existing works under the same data set. These models include ET-CNN [17] and ET-GRU [20].

Table 7 shows results of comparisons between our model and other approaches. It is easy to observe that MSF (SVM, XGBoost, KNN) ranks first with ACC = 95.7, MCC = 0.88, SN = 91.7%, and SP = 96.8%.

We believe that the main reason why ET-MSF performs better than ET-CNN and ET-GRU may be that our algorithm is more suitable for the task. Due to the characteristics of the neural network, only a large number of samples can make the network better fitting. However, the dataset of electron transport protein is relatively small, which limits the growth of neural network model size and leads to poor model effect. Therefore, using the ensemble strategy to combine machine learning classifiers can have a better effect.

We provide a simple web server that can be freely accessible at http://82.156.89.65/ to allow readers to evaluate and use our approaches online. The online version of ET-MSF uses the Java language and the Spring Boot framework. ET-MSF can be used by biologists to identify electron transport proteins online. Biologists can obtain the model’s prediction probability by simply entering the protein’s amino acid sequence(s) in a standard FASTA file format. Biologists can also download our public datasets and models at https://github.com/Kinkou626/ET-MSF and run them on own computer.