基于TPE的SpaRC算法超参数优化方法

宏基因组序列组装在计算和内存上面临着巨大挑战。SpaRC(Spark Reads Clustering)是基于Apache Spark的宏基因组序列片段聚类算法,为来自下一代测序技术的数十亿测序片段聚类提供了一种可扩展的解决方案。但是,SpaRC算法参数的设置是一项非常具有挑战性的工作。SpaRC算法拥有许多对算法性能有着很大影响的超参数,选择合适的超参数集对于充分发挥SpaRC算法的性能来说是至关重要的。为了提高SpaRC算法的性能,探索了一种基于树状结构Parzen估计方法(Tree Parzen Estimator,TPE)的超参数优化方法,其能够利用先验知识高效地调节参数,并通过减少计算任务加速寻找最优参数,达到最佳聚类效果,从而避免昂贵的参数探索。对长序列片段(PacBio)和短序列片段(CAMI2)进行实验,结果表明,该方法在改善SpaRC算法性能方面有着良好的效果。

SpaRC Algorithm Hyperparameter Optimization Methodology Based on TPE

The assembly of metagenomic sequences faces huge challenge in computing and storage.SpaRC(Spark Reads Clustering)is a metagenomic sequence fragment clustering algorithm based on Apache Spark,which provides a scalable solution for clustering of billions of sequencing fragments.However,setting SpaRC parameters is a very challenging task.SpaRC algorithm has many hyperparameters that have a great impact on the performance of the algorithm.Choosing the appropriate hyperparameter set is crucial to the performance of SpaRC algorithm.In order to improve the performance of SpaRC algorithm,a hyperpara-meter optimization method based on Tree Parzen Estimator(TPE)is explored,which can use prior knowledge to efficiently adjust the parameters,accelerate the search for the optimal parameters by reducing the calculation task to achieve the optimal clustering effect,thus avoding expensive parameter exploration.After experiments with long-reads(PacBio)and short-reads(CAMI2),the results show that the proposed method has a great effect on improving the performance of SpaRC algorithm.