Gossip-Based Load Balance Strategy in Big Data Systems with Hierarchical Processors

In big data systems, data are assigned to different processors by the system manager, which has a large amount of work to perform, such as achieving load balances and allocating data to the system processors in a centralized way. To alleviate its load, we claim that load balancing can be conducted in a decentralized way, and thus, the system manager need not be in charge of this task anymore. Two decentralized approaches are proposed for load balancing schemes, namely, a utilization scheme based on a load balance algorithm (UBLB) and a number of layers scheme based on a load balance algorithm (NLBLB). In the UBLB scheme, considering the hierarchy of the processor’s processing abilities, a gossip-based algorithm is proposed to achieve load balance using the jobs’ utilizations as load balance indicators in addition to the number of jobs. The reason for this action is that the processor’s process abilities are different from one another. Thus, the utilization indicator is more reasonable. In the NLBLB scheme, the processors are classified into different layers according to their processing abilities. In each layer, a sub-load balance is conducted, which means that the UBLB is achieved in a sub-region. The efficiencies of the two proposed schemes are validated by simulation, which proves their positive effect.