A parallel lattice Boltzmann method for large eddy simulation on multiple GPUs

To improve the simulation efficiency of turbulent fluid flows at high Reynolds numbers with large eddy dynamics, a CUDA-based simulation solution of lattice Boltzmann method for large eddy simulation (LES) using multiple graphics processing units (GPUs) is proposed. Our solution adopts the “collision after propagation” lattice evolution way and puts the misaligned propagation phase at global memory read process. The latest GPU platform allows a single CPU thread to control up to four GPUs that run in parallel. In order to make use of multiple GPUs, the whole working set is evenly partitioned into sub-domains. We implement Smagorinsky model and Vreman model respectively to verify our multi-GPU solution. These two LES models have different relaxation time calculation behavior and lead to different CUDA implementation characteristics. The implementation based on Smagorinsky model achieves 190 times speedup over the sequential implementation on CPU, while the implementation based on Vreman model archives more than 90 times speedup. The experimental results show that the parallel performance of our multi-GPU solution scales very well on multiple GPUs. Therefore large-scale (up to 10,240 $\times $ 10,240 lattices) LES–LBM simulation becomes possible at a low cost, even using double-precision floating point calculation.     

Multi-block lattice Boltzmann simulations of subcritical flow in open channel junctions

A two-dimensional lattice Boltzmann model (LBM) for subcritical flows in open channel junctions is developed. Shallow water equations coupled with the large eddy simulation model is numerically simulated by the lattice Boltzmann method, so that the turbulence, caused by the combination of the main channel and tributary flows, can be taken into account and modeled efficiently. In order to obtain more detailed and accurate results, a multi-block lattice scheme is designed and applied at the area of combining flows. The model is first verified by experimental data for a 90° junction flow, then is used to investigate the effect of the junction angle on flow characteristics, such as velocity field, water depth and separation zone. The objectives of this study are to validate the two-dimensional LBM in junction flow simulation and compare the results with available experimental data and classical analytical solutions in the separation zone.     

Improved coupling of time integration and hydrodynamic interaction in particle suspensions using the lattice Boltzmann and discrete element methods

This paper introduces improvements to the simulation of particle suspensions using the lattice Boltzmann method (LBM) and the discrete element method (DEM). First, the benefit of using a two-relaxation-time (TRT) collision operator, instead of the popular Bhatnagar–Gross–Krook (BGK) collision operator, is demonstrated. Second, a modified solid weighting function for the partially saturated method (PSM) for fluid–solid interaction is defined and tested. Results are presented for a range of flow configurations, including sphere packs, duct flows, and settling spheres, with good accuracy and convergence observed. Past research has shown that the drag, and consequently permeability, predictions of the LBM exhibit viscosity-dependence when used with certain boundary conditions such as bounce-back or interpolated bounce-back, and this is most pronounced when the BGK collision operator is employed. The improvements presented here result in a range of computational viscosities, and therefore relaxation parameters, within which drag and permeability predictions remain invariant. This allows for greater flexibility in using the relaxation parameter to adjust the LBM timestep, which can subsequently improve synchronisation with the time integration of the DEM. This has significant implications for the simulation of large-scale suspension phenomena, where the limits of computational hardware persistently constrain the resolution of the LBM lattice.     

Domain decomposition based coupling between the lattice Boltzmann method and traditional CFD methods—Part I: Formulation and application to the 2-D Burgers’ equation

The lattice Boltzmann method is being increasingly employed in the field of computational fluid dynamics due to its computational efficiency. Floating-point operations in the lattice Boltzmann method involve local data and therefore allow easy cache optimization and parallelization. Due to this, the cache-optimized lattice Boltzmann method has superior computational performance over traditional finite difference methods for solving unsteady flow problems. When solving steady flow problems, the explicit nature of the lattice Boltzmann discretization limits the time step size and therefore the efficiency of the lattice Boltzmann method for steady flows. To quantify the computational performance of the lattice Boltzmann method for steady flows, a comparison study between the lattice Boltzmann method (LBM) and the alternating direction implicit (ADI) method was performed using the 2-D steady Burgers’ equation. The comparison study showed that the LBM performs comparatively poor on high-resolution meshes due to smaller time step sizes, while on coarser meshes where the time step size is similar for both methods, the cache-optimized LBM performance is superior. Because flow domains can be discretized with multiblock grids consisting of coarse and fine grid blocks, the cache-optimized LBM can be applied on the coarse grid block while the traditional implicit methods are applied on the fine grid blocks. This paper finds the coupled cache-optimized lattice Boltzmann-ADI method to be faster by a factor of 4.5 over the traditional methods while maintaining similar accuracy.     

A viscosity counteracting approach in the lattice Boltzmann BGK model for low viscosity flow: Preliminary verification

Due to numerical instability, the lattice Boltzmann model (LBM) with the Bhatnagar–Gross–Krook (BGK) collision operator has some limitations in the simulation of low viscosity flows. In this paper, we propose a viscosity counteracting approach for simulating a moderate viscosity flow. An extra negative viscosity term is introduced to counteract part of the moderate viscosity by using the lattice Boltzmann equation with a source term. The counteracting viscosity term is treated as a non-uniform unsteady source. The stability is enhanced; thus small viscosity flows can be simulated. Model verification consists of benchmark cases such as those of Poiseuille flow, Couette flow, waterhammer waves, Taylor–Green vortex flow, and lid-driven cavity flow. The flow patterns, error characteristics, and representative parameters are carefully analyzed. It is shown that this approach can simulate flows with lower viscosities than may be simulated using the normal LBGK model; the second-order accuracy of the LBGK model is definitely retained, although a little dissipation is added. These preliminary studies prove the effectiveness and accuracy of the model. Sophisticated analysis and further verification of the stability mechanism will be done in the near future.     

Sailfish: A flexible multi-GPU implementation of the lattice Boltzmann method

We present Sailfish, an open source fluid simulation package implementing the lattice Boltzmann method (LBM) on modern Graphics Processing Units (GPUs) using CUDA/OpenCL. We take a novel approach to GPU code implementation and use run-time code generation techniques and a high level programming language (Python) to achieve state of the art performance, while allowing easy experimentation with different LBM models and tuning for various types of hardware. We discuss the general design principles of the code, scaling to multiple GPUs in a distributed environment, as well as the GPU implementation and optimization of many different LBM models, both single component (BGK, MRT, ELBM) and multicomponent (Shan–Chen, free energy). The paper also presents results of performance benchmarks spanning the last three NVIDIA GPU generations (Tesla, Fermi, Kepler), which we hope will be useful for researchers working with this type of hardware and similar codes.     

A fractional step lattice Boltzmann method for simulating high Reynolds number flows

A fractional step lattice Boltzmann scheme is presented to greatly improve the stability of the lattice Boltzmann method (LBM) in modelling incompressible flows at high Reynolds number. This method combines the good features of the conventional LBM and the fractional step technique. Through the fractional step, the flow at an extreme case of infinite Reynolds number (inviscid flow) can be effectively simulated. In addition, the non-slip boundary condition can be directly implemented.     

Application of lattice Boltzmann method in free surface flow simulation of micro injection molding

The filling flow in micro injection molding was simulated by using the lattice Boltzmann method (LBM). A tracking algorithm for free surface to handle the complex interaction between gas and liquid phases in LBM was used for the free surface advancement. The temperature field in the filling flow is also analyzed by combining the thermal lattice Boltzmann model and the free surface method. To simulate the fluid flow of polymer melt with a high Prandtl number and high viscosity, a modified lattice Boltzmann scheme was adopted by introducing a free parameter in the thermal diffusion equation to overcome the restriction of the thermal relaxation time. The filling flow simulation of micro injection molding was successfully performed in the study.     

Topology optimization of flow domains using the lattice Boltzmann method

We consider the optimal design of two- (2D) and three-dimensional (3D) flow domains using the lattice Boltzmann method (LBM) as an approximation of Navier-Stokes (NS) flows. The problem is solved by a topology optimization approach varying the effective porosity of a fictitious material. The boundaries of the flow domain are represented by potentially discontinuous material distributions. NS flows are traditionally approximated by finite element and finite volume methods. These schemes, while well established as high-fidelity simulation tools using body-fitted meshes, are effected in their accuracy and robustness when regular meshes with zero-velocity constraints along the surface and in the interior of obstacles are used, as is common in topology optimization. Therefore, we study the potential of the LBM for approximating low Mach number incompressible viscous flows for topology optimization. In the LBM the geometry of flow domains is defined in a discontinuous manner, similar to the approach used in material-based topology optimization. In addition, this non-traditional discretization method features parallel scalability and allows for high-resolution, regular fluid meshes. In this paper, we show how the variation of the porosity can be used in conjunction with the LBM for the optimal design of fluid domains, making the LBM an interesting alternative to NS solvers for topology optimization problems. The potential of our topology optimization approach will be illustrated by 2D and 3D numerical examples.     

An algorithm based on collision theory for the lattice Boltzmann simulation of isothermal mass diffusion with chemical reaction

The lattice Boltzmann method (LBM) provides a framework for the simulation of mass transport and chemical reaction in complex geometries when accurate pointwise mesoscopic-scale solutions are sought. Herein, an algorithm based on collision theory to determine the rate of chemical reaction during the collision step in LBM is proposed. The model is validated against three isothermal problems with simple analytical solutions: a batch reactor with homogeneous chemical reactions of first and second-order and a cylindrical pore with one-dimensional mass diffusion and surface (heterogeneous) chemical reaction. The results of the LBM simulations agree to within 1% when compared to the analytical solutions, presenting a promising opportunity for the simulation of detailed chemical reaction mechanisms.     

Adjoint lattice Boltzmann equation for parameter identification

The lattice Boltzmann equation is briefly introduced using moments to clearly separate the propagation and collision steps in the dynamics. In order to identify unknown parameters we introduce a cost function and adapt control theory to the lattice Boltzmann equation to get expressions for the derivatives of the cost function vs. parameters. This leads to an equivalent of the adjoint method with the definition of an adjoint lattice Boltzmann equation.To verify the general expressions for the derivatives, we consider two elementary situations: a linearized Poiseuille flow to show that the method can be used to optimize parameters, and a nonlinear situation in which a transverse shear wave is advected by a mean uniform flow. We indicate in the conclusion how the method can be used for more realistic situations.     

A unified approach for nonslip and slip boundary conditions in the lattice Boltzmann method

This work proposed a unified approach to impose both nonslip and slip boundary conditions for the lattice Boltzmann method (LBM). By introducing the tangential momentum accommodation coefficient (TMAC), the present implementation can determine the change of the tangential momentum on the wall and then impose the correct boundary conditions for LBM. The simulation results demonstrate that this implementation is equivalent to the first-order slip model.     

Lattice Boltzmann method for microfluidics: models and applications

The lattice Boltzmann method (LBM) has experienced tremendous advances and has been well accepted as a useful method to simulate various fluid behaviors. For computational microfluidics, LBM may present some advantages, including the physical representation of microscopic interactions, the uniform algorithm for multiphase flows, and the easiness in dealing with complex boundary. In addition, LBM-like algorithms have been developed to solve microfluidics-related processes and phenomena, such as heat transfer, electric/magnetic field, and diffusion. This article provides a practical overview of these LBM models and implementation details for external force, initial condition, and boundary condition. Moreover, recent LBM applications in various microfluidic situations have been reviewed, including microscopic gaseous flows, surface wettability and solid–liquid interfacial slip, multiphase flows in microchannels, electrokinetic flows, interface deformation in electric/magnetic field, flows through porous structures, and biological microflows. These simulations show some examples of the capability and efficiency of LBM in computational microfluidics.     

A two-dimensional lattice Boltzmann model for uniform channel flows

In this paper a novel two-dimensional lattice Boltzmann model (LBM) is developed for uniform channel flows. The axial velocity is solved from a momentum diffusion equation over the cross-sectional plane. An extrapolation boundary condition is also introduced to enhance the no-slip boundary in the momentum equation. This boundary treatment can also be applied to LBM simulations of other diffusion processes. The algorithm and boundary treatment are validated by simulations of steady Poiseuille and pulsatile Womersley flows in circular pipes. The numerical convergence and accuracy are comparable to those of existing models. Moreover, comparison with general three-dimensional lattice Boltzmann simulations demonstrates the advantages of our two-dimensional model, including lower computational resource requirements (memory and time), easier boundary treatment for arbitrary cross-sectional shapes, and no velocity constraint. These features are attractive for practical applications with uniform channel flows.     

Numerical simulation of the collision of two microdroplets with a pseudopotential multiple-relaxation-time lattice Boltzmann model

Collisions of two equally sized liquid microdroplets in gaseous phase are numerically studied by the lattice Boltzmann method (LBM). The multiphase formulation adopted is a pseudopotential model with improved treatment of the equation of state and force incorporation which is then coupled with the multiple-relaxation-time scheme. That allows a detailed investigation into microdroplet collisions characterized by high-density ratios as well as by relevant inertial effects. Simulations related to a wide range of flow parameters (e.g. Weber and Reynolds numbers) are reported, in order to embrace all the collisional regimes presented in previous experimental studies. From surface tension-driven coalescence (both inertial and viscous coalescence have been examined) to catastrophic break-up with the formation of children microdroplets, the simulations demonstrate that the LBM correctly reproduces the collision dynamics and the final outcomes in almost all the regimes. Different break-up mechanisms like end-pinching and capillary wave-induced break-up have been observed. Finally, the initial stages of the inertia-dominated head-on collision process have been studied, showing once more the effectiveness and reliability of this multiphase LBM implementation.     

Lattice Boltzmann simulation of turbulent flow laden with finite-size particles

The paper describes a particle-resolved simulation method for turbulent flow laden with finite size particles. The method is based on the multiple-relaxation-time lattice Boltzmann equation. The no-slip boundary condition on the moving particle boundaries is handled by a second-order interpolated bounce-back scheme. The populations at a newly converted fluid lattice node are constructed by the equilibrium distribution with non-equilibrium corrections. MPI implementation details are described and the resulting code is found to be computationally efficient with a good scalability. The method is first validated using unsteady sedimentation of a single particle and sedimentation of a random suspension. It is then applied to a decaying isotropic turbulence laden with particles of Kolmogorov to Taylor microscale sizes. At a given particle volume fraction, the dynamics of the particle-laden flow is found to depend mainly on the effective particle surface area and particle Stokes number. The presence of finite-size inertial particles enhances dissipation at small scales while reducing kinetic energy at large scales. This is in accordance with related studies. The normalized pivot wavenumber is found to not only depend on the particle size, but also on the ratio of particle size to flow scales and particle-to-fluid density ratio.     

Lattice kinetic simulations of 3-D MHD turbulence

A recently proposed lattice Boltzmann kinetic scheme offers a promising tool for simulating complex 3-D MHD flows. The algorithm is based on the BGK modeling of the collision term. The conventional approach for implementing magnetic behavior in LBM methods is based on one tensor-valued distribution function to present both the fluids variables (density and momentum) and the magnetic field. This formulation, however, has been proven a rather inefficient approach. The present scheme calls for a separate BGK-like evolution equation for the magnetic field which models the induction equation and enhances simplicity while allowing for the independent adjustment of the magnetic resistivity. Furthermore the algorithm correctly recovers the macroscopic dissipative MHD equations. Numerical results for the 3-D Taylor–Green vortex problem are presented with corresponding results computed with a pseudo-spectral code used as benchmark.     

An analysis of natural convection using the thermal finite element discrete Boltzmann equation

The lattice Boltzmann method (LBM) is the simple numerical simulator for fluids because it consists of linear equations. Excluding the higher differential term, the LBM for a temperature field is also achieved as an easy numerical simulation method. However, the LBM is hardly applied to body fitted coordinates for its formulation. It is then difficult to calculate complex lattices using the LBM. In this paper, the finite element discrete Boltzmann equation (FEDBE) is introduced to deal with this weakness of the LBM. The finite element method is applied to the discrete Boltzmann equation (DBE) of the basic equation of the LBM. For FEDBE, the simulation using complex lattices is achieved, and it will be applicable for the development in engineering fields. The natural convection in a square cavity and the Rayleigh–Bernard convection are chosen as the test problem. Each simulation model is accurate enough for the flow patterns, the temperature distribution and the Nusselt number. This method is now considered good for the flow and temperature field, and is expected to be introduced for complex lattices using the DBE.     

Parallelizing and optimizing large‐scale 3D multi‐phase flow simulations on the Tianhe‐2 supercomputer

The lattice Boltzmann method (LBM) is a widely used computational fluid dynamics method for flow problems with complex geometries and various boundary conditions. Large‐scale LBM simulations with increasing resolution and extending temporal range require massive high‐performance computing (HPC) resources, thus motivating us to port it onto modern many‐core heterogeneous supercomputers like Tianhe‐2. Although many‐core accelerators such as graphics processing unit and Intel MIC have a dramatic advantage of floating‐point performance and power efficiency over CPUs, they also pose a tough challenge to parallelize and optimize computational fluid dynamics codes on large‐scale heterogeneous system. In this paper, we parallelize and optimize the open source 3D multi‐phase LBM code openlbmflow on the Intel Xeon Phi (MIC) accelerated Tianhe‐2 supercomputer using a hybrid and heterogeneous MPI+OpenMP+Offload+single instruction, mulitple data (SIMD) programming model. With cache blocking and SIMD‐friendly data structure transformation, we dramatically improve the SIMD and cache efficiency for the single‐thread performance on both CPU and Phi, achieving a speedup of 7.9X and 8.8X, respectively, compared with the baseline code. To collaborate CPUs and Phi processors efficiently, we propose a load‐balance scheme to distribute workloads among intra‐node two CPUs and three Phi processors and use an asynchronous model to overlap the collaborative computation and communication as far as possible. The collaborative approach with two CPUs and three Phi processors improves the performance by around 3.2X compared with the CPU‐only approach. Scalability tests show that openlbmflow can achieve a parallel efficiency of about 60% on 2048 nodes, with about 400K cores in total. To the best of our knowledge, this is the largest scale CPU‐MIC collaborative LBM simulation for 3D multi‐phase flow problems. Copyright © 2015 John Wiley & Sons, Ltd.     

Memory transfer optimization for a lattice Boltzmann solver on Kepler architecture nVidia GPUs

The Lattice Boltzmann method (LBM) for solving fluid flow is naturally well suited to an efficient implementation for massively parallel computing, due to the prevalence of local operations in the algorithm. This paper presents and analyses the performance of a 3D lattice Boltzmann solver, optimized for third generation nVidia GPU hardware, also known as ‘Kepler’. We provide a review of previous optimization strategies and analyse data read/write times for different memory types. In LBM, the time propagation step (known as streaming), involves shifting data to adjacent locations and is central to parallel performance; here we examine three approaches which make use of different hardware options. Two of which make use of ‘performance enhancing’ features of the GPU; shared memory and the new shuffle instruction found in Kepler based GPUs. These are compared to a standard transfer of data which relies instead on optimized storage to increase coalesced access. It is shown that the more simple approach is most efficient; since the need for large numbers of registers per thread in LBM limits the block size and thus the efficiency of these special features is reduced. Detailed results are obtained for a D3Q19 LBM solver, which is benchmarked on nVidia K5000M and K20C GPUs. In the latter case the use of a read-only data cache is explored, and peak performance of over 1036 Million Lattice Updates Per Second (MLUPS) is achieved. The appearance of a periodic bottleneck in the solver performance is also reported, believed to be hardware related; spikes in iteration-time occur with a frequency of around 11 Hz for both GPUs, independent of the size of the problem.