Parallel three-dimensional DSMC method using mesh refinement and variable time-step scheme

Application of variable time-step and unstructured adaptive mesh refinement in parallel three-dimensional Direct Simulation Monte Carlo (DSMC) method is presented. A variable time-step method using the particle fluxes conservation (mass, momentum and energy) across the cell interface is implemented to reduce the number of simulated particles and the number of iterations of transient period towards steady state, without sacrificing the solution accuracy. In addition, a three-dimensional h-refined unstructured adaptive mesh with simple but effective mesh-quality control, obtained from a preliminary parallel DSMC simulation, is used to increase the accuracy of the DSMC solution. Completed code is then applied to compute several external and internal flows, and compared with previous results wherever available.     

Simulation of unsteady two-phase flows using a parallel Eulerian–Lagrangian approach

The present study details the implementation of a time accurate method for the tracking of particles being acted upon by a continuous gas phase and gravity. The Lagrangian particle tracking approach was implemented within the framework of a parallel, incompressible, unstructured, node-centered finite-volume flow solver. The paper gives a method for selecting time steps for individual particles such that interactions with the continuum phase are updated at particle locations nearest the continuum-phase nodes while constraining the particle from passing beyond boundaries of the relevant adjacent cell. An implementation of this technique for three-dimensional nonuniform multi-element unstructured grids is given in the context of domain decomposition for implementation on distributed-memory parallel computers. Results of simulations with and without particle–particle collisions compare favorably with experimental validation results.     

Optimizations and OpenMP implementation for the direct simulation Monte Carlo method

Parallel implementation of a three-dimensional direct simulation Monte Carlo (DSMC) code employing complex data structures and dynamic memory allocation is detailed for shared memory systems using Open Multi-Processing (OpenMP). Several techniques to optimize the serial implementation of the DSMC method are first discussed. Specifically for a 3-level Cartesian grid, a Cartesian-based movement technique including particle indexing is demonstrated to result in a modest decrease in overall simulation expense of 34% compared with a ray-tracing technique combined with stored cell-connectivity. Two strategies for data localization leading to optimal usage of cache memory are demonstrated to speed up certain cell-based functions (such as collision computations) by a factor of 3.38–4.36. The shared-memory parallel implementation using OpenMP is then described in detail. Synchronization points and related critical sections are identified as major factors that impact the OpenMP parallel performance. Techniques to remove all such synchronization points in the OpenMP implementation of the DSMC method are outlined. For dual-core and quad-core systems, speedups of 1.99 and 3.74, respectively, are obtained for a (free-stream flow) test simulation with low granularity. Finally, the parallel performance of identical source code employing OpenMP is shown to be strongly correlated to the underlying computer architecture. Both Symmetric Multiprocessor (SMP) and non-uniform memory access (NUMA) systems are studied in order to achieve a better understanding of their impacts on parallel scalability when using OpenMP.     

Gas-kinetic numerical study of complex flow problems covering various flow regimes

The Boltzmann simplified velocity distribution function equation, as adapted to various flow regimes, is described on the basis of the Boltzmann–Shakhov model from the kinetic theory of gases in this study. The discrete velocity ordinate method of gas-kinetic theory is studied and applied to simulate complex multi-scale flows. On the basis of using the uncoupling technique on molecular movements and collisions in the DSMC method, the gas-kinetic finite difference scheme is constructed by extending and applying the unsteady time-splitting method from computational fluid dynamics, which directly solves the discrete velocity distribution functions. The Gauss-type discrete velocity numerical quadrature technique for flows with different Mach numbers is developed to evaluate the macroscopic flow parameters in the physical space. As a result, the gas-kinetic numerical algorithm is established for studying the three-dimensional complex flows with high Mach numbers from rarefied transition to continuum regimes. On the basis of the parallel characteristics of the respective independent discrete velocity points in the discretized velocity space, a parallel strategy suitable for the gas-kinetic numerical method is investigated and, then, the HPF (High Performance Fortran) parallel programming software is developed for simulating gas dynamical problems covering the full spectrum of flow regimes. To illustrate the feasibility of the present gas-kinetic numerical method and simulate gas transport phenomena covering various flow regimes, the gas flows around three-dimensional spheres and spacecraft-like shapes with different Knudsen numbers and Mach numbers are investigated to validate the accuracy of the numerical methods through HPF parallel computing. The computational results determine the flow fields in high resolution and agree well with the theoretical and experimental data. This computing, in practice, has confirmed that the present gas-kinetic algorithm probably provides a promising approach for resolving hypersonic aerothermodynamic problems with the complete spectrum of flow regimes from the gas-kinetic point of view for solving the mesoscopic Boltzmann model equation.     

Parallel-multigrid computation of unsteady incompressible viscous flows using a matrix-free implicit method and high-resolution characteristics-based scheme

A three-dimensional parallel unstructured non-nested multigrid solver for solutions of unsteady incompressible viscous flow is developed and validated. The finite-volume Navier–Stokes solver is based on the artificial compressibility approach with a high-resolution method of characteristics-based scheme for handling convection terms. The unsteady flow is calculated with a matrix-free implicit dual time stepping scheme. The parallelization of the multigrid solver is achieved by multigrid domain decomposition approach (MG-DD), using single program multiple data (SPMD) and multiple instruction multiple data (MIMD) programming paradigm. There are two parallelization strategies proposed in this work, first strategy is a one-level parallelization strategy using geometric domain decomposition technique alone, second strategy is a two-level parallelization strategy that consists of a hybrid of both geometric domain decomposition and data decomposition techniques. Message-passing interface (MPI) and OpenMP standard are used to communicate data between processors and decompose loop iterations arrays, respectively. The parallel-multigrid code is used to simulate both steady and unsteady incompressible viscous flows over a circular cylinder and a lid-driven cavity flow. A maximum speedup of 22.5 could be achieved on 32 processors, for instance, the lid-driven cavity flow of Re = 1000. The results obtained agree well with numerical solutions obtained by other researchers as well as experimental measurements. A detailed study of the time step size and number of pseudo-sub-iterations per time step required for simulating unsteady flow are presented in this paper.     

Fast cell determination of the DSMC molecules in multi-stage turbo molecular pump design

In this study, an existing 2D parallel DSMC solver is modified, to analyze the multi-stage turbomolecular pumps more efficiently. Generally, molecule movements are traced cell-by-cell in DSMC solvers both in structured and unstructured meshes in order to determine which cell the DSMC molecule is positioned in. These calculations require time consuming trigonometric operations. If a nonrectangular physical domain can be converted into a rectangular computational domain using curvilinear coordinates, then it would be possible to calculate the DSMC molecule cell information not only in a very short time, but also with simple arithmetic operations. In this study, it is shown that the curvilinear coordinate technique is quite faster compared to cell-by-cell tracing technique. After that, the present 2D parallel DSMC solver is renewed to use implicit molecule indexing to shorten the calculation time even further. Thirdly, dynamically changing representative molecular ratios are used to decrease the statistical errors. Following that, molecule transfer method between computational domains is revised to employ different time steps and blade spacings. Finally, calculations are shown to be in close agreement with the previously published experimental results.     

Parallel computing of diatomic molecular rarefied gas flows

A parallel algorithm for direct simulation Monte Carlo calculation of diatomic molecular rarefied gas flows is presented. For reliable simulation of such flow, an efficient molecular collision model is required. Using the molecular dynamics method, the collision of N₂ molecules is simulated. For this molecular dynamics simulation, the parameter decomposition method is applied for parallel computing. By using these results, the statistical collision model of diatomic molecule is constructed. For validation this model is applied to the direct simulation Monte Carlo method to simulate the energy distribution at equilibrium condition and the structure of normal shock wave. For this DSMC calculation, the domain decomposition is applied. It is shown that the collision process of diatomic molecules can be calculated precisely and the parallel algorithm can be efficiently implemented on the parallel computer.     

Transition prediction for three-dimensional flows using parallel computation

A computational method to predict transition lines for general three-dimensional configurations is presented. The method consists of a coupled program system including a 3D Navier-Stokes solver, a transition module, a boundary layer code and a stability code. The newly developed transition module has been adapted to be used with parallel computation to account for the high computational demand for three-dimensional configurations. Detailed computations have been performed to show the ability of the Navier-Stokes code to provide three-dimensional boundary layer data of high accuracy needed for the stability analysis. A comprehensive investigation on general computational and parallel performance identifies the numerical effort for the transition prediction method. The procedure has been validated comparing the numerical results with experiments for the flow around an inclined prolate spheroid. Feasibility studies on generic transport aircraft have been performed to show the code’s capability to predict transition lines on general complex geometries.     

槽流拟颗粒模型的并行算法

将流体处理为离散粒子,应用拟颗粒硬球模型来研究槽流中的流动现象,与分子动力学模拟的算法类似,是研究槽流机理的一种行之有效的方法。为了作大规模的模拟,本文采用区域分解算法和消息传递编程模型技术,将该模型串行程序并行化,应用一维划分、单相传递的方法简化了并行算法,采用轮换搜索法来避免硬球碰撞次序对结果的影响。在可扩展的机群系统上用实例计算,通过与串行程序的对比,验证了并行程序的正确性,表明本文设计的并行算法取得了较高的并行计算效率。     

Parallel implementation of molecular dynamics simulation for short-ranged interaction

A parallel molecular dynamics simulation method, designed for large-scale problems, employing dynamic spatial domain decomposition for short-ranged molecular interactions is proposed. In this parallel cellular molecular dynamics (PCMD) simulation method, the link-cell data structure is used to reduce the searching time required for forming the cut-off neighbor list as well as for domain decomposition, which utilizes the multi-level graph-partitioning technique. A simple threshold scheme (STS), in which workload imbalance is monitored and compared with some threshold value during the runtime, is proposed to decide the proper time for repartitioning the domain. The simulation code is implemented and tested on the memory-distributed parallel machine, e.g., PC-cluster system. Parallel performance is studied using approximately one million L-J atoms in the condensed, vaporized and supercritical states. Results show that fairly good parallel efficiency at 49 processors can be obtained for the condensed and supercritical states (∼60%), while it is comparably lower for the vaporized state (∼40%).     

基于国产十亿亿次超算系统的近连续过渡流区N-S/DSMC耦合算法并行优化研究

过渡流区气动问题的数值模拟一直是空气动力学领域的难点。首先介绍了在已有 N-S解算器和 DSMC方法研究基础上,采用 MPC耦合技术建立N-S/DSMC 耦合算法,把 DSMC 方法和 N-S 方法的应用范围拓展到近连续过渡流区。然后详述了基于国家超级计算无锡中心的国产十亿亿次超级计算机开展的耦合算法多级并行优化技术,并首次实现了耦合算法的众核并行。测试表明,本文的进程级优化技术取得了超线性加速比;众核级优化受制于原算法特点和系统特点没有取得预期效果,但进行了探讨和分析,为 N-S/DSMC 耦合算法的众核并行提供了研究和分析依据,为过渡流区高超声速气动特性数值模拟研究提供了有效的途径。     

Towards a three-dimensional moving body incompressible flow solver with a linear deformable model

In this study, a three-dimensional fluid–structured parallelized solver is extended from the previous work (Niu et al., 2009 [1]) for moving body simulations. Based on the unified Eulerian and Lagrangian coordinate transformations, the unsteady three-dimensional incompressible Navier–Stokes equations with artificial compressibility (Chorin, 1967 [2]) in a dual-time stepping approach are first derived. To implement unsteady flow calculations, the dual-time stepping strategy including the LU decomposition method is used in the pseudo-time iteration and the second-order accurate backward difference is adopted to discretize the unsteady flow terms. Also, a third-order Roe type flux limited splitting is derived to evaluate the spatial difference of the convective fluxes. The original FORTRAN code is converted to the MPI code and tested on a 64-CPU IBM SP2. The parallel strategy here is based on the partitions of all do-loops in the original FORTRAN code and transferring the calculations inside the do-loop into different CPUs. The partition of the do-loop can be applied on the innermost loop, only or the last two inner loops depending on two-dimensional or three-dimensional problems. This kind of the parallel data partition of the loops is independent of what kind of the explicit or implicit type numerical algorithm used. Therefore, the current parallel approach can take advantage of the MPI language fully to transfer data efficiently among CPUs even for solving the governing equation implicitly. The test results show that a significant reduction of computing time in running the model and a near-linear speed up rate is achieved up to 32 CPUs at IBM SP2. The speed up rate is as high as 31 for using 64 IBM SP2 processors The test shows efficient parallel processing to provide prompt simulation of 3D cavity, unsteady dropping airfoil and blood flows in an aortic tube with a linear elastic modeling of wall motion is included here.     

Particle partitioning strategies for the parallel computation of solid-liquid flows

The numerical investigation of the interaction of large, solid particles with fluids is an important area of research for many manufacturing processes. Such studies frequently lead to models that are very large and require the use of parallel solution techniques. This paper presents the results of a parallel implementation of a serial code for the direct numerical simulation of solid-liquid flows. The base code is a serial, arbitrary Lagrangian-Eulerian (ALE) formulation of the equations of motion, which views that particles as solid bodies are embedded into the flow domain. This particular model poses some interesting difficulties for domain decomposition type approaches for parallel solutions. In particular, it is not fully understood how the partitioning of the particles among the subdomains influences the performance of parallel solvers. We present several strategies for the partitioning of the solid particles, focusing on the effectiveness of these techniques in terms of parallel speedup and efficiency.     

Coupled solid (FVM)–fluid (DSMC) simulation of micro-nozzle with unstructured-grid

The flow field and temperature distributions of free molecular micro-electro-thermal resist jet (FMMR) were studied resorting to DSMC–FVM coupled method. Direct simulation Monte Carlo (DSMC) method is the most useful tool to simulate the flow field of FMMR and unstructured grid is suitable for the flow simulation in a complicated region with tilted wall surface. DSMC code based on unstructured grid system was developed and the result was compared with that of structured grid and analytical solution to validate the reliability of the developed code. The DSMC method was then used to simulate the fluid flow in the micro-nozzle (Kn > 0.01) and the temperature distribution in the nozzle wall was obtained by the finite volume method (FVM). The Dirichlet–Neumann method was used to couple the wall heat flux and temperature between flow field and solid area. The effect of different income pressure was studied in detail and the results showed that the temperature of solid area changed drastically at different income pressure, so the commonly-adopted method of pre-setting boundary temperature before simulation was unreasonable.     

Parallel computation for shallow water flow: A domain decomposition approach

This paper describes a strategy for the parallelization of a finite element code for the numerical simulation of shallow water flow. The numerical scheme adopted for the discretization of the equations in the scalar algorithm is briefly described, with emphasis on the aspects concerning its porting to a parallel architecture. The parallelization strategy is of the domain decomposition type: the implicit computational kernel of the scheme, a Poisson problem, is solved by an additive Schwarz preconditioning technique within conjugate gradient iterations. Both the theoretical and the implementation aspects of the domain decomposition method are described as applied in the present context. Finally, some computational examples are shown and discussed.     

Application of a parallel DSMC technique to predict flowcharacteristics in microfluidic filters

Using a parallel implementation of the direct simulation Monte Carlo (DSMC) method, periodic MEMS microfilters are studied in detail. The dependence of the flow characteristics on geometry, Knudsen number, pressure difference, spacing between the filter elements, and accommodation coefficients are investigated. By comparing DSMC results with the widely used analytical formulas, the validity range of the analytical approaches is evaluated. The simulation results show that velocity slip exists both on the filter channel walls and on the filter membrane and results in an increased flow rate. Velocity slip increases strongly with decreasing accommodation coefficients. For long channels, this results in a strong increase in flow rate; whereas for short channels, the increase in flow rate is limited. For the filter separations considered in this paper, we observe that separation between filter channels does not influence the flow rate within each channel     

多群粒子输运问题在多核集群系统上的混合并行计算

本文分析了非结构网格多群粒子输运Sn方程求解的并行性,拟合多核机群系统的特点,设计了MPI/OpenMP混合程序,针对空间网格点采用区域分解划分,计算结点间基于消息传递MPI编程,每个MPI计算进程在计算过程中碰到关于能群的计算,就生成多个OpenMP线程,计算结点内针对能群进行多线程并行计算。数值测试结果表明,非结构网格上的粒子输运问题的混合并行计算能较好地匹配多核机群系统的硬件结构,具有良好的可扩展性,可以扩展到1024个CPU核。     

3D hexagonal parallel code QuDiff for calculating a fast reactor’s critical parameters

Being applicable for high-performance computing systems, a parallel code based on the sequential QuDiff code was developed to calculate the critical assembly of the fast neutron reactor active zone. The multigroup transport equation calculation method was based on V. Ya. Goldin’s quasi-diffusion method. For efficient algorithm construction, it was suggested to use all the possible reactor assembly symmetries for the self-adjustable neutron-nuclear operation mode. MPI was applied as a parallel interface. The domain decomposition method was used. The pipelined parallelization method was applied for the consistent parallelization of the calculation of the quasi-diffusion system of equations and for the calculation of the transport equation. The calculations performed for the 3D active zone model of the BN-800 reactor capable of operating in a self-adjustable neutron-nuclear mode showed that the parallel code QuDiff is highly scalable. The present results can be used in the dynamic numerical simulation of a fast reactor’s active zones.     

三维粒子模拟并行化技术研究

自行研制的三维并行全电磁PIC模拟软件UNIPIC-3D具有模拟高功率微波器件的能力。软件实现了并行的三维FDTD、粒子推进算法以及边界条件处理。软件通过读入输入文件进行规则与不规则两种区域划分方式,电磁场和粒子的并行化采用MPI机制,让粒子和电磁场的计算与通信同步,在高性能并行计算机上对软件的并行效率进行了测试。通过与2.5维UNIPIC软件的结果比较,验证了UNIPIC-3D软件并行模块的正确性。