Multi-scale gas discharge simulations using asynchronous adaptive mesh refinement

The breakdown of a gas gap at high products of pd (pressure × distance) is a multi-scale phenomenon in both time and space. This is especially true when the plasma is interacting with a gas flow, a problem of considerable recent interest in the context of aerodynamic applications of surface discharges. This paper presents a contribution to the numerical modeling of such discharges. We describe here a new approach for adaptive meshing which is suitable for use with the explicit asynchronous integration scheme described in our previous publication. Rather than relying on a family of nested grids as is commonly done, this technique is based on a single unstructured mesh with possible non-conforming cells at the interface between coarse and fine areas. Substantial computational time saving has been achieved for a surface dielectric barrier discharge configuration of the kind proposed as plasma actuators for flow control.     

Massively parallel adaptive mesh refinement and coarsening for dynamic fracture simulations

We use the graphical processing unit (GPU) to perform dynamic fracture simulation using adaptively refined and coarsened finite elements and the inter-element cohesive zone model. Due to the limited memory available on the GPU, we created a specialized data structure for efficient representation of the evolving mesh given. To achieve maximum efficiency, we perform finite element calculation on a nodal basis (i.e., by launching one thread per node and collecting contributions from neighboring elements) rather than by launching threads per element, which requires expensive graph coloring schemes to avoid concurrency issues. These developments made possible the parallel adaptive mesh refinement and coarsening schemes to systematically change the topology of the mesh. We investigate aspects of the parallel implementation through microbranching examples, which has been explored experimentally and numerically in the literature. First, we use a reduced-scale version of the experimental specimen to demonstrate the impact of variation in floating point operations on the final fracture pattern. Interestingly, the parallel approach adds some randomness into the finite element simulation on the structured mesh in a similar way as would be expected from a random mesh. Next, we take advantage of the speedup of the implementation over a similar serial implementation to simulate a specimen whose size matches that of the actual experiment. At this scale, we are able to make more direct comparisons to the original experiment and find excellent agreement with those results.     

Electromagnetic particle-in-cell simulations on magnetic reconnection with adaptive mesh refinement

The electromagnetic particle-in-cell (EM-PIC) model using the adaptive mesh refinement (AMR) is reconsidered so that it is properly and efficiently applied to the current sheet evolution associated with magnetic reconnection. It is very important to adequately select the refinement criteria for cell splitting. It is demonstrated that fine cells have to be distributed not only in the region where the electron Debye length is small, but also in the region where the electron-scale structure is expected to be significant. While the AMR reduces the number of cells drastically, the total simulation cost is also controlled by the number of particles. In order to reduce the total number of particles in the entire system, the present code controls the local number of particles per cell by splitting or coalescing particles. It is shown that the particle splitting and coalescence are quite effective as well as the AMR to enhance the efficiency of the EM-PIC simulations. A new 3D code extended from the 2D code is also introduced. The code is checked against the tearing instability and the lower hybrid drift instability, and it is confirmed that the code has been successfully developed. It is also found that the 3D simulations can gain more efficiency by using the AMR than the 2D simulations.     

libMesh : a C++ library for parallel adaptive mesh refinement/coarsening simulations

In this paper we describe the libMesh (http://libmesh.sourceforge.net) framework for parallel adaptive finite element applications. libMesh is an open-source software library that has been developed to facilitate serial and parallel simulation of multiscale, multiphysics applications using adaptive mesh refinement and coarsening strategies. The main software development is being carried out in the CFDLab (http://cfdlab.ae.utexas.edu) at the University of Texas, but as with other open-source software projects; contributions are being made elsewhere in the US and abroad. The main goals of this article are: (1) to provide a basic reference source that describes libMesh and the underlying philosophy and software design approach; (2) to give sufficient detail and references on the adaptive mesh refinement and coarsening (AMR/C) scheme for applications analysts and developers; and (3) to describe the parallel implementation and data structures with supporting discussion of domain decomposition, message passing, and details related to dynamic repartitioning for parallel AMR/C. Other aspects related to C++ programming paradigms, reusability for diverse applications, adaptive modeling, physics-independent error indicators, and similar concepts are briefly discussed. Finally, results from some applications using the library are presented and areas of future research are discussed.     

An algorithm for adaptive mesh refinement inn dimensions

The author describes a fast algorithm for local adaptive mesh refinement inn dimensions based on simplex bisection. A ready-to-use implementation of the algorithm in C++ pseudocode is given. It is proven that the scheme satisfies all conditions one usually places on grid refinement in the context of finite-element calculations. Bisection refinement also offers an interesting additional feature over the usual, regular, refinement scheme: all linear finite-element basis functions of one generation are of disjoint support. In the way the scheme is presented here, all generated simplex meshes satisfy a ‘structural condition’ which is exploited to simplify bookkeeping of the neighbour graph. However, bisection refinement places certain restrictions on the initial, coarsest grid. For a simply connected domain, a precise and useful criterion for the applicability of the described refinement scheme is formulated and proven.     

A new scheme for mesh generation and mesh refinement using graph theory

There are an extensive number of algorithms available from graph theory, some of which, for problems with geometric content, make graphs an attractive framework in which to model an object from its geometry to its discretization into a finite element mesh. This paper presents a new scheme for finite element mesh generation and mesh refinement using concepts from graph theory. This new technique, which is suitable for an interactive graphical environment, can also be used efficiently for fully automatic remeshing in association with self-adaptive schemes. Problems of mesh refinement around holes and local mesh refinement are treated. The suitability of the algorithms presented in this paper is demonstrated by some examples.     

Development of an object-oriented finite element program with adaptive mesh refinement for multi-physics applications

In this paper, an object-oriented framework for numerical analysis of multi-physics applications is presented. The framework is divided into several basic sets of classes that enable the code segments to be built according to the type of problem to be solved. Fortran 2003 was used in the development of this finite element program due to its advantages for scientific and engineering programming and its new object-oriented features. The program was developed with h-type adaptive mesh refinement, and it was tested for several classical cases involving heat transfer, fluid mechanics and structural mechanics. The test cases show that the adaptive mesh is refined only in the localization region where the feature gradient is relatively high. The overall mesh refinement and the h-adaptive mesh refinement were justified with respect to the computational accuracy and the CPU time cost. Both methods can improve the computational accuracy with the refinement of mesh. The overall mesh refinement causes the CPU time cost to greatly increase as the mesh is refined. However, the CPU time cost does not increase very much with the increase of the level of h-adaptive mesh refinement. The CPU time cost can be saved by up to 90%, especially for the simulated system with a large number of elements and nodes.     

Query-driven visualization of time-varying adaptive mesh refinement data

The visualization and analysis of AMR-based simulations is integral to the process of obtaining new insight in scientific research. We present a new method for performing query-driven visualization and analysis on AMR data, with specific emphasis on time-varying AMR data. Our work introduces a new method that directly addresses the dynamic spatial and temporal properties of AMR grids that challenge many existing visualization techniques. Further, we present the first implementation of query-driven visualization on the GPU that uses a GPU-based indexing structure to both answer queries and efficiently utilize GPU memory. We apply our method to two different science domains to demonstrate its broad applicability.     

Texture-based volume rendering of adaptive mesh refinement data

Many phenomena in nature and engineering happen simultaneously on rather diverse spatial and temporal scales. In other words, they exhibit a multi-scale character. A special numerical multilevel technique associated with a particular hierarchical data structure is adaptive mesh refinement (AMR). This scheme achieves locally very high spatial and temporal resolutions. Due to its popularity, many scientists are in need of interactive visualization tools for AMR data. In this article, we present a 3D texture-based volume-rendering algorithm for AMR data that directly utilizes the hierarchical structure. Thereby fast rendering performance is achieved even for high-resolution data sets. To avoid multiple rendering of regions that are covered by grids of different levels of resolution, we propose a space partitioning scheme to decompose the volume into axis-aligned regions of equal-sized cells. Furthermore the problems of interpolation artifacts, opacity corrections, and texture memory limitations are addressed. Published online: November 6, 2002 Correspondence to: R. K?hler     

Practical applications of adaptive mesh refinement (Rezoning)

A collection of finite element problems that have been examined with a posteriori convergence measures is presented. Results show these local convergence measures can be used to indicate the quality of a finite element solution and to suggest boundaries for mesh rezones.     

Parallelization of structured,hierarchical adaptive mesh refinement algorithms

We describe an approach to parallelization of structured adaptive mesh refinement algorithms. This type of adaptive methodology is based on the use of local grids superimposed on a coarse grid to achieve sufficient resolution in the solution. The key elements of the approach to parallelization are a dynamic load-balancing technique to distribute work to processors and a software methodology for managing data distribution and communications. The methodology is based on a message-passing model that exploits the coarse-grained parallelism inherent in the algorithms. The approach is illustrated for an adaptive algorithm for hyperbolic systems of conservation laws in three space dimensions. A numerical example computing the interaction of a shock with a helium bubble is presented. We give timings to illustrate the performance of the method. Received: 28 April 1999 / Accepted: 25 November 1999     

A methodology for adaptive mesh refinement in optimum shape design problems

This work presents a methodology based on the use of adaptive mesh refinement (AMR) techniques in the context of shape optimization problems analyzed by the Finite Element Method (FEM). A suitable and very general technique for the parametrization of the optimization problem using B-splines to define the boundary is first presented. Then, mesh generation using the advancing front method, the error estimation and the mesh refinement criteria are dealt with in the context of a shape optimization problems. In particular, the sensitivities of the different ingredients ruling the problem (B-splines, finite element mesh, design behaviour, and error estimator) are studied in detail. The sensitivities of the finite element mesh coordinates and the error estimator allow their projection from one design to the next, giving an “a priori knowledge” of the error distribution on the new design. This allows to build up a finite element mesh for the new design with a specified and controlled level of error. The robustness and reliability of the proposed methodology is checked out with some 2D examples.     

Enabling scalable parallel implementations of structured adaptive mesh refinement applications

Parallel implementations of dynamic structured adaptive mesh refinement (SAMR) methods lead to significant runtime management challenges that can limit their scalability on large systems. This paper presents a runtime engine that addresses the scalability of SAMR applications with localized refinements and high SAMR efficiencies on large numbers of processors (upto 1024 processors). The SAMR runtime engine augments hierarchical partitioning with bin-packing based load-balancing to manage the space-time heterogeneity of the SAMR grid hierarchy, and includes a communication substrate that optimizes the use of MPI non-blocking communication primitives. An experimental evaluation on the IBM SP2 supercomputer using the 3-D Richtmyer-Meshkov compressible turbulence kernel demonstrates the effectiveness of the runtime engine in improving SAMR scalability.

Radiative cooling in numerical astrophysics: The need for adaptive mesh refinement

Energy loss through optically thin radiative cooling plays an important part in the evolution of astrophysical gas dynamics and should therefore be considered a necessary element in any numerical simulation. Although the addition of this physical process to the equations of hydrodynamics is straightforward, it does create numerical challenges that have to be overcome in order to ensure the physical correctness of the simulation. First, the cooling has to be treated (semi-)implicitly, owing to the discrepancies between the cooling timescale and the typical timesteps of the simulation. Secondly, because of its dependence on a tabulated cooling curve, the introduction of radiative cooling creates the necessity for an interpolation scheme. In particular, we will argue that the addition of radiative cooling to a numerical simulation creates the need for extremely high resolution, which can only be fully met through the use of adaptive mesh refinement.     

Scalable parallel regridding algorithms for block‐structured adaptive mesh refinement

Block‐structured adaptive mesh refinement (BSAMR) is widely used within simulation software because it improves the utilization of computing resources by refining the mesh only where necessary. For BSAMR to scale onto existing petascale and eventually exascale computers all portions of the simulation need to weak scale ideally. Any portions of the simulation that do not will become a bottleneck at larger numbers of cores. The challenge is to design algorithms that will make it possible to avoid these bottlenecks on exascale computers. One step of existing BSAMR algorithms involves determining where to create new patches of refinement. The Berger–Rigoutsos algorithm is commonly used to perform this task. This paper provides a detailed analysis of the performance of two existing parallel implementations of the Berger–Rigoutsos algorithm and develops a new parallel implementation of the Berger–Rigoutsos algorithm and a tiled algorithm that exhibits ideal scalability. The analysis and computational results up to 98 304 cores are used to design performance models which are then used to predict how these algorithms will perform on 100 M cores. Copyright © 2011 John Wiley & Sons, Ltd.     

A new MHD code with adaptive mesh refinement and parallelization for astrophysics

A new code, named MAP, is written in FORTRAN language for magnetohydrodynamics (MHD) simulations with the adaptive mesh refinement (AMR) and Message Passing Interface (MPI) parallelization. There are several optional numerical schemes for computing the MHD part, namely, modified Mac Cormack Scheme (MMC), Lax–Friedrichs scheme (LF), and weighted essentially non-oscillatory (WENO) scheme. All of them are second-order, two-step, component-wise schemes for hyperbolic conservative equations. The total variation diminishing (TVD) limiters and approximate Riemann solvers are also equipped. A high resolution can be achieved by the hierarchical block-structured AMR mesh. We use the extended generalized Lagrange multiplier (EGLM) MHD equations to reduce the non-divergence free error produced by the scheme in the magnetic induction equation. The numerical algorithms for the non-ideal terms, e.g., the resistivity and the thermal conduction, are also equipped in the code. The details of the AMR and MPI algorithms are described in the paper.     

Addressing spatiotemporal and computational heterogeneity in structured adaptive mesh refinement

Structured adaptive mesh refinement (SAMR) techniques can provide accurate and cost- effective solutions to realistic scientific and engineering simulations modeling complex physical phenomena. However, the adaptive nature and inherent space–time heterogeneity of SAMR applications result in significant runtime management challenges. Moreover, certain SAMR applications involving reactive flows exhibit pointwise varying workloads and cannot be addressed by traditional parallelization approaches, which assume homogeneous loads. This paper presents hierarchical partitioning, bin-packing based load balancing, and Dispatch structured partitioning strategies to manage the spatiotemporal and computational heterogeneity in SAMR applications. Experimental evaluation of these schemes using 3-D Richtmyer–Meshkov compressible turbulence and 2-D reactive-diffusion kernels demonstrates the improvement in overall performance.     

Three-dimensional MHD high-resolution computations with CWENO employing adaptive mesh refinement

Until recently, numerical simulations of discontinuities in highly super-Alfvénic plasmas have been severely limited by comparatively crude resolution and accuracy. Significant progress in the numerical simulation of such plasmas was achieved with the recently implemented Central Weighted Essentially Non-Oscillatory (CWENO) scheme. Combining this technique with that of adaptive mesh refinement (AMR), we have developed a third-order numerical scheme, which is able to efficiently capture strong gradients on spatial scales being small compared to the overall scale of the plasma system considered. Here, we first describe important algorithmic aspects of the scheme as well as the physics included in it. Second, we present the results of various performance tests. And, third, we illustrate its application to ‘real world problems’ using the example of the dynamics of a Sedov-type explosion.     

The NIRVANA code: Parallel computational MHD with adaptive mesh refinement

I report on a new version of the magnetohydrodynamics code NIRVANA¹ which is targeted at the study of astrophysical problems. The new version allows for distributed-memory simulations supporting adaptive mesh refinement. Numerical algorithms include dissipative terms (viscosity, Ohmic diffusion, thermal heat conduction) in a conservative form. Domain decomposition is preferably block-wise in case of unigrid applications but adopts space-filling curve techniques for adaptive mesh applications with a hierarchical block-structured mesh. The code architecture facilitates workload balancing among processors for arbitrary mesh refinement depths maintaining intra-level data locality via space-filling curve mappings and, at the same time, ensuring inter-level data locality by applying a novel technique called block sharing. This way, it is demonstrated that comparable performance can be achieved for problems with locally highly refined grid. The data transfer between processors extensively utilizes the coarse-granularity concept of parallel computing and makes use of the MPI library. Conservation properties of the numerical method carry over to the parallel framework. In particular, the solenoidality condition for the magnetic field is preserved to roundoff precision applying the constrained transport machinery. This paper has its focus of discussion on implementation details related to the parallelization and on a code performance analysis.