New exaFMM paper accepted
FMM-based vortex method for simulation of isotropic turbulence on GPUs, compared with a spectral method, Rio Yokota, L. A. Barba, Computers & Fluids, in press (available online 13 August 2012), doi:10.1016/j.compfluid.2012.08.002
This paper presents the results of comparing a Lagrangian vortex method with a trusted spectral method for the simulation of isotropic fluid turbulence. The numerical engine of the vorticity-based fluid solver is a massively parallel fast multipole method (FMM) running on GPU hardware using CUDA. Thus, there are several aspects to the validation: the particle method itself, the fast summation via FMM, and the use of GPUs in this application.
There have been insufficient efforts so far to fully validate the vortex method in direct numerical simulations of turbulent flows, which has left ample room for skepticism. The benchmark of decaying, homogeneous isotropic turbulence is an ideal test for quantitave validation of fluid solvers, with the simplest possible geomety (a periodic cube) and many years of experience using turbulence statistics in its analysis. It is not the best problem for showcasing a vortex particle method, which shines in wake flows and other flows dominated by vorticity, but it allows quantitative validation. We use a Taylor microscale Reynolds number of 50 and 100 in a 256^3 mesh and look at various turbulence statistics, including high-order moments of velocity derivatives. A parametric study looks at the effect of the order of series expansion in the FMM, the frequency of particle reinitialization in the vortex method, the overlap ratio of particles and the time step. At this resolution, the vortex method matches quantitatively the spectral method at Re=50, but some deviations are appreciable in the velocity skewness and flatness for Re=100.
The vortex method application uses a parallel FMM code, called exaFMM, that runs on GPU hardwared using CUDA, while the reference spectral code (developed and used at the Center for Turbulence Research in Stanford) runs on CPU only. Results indicate that, for this application, the spectral method is an order of magnitude faster than the vortex method when using a single GPU for the FMM and six CPU cores for the FFT.
The entire code that was used to obtain the present results is available under the MIT license from: https://bitbucket.org/exafmm/exafmm
The revision number used for the results in the paper was 146. Documentation and links to other publications asre found int he project homepage at http://exafmm.org
In another publication (coming soon), we will focus on the performance of the FMM-based vortex method on massively parallel systems, compared to the spectral method. That work included simulations of isotropic turbulence on up to 4096 GPUs, with a 4096^3 problem size and exceeding 1 petaflop/s of sustained performance. Preliminary results are available via the following figure:
- "Weak scaling of parallel FMM vs. FFT up to 4096 processes", Rio Yokota, L. A. Barba. (18 June 2012). 10.6084/m9.figshare.92425 // exaFMM webpage
Figure, data and plotting script published on figshare under CC-BY