We overview the current status and future development directions of our framework for developing particle simulator (FDPS). Many of particle-based simulation codes share the same characteristic that the most time-consuming part of the simulation is the calculation of the interactions between particles, and a large fraction of programming effort is spent for procedures to make the force calculation efficient, such as the decomposition of computational domain, exchange of particles between domains, exchange of information necessary to calculate the interaction to particles in different domains, and efficient neighbor search. The basic idea of FDPS is to provide generic and high-performance library for these procedures. Using these procedures, researchers or application programmers in various fields can write their programs without taking care of parallelization and performance tuning. In order to make FDPS useful on advanced HPC platforms at present and in (near) future, we investigated its performance on several modern platforms and learned what can be the bottleneck. In this paper we summarize what we learned.

We describe algorithms implemented in FDPS (Framework for Developing Particle Simulators) to make efficient use of accelerator hardware such as GPGPUs (general-purpose computing on graphics processing units). We have developed FDPS to make it possible for researchers to develop their own high-performance parallel particle-based simulation programs without spending large amounts of time on parallelization and performance tuning. FDPS provides a high-performance implementation of parallel algorithms for particle-based simulations in a "generic" form, so that researchers can define their own particle data structure and interparticle interaction functions. FDPS compiled with user-supplied data types and interaction functions provides all the necessary functions for parallelization, and researchers can thus write their programs as though they are writing simple non-parallel code. It has previously been possible to use accelerators with FDPS by writing an interaction function that uses the accelerator. However, the efficiency was limited by the latency and bandwidth of communication between the CPU and the accelerator, and also by the mismatch between the available degree of parallelism of the interaction function and that of the hardware parallelism. We have modified the interface of the user-provided interaction functions so that accelerators are more efficiently used. We also implemented new techniques which reduce the amount of work on the CPU side and the amount of communication between CPU and accelerators. We have measured the performance of N-body simulations on a system with an NVIDIA Volta GPGPU using FDPS and the achieved performance is around 27% of the theoretical peak limit. We have constructed a detailed performance model, and found that the current implementation can achieve good performance on systems with much smaller memory and communication bandwidth. Thus, our implementation will be applicable to future generations of accelerator system.

In this paper, we report the implementation and measured performance of our extreme-scale whole planetary ring simulation code on Sunway TaihuLight and two PEZY-SC2 systems: Shoubu System B and Gyoukou. The numerical algorithm is the parallel Barnes-Hut tree algorithm, which has been used in many large-scale astrophysical particle-based simulations. Our implementation is based on our FDPS framework. However, the extremely large numbers of cores of the systems used (10 M on TaihuLight and 16 M on Gyoukou) and their relatively poor memory and network bandwidth pose new challenges. We describe the new algorithms introduced to achieve high efficiency on machines with low memory bandwidth. The measured performance is 47.9, 10.6 PF, and 1.01PF on TaihuLight, Gyoukou and Shoubu System B (efficiency 40%, 23.5% and 35.5%). The current code is developed for the simulation of planetary rings, but most of the new algorithms are useful for other simulations, and are now available in the FDPS framework.

Numerical simulations based on particle methods have been widely used in various fields including astrophysics. To date, simulation softwares have been developed by individual researchers or research groups in each field, with a huge amount of time and effort, even though numerical algorithms used are very similar. To improve the situation, we have developed a framework, called FDPS, which enables researchers to easily develop massively parallel particle simulation codes for arbitrary particle methods. Until version 3.0, FDPS have provided API only for C++ programing language. This limitation comes from the fact that FDPS is developed using the template feature in C++, which is essential to support arbitrary data types of particle. However, there are many researchers who use Fortran to develop their codes. Thus, the previous versions of FDPS require such people to invest much time to learn C++. This is inefficient. To cope with this problem, we newly developed a Fortran interface layer in FDPS, which provides API for Fortran. In order to support arbitrary data types of particle in Fortran, we design the Fortran interface layer as follows. Based on a given derived data type in Fortran representing particle, a Python script provided by us automatically generates a library that manipulates the C++ core part of FDPS. This library is seen as a Fortran module providing API of FDPS from the Fortran side and uses C programs internally to interoperate Fortran with C++. In this way, we have overcome several technical issues when emulating `template' in Fortran. By using the Fortran interface, users can develop all parts of their codes in Fortran. We show that the overhead of the Fortran interface part is sufficiently small and a code written in Fortran shows a performance practically identical to the one written in C++.

In this paper, we report the implementation and measured performance of a global simulation of planetary rings on Sunway TaihuLight. The basic algorithm is the Barnes-Hut tree, but we have made a number of changes to achieve good performance for extremely large simulations on machines with an extremely large number of cores. The measured performance is around 35% of the theoretical peak. The main limitation comes from the performance of the interaction calculation kernel itself, which is currently around 50%.

We explore the gas dynamics near the dust sublimation radius of active galactic nucleus (AGN). For the purpose, we perform axisymmetric radiation hydrodynamic simulations of a dusty gas disc of radius approximate to 1 pc around a supermassive black hole of mass 10(7) M-circle dot taking into account (1) anisotropic radiation of accretion disc, (2) X-ray heating by corona, (3) radiative transfer of infrared (IR) photons re-emitted by dust, (4) frequency dependence of direct and IR radiations, and (5) separate temperatures for gas and dust. As a result, we find that for Eddington ratio approximate to 0.77, a nearly neutral, dense (approximate to 10(6-8) cm(-3)), geometrically thin (h/r < 0.06) disc forms with a high-velocity (approximate to 200 similar to 3000 km s(-1)) dusty outflow launched from the disc surface. The disc temperature is determined by the balance between X-ray heating and various cooling, and the disc is almost supported by thermal pressure. Contrary to Krolik (2007), the radiation pressure by IR photons is not effective to thicken the disc, but rather compresses it. Thus, it seems difficult for a radiation-supported, geometrically thick, obscuring torus to form near the dust sublimation radius as far as the Eddington ratio is high (similar to 1). The mass outflow rate is 0.05-0.1 M-circle dot yr(-1) and the column density of the outflow is N-H similar to 10(21) cm(-2). To explain observed type-II AGN fraction, it is required that outflow gas is extended to larger radii (r greater than or similar to 10 pc) or that a denser dusty wind is launched from smaller radii (r similar to 10(4) R-g).

We perform three-dimensional radiation hydrodynamic simulations of uniform dusty gas clouds irradiated by an active galactic nucleus (AGN) to investigate the dependence of evolution of clouds on the ionization parameter $\mathcal{U}$ and the Str{\"o}mgren number $\mathcal{N}_{S}$. We find that the evolution can be classified into two cases depending on $\mathcal{U}$. In low $\mathcal{U}$ cases ($\mathcal{U}\approx 10^{-2}$), the evolution is mainly driven by photo-evaporation. A approximately spherically-symmetric evaporation flow with velocity of $100\operatorname{-}150\;\mathrm{km\;s^{-1}}$ is launched from the irradiated face. The cloud is compressed by a D-type shock with losing its mass due to photo-evaporation and is finally turned into a dense filament by $t\lesssim 1.5t_{\mathrm{sc}}$. In high $\mathcal{U}$ cases ($\mathcal{U}\approx 5\times 10^{-2}$), radiation pressure suppresses photo-evaporation from the central part of the irradiated face, reducing photo-evaporation rate. A evaporation flow from the outskirts of the irradiated face is turned into a high velocity ($\lesssim 500\;\mathrm{km\;s^{-1}}$) gas wind because of radiation pressure on dust. The cloud is swept by a radiation pressure-driven shock and becomes a dense gas disk by $t\approx t_{\mathrm{sweep}}$. Star formation is expected in these dense regions for both cases of $\mathcal{U}$. We discuss the influences of the AGN radiation on the clumpy torus. A simple estimate suggests that the clumps are destroyed in timescales shorter than their orbital periods. For the clumpy structure to be maintained over long period, the incident radiation field needs to be sufficiently weaken for most of the clumps, or, some mechanism that creates the clumps continuously is needed.

We investigate the properties of massive, dense clouds formed in a barred galaxy and their possible relation to star formation, performing a two-dimensional hydrodynamical simulation with the gravitational potential obtained from the 2Mass data from the barred spiral galaxy, M83. Since the environment for cloud formation and evolution in the bar region is expected to be different from that in the spiral arm region, barred galaxies are a good target to study the environmental effects on cloud formation and the subsequent star formation. Our simulation uses for an initial 80 Myr an isothermal flow of non-self gravitating gas in the barred potential, then including radiative cooling, heating and self-gravitation of the gas for the next 40 Myr, during which dense clumps are formed. We identify many cold, dense gas clumps for which the mass is more than $10^4M_{\odot}$ (a value corresponding to the molecular clouds) and study the physical properties of these clumps. The relation of the velocity dispersion of the identified clump's internal motion with the clump size is similar to that observed in the molecular clouds of our Galaxy. We find that the virial parameters for clumps in the bar region are larger than that in the spiral arm region. From our numerical results, we estimate star formation in the bar and spiral arm regions by applying the simple model of Krumholtz and McKee (2005). The mean relation between star formation rate and gas surface density agrees well with the observed Kennicutt-Schmidt relation. The SFE in the bar region is $\sim 60 %$ of the spiral arm region. This trend is consistent with observations of barred galaxies.

We investigate the gas supply process from a nuclear gas disk to the Galactic center by performing two-dimensional hydrodynamic simulations including self-gravity and radiative cooling. We assume that the gas is supplied to the nuclear gas disk by the host galaxy disk. We show that many gas clumps are formed in the nuclear gas disk after the nuclear gas disk becomes massive enough to become gravitationally unstable. Coalescence of the gas clumps leads to the formation of more massive gas clumps. Gravitational encounters between these massive gas clumps induce angular momentum transfer between the gas clumps. Massive gas clumps that lose their angular momentum due to gravitational encounters move to the galactic center. In this way, the gas supply to the galactic center is realized. We also perform a numerical simulation with twice the spatial resolution to check numerical resolution effects. Time evolution of azimuthally averaged surface density distributions in both models agree well with each other. The high-mass part of the clump mass function in the low-resolution model is in good agreement with that of the high-resolution model, and lower mass clumps are well resolved in the high-resolution model. Therefore, we conclude that massive gas clumps play a major role in the angular momentum transfer. The gas supply rate to the center is about half of the gas supply rate from the host galaxy when the whole region of the nuclear gas disk becomes self-gravitationally unstable. More massive gas clumps are formed and migrate to the Galactic center in the models of the higher gas supply rate from the host galaxy.

We investigate rapid mass supply process by nested bars in the Galaxy by numerical simulation. We simulate gas flow in the whole galaxy disk with nested bars, which are the outer bar and the inner bar, especially with highly spatial resolution in the galactic central region. We assume two cases of inner bar size which are a smaller one and a larger one than the radius of the 200 pc gas ring which is corresponds to the Central Molecular Zone. From our numerical results, in the large size bar cases, the inner bars with large elongation induce sufficient mass inflow and destroy the 200 pc gas ring. On the other hand, in the small size bar cases, the inner bars with large elongation induce large mass inflow and do not destroy the 200 pc gas ring. This mass inflow is caused by straight shocks excited by the inner bar. In this case, nuclear gas disks of ~ 15 pc radius are formed. The nuclear gas disks are self-gravitationally unstable and we expect formation of compact star clusters under strong tidal force in the nuclear gas disks. We discuss evolution of the nuclear gas disk.

We investigate possible roles of a inner bar, evidence of which is recently given in our galaxy in the gas fueling to the Galactic center. For this purpose, we perform two dimensional hydrodynamical simulations in the gravitational potential model of our galaxy, which take into account the inner bar. In our simulations, a massive nuclear disk (M similar to 10(7) M-circle dot and R similar to 20 pc) is usually formed by a large gas inflow clue to the inner bar. We show that such nuclear disks can fragment into many small massive clumps and a part of these clumps spiral into the galactic center by the exchange of angular momentum between the clumps.