Performance of parallel implementations of an explicit finite-volume shallow-water model

Explicit total variation diminishing finite-volume schemes are being adopted on a widespread basis for the solution of depth-averaged hydrodynamic equations. Explicit schemes are constrained by the Courant-Friedrichs-Lewy condition for stability purposes, and therefore require use of a small time step. As grid resolution increases, the ratio of run time to integration time may approach unity, so strategies to reduce run times are sought. This paper characterizes the performance gains of two parallel computing optimizations exercised on two different computer architectures. The optimizations include removal of explicit synchronization mechanisms (Level 1) and conversion of blocking to nonblocking communications (Level 2). Our findings show that Level 1 always improves speed-up over Level 0, while the effectiveness of Level 2 over Level 1 is mixed. Level 2 results in the best performance on a system with a relatively small bandwidth (100 Mb) interconnect switch, but in a few cases involving a system with a gigabit interconnect switch, Level 2 actually leads to slow-down as compared to Level 1. Level 2 was found to be more difficult to implement than Level 1, and the resulting code was less modular and more difficult to read. Overall, the marginal performance improvements of nonblocking communications (Level 2) cannot justify the effort to realize the optimization and the cost of a less readable program. In the context of algorithm development, we emphasize delaying optimizations until a correct parallel implementation has been obtained. The benefit is that optimizations best suited to the underlying hardware architecture can be identified.