Instability of rotating convection

Convection rolls in a rotating layer can become unstable to the Kuppers-Lortz instability. When the horizontal boundaries are stress free and the Prandtl number is finite, this instability diverges in the limit where the perturbation rolls make a small angle with the original rolls. This divergence is resolved by taking full account of the resonant mode interactions that occur in this limit: it is necessary to include two roll modes and a large-scale mean flow in the perturbation. It is found that rolls of critical wavelength whose amplitude is of order epsilon are always unstable to rolls oriented at an angle of order E-2/5. However, these rolls are unstable to perturbations at an infinitesimal angle if the Taylor number is greater than 4 pi(4). Unlike the Kuppers-Lortz instability, this new instability at infinitesimal angles does not depend on the direction of rotation; it is driven by the flow along the axes of the rolls. It is this instability that dominates in the limit of rapid rotation. Numerical simulations confirm the analytical results and indicate that the instability is subcritical, leading to an attracting heteroclinic cycle. We show that the small-angle instability grows more rapidly than the skew-varicose instability.