Instabilities in heterotic M-theory induced by open membrane instantons

We study the effective low energy supergravity of the strongly coupled heterotic string compactified on a Calabi-Yau 3-fold with generic E-8 x E-8 gauge bundle. We focus on the effective potential for the chiral scalars. The effective superpotential can be studied using the dual 11-dimensional M-theory background involving insertions of M5-branes along an interval. In such backgrounds, in some regions of moduli space, the leading nonperturbative contributions are due to open membrane instantons. These instantons lead to both attractive and repulsive forces between the 5-branes and the orientifold "M9-branes", depending on the region of moduli space. The resulting dynamics on moduli space include a strong coupling dual to the Dine-Seiberg instability, in which the interval grows. We discuss conditions under which the 5-branes are attracted to the wall and comment on the relevance of these results to the study of chirality-changing phase transitions in heterotic M-theory. (C) 2001 Published by Elsevier Science B.V.