Three-dimensional numerical simulations of non-axisymmetric dynamical instabilities in rotating stars are presented. These instabilities may operate during the collapse phase of rapidly rotating pre-supernova cores or in neutron stars spun up by accretion. Using a smoothed particle hydrodynamics (SPH) code, the effect of varying the rotation rate on the gravitational wave quantities, growth rate, and pattern speed of the instabilities is investigated. The star is modelled as a polytrope with index n = 0.5 and initially possesses T(rot)/\W\ = 0.26, 0.28, 0.30, 0.32, and 0.34. Here T(rot) is the rotational kinetic energy and \W\ is the gravitational potential energy of the entire system. The differentially rotating initial models are generated using a new rotational cooling method which properly allows odd-mode growth. This cooling technique is an important improvement over a previously used method that retained artificial grid symmetries and exhibited unrealistic odd-mode evolution. The code assumes a Newtonian gravitational field, and the gravitational radiation is calculated in the quadrupole approximation. This analysis shows that either a bar or a pear mode instability develops if the initial model possesses T(rot)/\W\ greater than or similar to 0.28. The instability sheds between 4-21 per cent and 14-62 per cent of the initial mass and angular momentum, respectively, in the form of a one- or two-armed spiral pattern. All initial models with T(rot)/\W\ greater than or equal to 0.28 become unstable to the growth of nonaxisymmetric instabilities and evolve toward an axisymmetric state surrounded by a flattened disc, The duration, amplitude, and frequency of the gravitational wave signals are sensitive to the initial value of T(rot)/\W\. Increases in this quantity result in a greater gravitational wave amplitude and frequency, but the duration of these signals is inversely proportional to increases in the initial stellar core rotation rate.
