The strain-induced 2D-3D growth mode transition in the molecular beam epitaxy of InxGa1-xAs on GaAs has been studied by in situ reflection high-energy electron diffraction (RHEED). For a given In composition, the thickness for which 3D islands appear is shown to be strongly dependent on the growth temperature. It is found that this thickness corresponds to the critical thickness (onset of strain release by dislocations formation) only for highly strained layers (x greater-than-or-equal-to 0.4). For lower strain (0.2 < x < 0.4), the 2D-3D transition is observed for a thickness larger than the critical one, which is shown to correspond to the occurrence of a surface disordering effect detected in RHEED. Finally, when the In composition is decreased such as x < 0.2, the 2D-3D transition is no more observed. In addition, the comparison of the RHEED analysis and photoluminescence results obtained on In0.35Ga0.65As/GaAs quantum wells shows that the degradation of the optical properties occurs for a well thickness corresponding to the growth mode transition rather than to the critical layer thickness.