It is now well-known that the wake transition regime for a circular cylinder involves two modes of small-scale three-dimensional instability (modes A and B), depending on the regime of Reynolds number (Re), although almost no understanding of the physical origins of these instabilities, or indeed their effects on near-wake formation, have hitherto been made clear. We address these questions in this paper. In particular, it is found that the two different modes A and B scale on different physical features of the flow. Mode A has a larger spanwise wavelength of around 3-4 diameters, and scales on the larger physical structure in the flow, namely the primary vortex core. The wavelength for mode A is shown to be the result of an 'elliptic instability' in the near-wake vortex cores. The subsequent nonlinear growth of vortex loops is due to a feedback from one vortex to the next, involving spanwise-periodic deformation of core vorticity, which is then subject to streamwise stretching in the braid regions. This mode gives an out-of-phase streamwise vortex pattern. In contrast, mode-B instability has a distinctly smaller wavelength (1 diameter) which scales on the smaller physical structure in the flow, the braid shear layer. It is a manifestation of an instability in a region of hyperbolic flow. It is quite distinct from other shear flows, in that it depends on the reverse flow of the bluff-body wake; the presence of a fully formed streamwise vortex system, brought upstream from a previous half-cycle, in proximity to the newly evolving braid shear layer, leads to an in-phase streamwise vortex array, in strong analogy with the 'Mode 1' of Meiburg & Lasheras (1988) for a forced unseparated wake. In mode B, we also observe amalgamation of streamwise vortices from a previous braid with like-sign vortices in the subsequent braid. It is deduced that the large scatter in previous measurements concerning mode A is due to the presence of vortex dislocations. Dislocations are triggered at the sites of some vortex loops of mode A, and represent a natural breakdown of the periodicity of mode A. instability. By minimizing or avoiding the dislocations which occur from end contamination or which occur during wake transition, we find an excellent agreement of both critical Re and spanwise wavelength of mode A with the recent secondary stability analysis of Barkley & Henderson (1996). Wake transition is further characterized by velocity and pressure measurements. It is consistent that, when mode-A instability and large-scale dislocations appear, one finds a reduction of base suction, a reduction of (two-dimensional) Reynolds stress level, a growth in size of the formation region, and a corresponding drop in Strouhal frequency. Finally, the present work leads us to a new clarification of the possible flow states through transition. Right through this regime of Re, there exist two distinct and continuous Strouhal frequency curves: the upper one corresponds with purely small-scale instabilities (e.g, denoted as mode A), while the lower curve corresponds with a combination of small-scale plus dislocation structures (e.g. mode A*). However, some of the flow states are transient or 'unstable', and the natural transitioning wake appears to follow the scenario: (2D --> A* --> B).