Bimolecular triple-helical DNA complexes recently have found use in a new strategy for the recognition of single-stranded nucleic acids, in which circular (Kool, 1991; Prakash & Kool, 1992) or hairpin-shaped (Giovannangeli et al., 1991; D'Souza & Kool, 1992) oligonucleotides bind these single strands by tripler formation. Bimolecular triplexes may also be formed in vivo as H-DNA, where this structure may potentially play a role in gene expression and recombination (Belotserkovskii et al., 1990; Hanvey et al., 1989; Shimizu et al., 1989). In all of these complexes, the central strand of the tripler must pass beyond the loop that bridges the outer two strands, and models and preliminary experiments have indicated that there may be important interactions between this central strand and the loop (Prakash & Kool, 1992). We now report thermal denaturation studies carried out specifically to investigate these interactions in detail, using as a model the 5'-loop and 3'-loop complexes formed between 14 pyrimidine oligodeoxynucleotides having the sequence 5'-dTTCTTTTCL(1)TTTL(5)CTTTTCTT, where L(1) and L(5) represent varied nucleotides in the loop (which is underlined), and eight target strands having the sequence 5'-dCCCCCFAAGAAAAG-3' or 5'-dGAAAAGAAFCCCCC-3', where F is a varied nucleotide flanking the tripler in the central strand. Results correlated from 64 different sequence combinations show that there is wide variation in the stabilities of the complexes, indicating specific and substantial interactions between the nucleotides at the L(1), F, and LS positions. Melting temperatures at pH 7.0 range from 17.0 degrees C to 34.6 degrees C, and free energies (37 degrees C) range from -3.2 to -7.8 kcal mol(-1). Several general conclusions are drawn from the 64 combinations studied: (1) Extra stability is gained when one of the first (L(1)) or fifth (L(5)) nucleotides in the loop is complementary to the F nucleotide in the purine strand, with an average advantage of 1.9-3.0 degrees C in T-m and 0.6 kcal in free energy. (2) Even higher stability is gained when both the L(1) and L(5) nucleotides are complementary to the flanking F nucleotide. The advantage (relative to no complementarity) is 3.2-4.7 degrees C in T-m and 1.1-1.5 kcal in free energy, on average; however, evidence indicates that this interaction is not the result of standard triple-helix pairing. (3) The correct choice of loop nucleotides can add both binding affinity and sequence selectivity to bimolecular triplexes. Binding studies of two circular oligodeoxynucleotides constructed as a test of the loop studies shows that the results allow semiquantitative predictions of the stabilities of bimolecular triplexes that involve these loop interactions. In the design of synthetic oligonucleotides as tripler-forming agents, the results suggest optimal choices of loop nucleotides for binding a given target sequence. The results may also aid in the understanding of the relative stabilities of H-DNA complexes.