Zhao has proposed that the microlensing events observed toward the Large Magellanic Cloud (LMC) could be due to faint stars in a dwarf galaxy or tidal debris lying along the line of sight to the LMC. Zaritsky & Lin claim to have detected such a structure which, they believe, could account for most of the observed microlensing optical depth. Here I show that a large-area surface-brightness map made by de Vaucouleurs constrains any such structure to one of four possibilities. Either (1) it does not account for a significant fraction of the observed microlensing, (2) it covers the inner similar to 3 degrees of the LMC but does not extend beyond similar to 5 degrees from the LMC center, (3) it is smooth on scales of similar to 15 degrees in both transverse directions, or (4) it has a stellar mass-to-light ratio which exceeds by a factor greater than or similar to 10 that of known stellar populations. The second and third possibilities would not be expected to apply to tidal debris. The last merely rephrases the dark-matter problem in a new form.