Atmospheric airglow lines, in particular those from the OH molecule, are a major source of background radiation at near-infrared wavelengths (1-2.3 mu m). Because of this, several observatories around the world are investigating the potential for blocking these emission lines using instruments based on filters or diffraction gratings. This paper presents numerical calculations of the performance of such devices. A flexible computer program automates the evaluation of several OH-blocking schemes and allows optimization of instrument parameters. A number of important conclusions arise from the study of filter-based and general grating (dispersive) configurations designed for the H(1.5-1.8 mu m) and J(1.13-1.37 mu m) photometric bands: (1) State-of-the-art multilayer dielectric bandpass filters tailored to isolate a relatively or completely emission-free region do not provide provide arty improvement over the broadband H filter itself Similarly, a high spectral resolution Fabry-Perot interferometer acting as a fixed filter gives disappointing performance. This is due to the shape of technically feasible filter profiles and the strength and distribution of OH lines in the H band. (2) Several small gaps in the distribution of J-band emission lines provide the opportunity for moderate (similar to 0.5 mag) improvement in sensitivity using low-cost dielectric bandpass filters. The broad wings in the transmission profile of Fabry-Perot interferometers makes them impractical to take advantage of these gaps. (3) A stack of nearly ideal dielectric notch filters designed to remove the brightest individual OH lines gives poor performance in both the H and J photometric bands, and the performance degrades with increasing number of filters in series. Broader blocking filters centered on the brightest groupings of lines produce no improvement in sensitivity in the H band. Blocking all the J airglow lines leaves the equivalent of two bandpass filters at the short and long-wavelength extremes of the band, and provides a moderate (similar to 0.5 mag) increase in performance. (4) Dispersive instruments, such as those employing gratings, can provide similar to 1.3 mag of improved sensitivity in the H band, and similar to 1.6 mag in the J band. The improvement in sensitivity increases monotonically with the spectral resolution of the dispersive instrument, although significant gains are attainable with R greater than or equal to 4000. All the OH lines should be blocked to give reasonable performance. It is important to understand and control diffraction and scattering in dispersive systems. Each additional percent of unblocked OH-line radiation leads to a 10% decrease in sensitivity. (5) The H- and J-band airglow continuum is poorly understood, and few precise measurements exist in the literature. It is imperative to improve this situation before major instrumentation projects begin, since the range of published fluxes encompasses values that can make such instruments pointless.