We have measured the surface brightness of the far-infrared sky at lambda = 134, 154, and 186 mu m at high Galactic latitude using a liquid-He-cooled, rocket-borne telescope. The telescope scanned over a 5 degrees x 20 degrees region which includes infrared cirrus, high-latitude molecular clouds, the starburst galaxy M82, and the H I Hole in Ursa Major, a region with uniquely low H I column density. The measured brightness at 134, 154, and 186 mu m is well correlated with the 100 gamma m brightness measured by IRAS and, in regions excluding molecular clouds, with H I column density. The spectrum of the component correlated with H I is well fitted by a gray-body spectrum with a temperature of 16.4(+2.3/-1.8) K, assuming an emissivity proportional to lambda(-2). Assuming a constant far-infrared dust emissivity per hydrogen nucleus, the ratio of the H-2 column density to the velocity-integrated CO intensity in the high-latitude molecular cloud is N(H-2)/W-CO = (1.6+/-0.3) x 10(20) cm(-2) (K km s(-1))(-1). The residual brightness after subtracting the emission correlated with H I column density is lambda I-lambda(154 mu m) = (1.4+/-0.6) x 10(-12) W cm(-2) sr(-1), yielding an upper limit to the far-infrared extragalactic background radiation of lambda I-lambda(154 mu m) < 2.6 x 10(-12) W cm(-2) sr(-1).