SCARF is a color vision system that recognizes difficult roads and intersections. It has been integrated into several navigation systems that drive a robot vehicle, the Navlab, on a variety of roads in many different weather conditions. SCARF recognizes roads that have degraded surfaces and edges with no lane markings in difficult shadow conditions. It also recognizes intersections with or without predictions from the navigation system. This is the first system that detects intersections in images without a priori knowledge of the intersection shape and location. SCARF uses Bayesian classification, a standard pattern recognition technique, to determine a road-surface likelihood for each pixel in a reduced color image. It then evaluates a number of road and intersection candidates by matching an ideal road-surface likelihood image with the results from the Bayesian classification. The best matching candidate is passed to a path-planning system that navigates the robot vehicle on the road or intersection. This paper describes the SCARF system in detail, presents results on a variety of images, and discusses the Navlab test runs using SCARF.