"Faculae" (Latin for "small torches") is a term applied to the small-scale (~1000~km in diameter or less) components of the solar magnetic field when they are seen as bright patterns near the solar limb. The term encompasses both magnetic fields that surround sunspots as well as elements of the supergranular network formed by active region decay and small-scale "ephemeral region" emergence. On time scales of months, the brightness of faculae more than compensates for the deficit in solar irradiance caused by dark sunspots and thus is primarily responsible for the total solar irradiance increase of about 0.1% seen during the peak of the 11-year sunspot cycle. But small-scale magnetic fields exist all over the solar disk - why are they brightest only at the limb? Here we analyze 100~km resolution images of the solar photosphere - the highest resolution images of the Sun yet obtained - to show that small-scale magnetic fields less than about 350~km in diameter are bright at all disk positions while structures larger than this (so-called "micropores") are dark near disk center. However as micropores rotate towards the limb their centerward walls become the brightest facular elements. Thus small-scale magnetic structure irradiance is highly size and disk-position dependent. We compare recent 3D compressible magnetohydrodyamic simulations that closely replicate the observations. The simulations imply that lateral radiative transfer through the low-density magnetic field is primarily responsible for the brightness of the "hot-walls" of faculae seen at the limb.