Normally, when people hear buff they think of polishing or rubbing. I recently learnt that it’s also a color… a pale yellowish brown color found on undyed leather of some animals.
The first recorded use of the word “buff” to describe a colour was in the London Gazette of 1686, describing a uniform to be “…a Red Coat with a Buff-colour’d lining” (Wikipedia). Such buff leather was uset for buffing or serving as a buffer between polished objects.
Some sand and rocks are buff colored, almost a light beige-yellow color. A lot of species are named because of their buff coloring…like buff arches moth, buff scales on snakes which help them hide in sand, buff fungi that help them hide on rocks etc.