Spencer was raised in Milwaukee, WI. He received his BA and PhD from Duke University.
His wife Melissa Banzhaf also is an economist, specializing in the labor market. With a BA in economics from Duke and PhD from the University of North Carolina, she serves as the administrator of the Atlanta Research Data Center.
They have two children, Elizabeth and John. They attend North Avenue Presbyterian Church, where Spencer is an ordained elder.