Spencer was raised in Milwaukee, WI. He received his BA and PhD from Duke University.
His wife Melissa Banzhaf also is an economist, specializing in the labor market. With a BA in economics from Duke and PhD from the University of North Carolina, she works with the US Census and the Federal Statistical Research Data Centers.
They have two children, Elizabeth and John. They attend the Church of the Apostles in Raleigh, NC.