I elect to stay on the soil of which I was born and on the plot of ground which I have fairly bought and honestly paid for. Don't advise me to leave, and don't add insult to injury by telling me it's for my own good; of that I am to be the judge.

They returned to Philadelphia where their family was among the black elite. In 1828 Purvis' older brother William died of tuberculosis which resulted in Robert and Joseph's having increased shares of the estate. His father was the English immigrant William Purvis.

He was born in Charleston South Carolina educated at Amherst College and lived most of his life in Philadelphia. Purvis and his brothers were three-quarters European by ancestry and inherited considerable wealth from their native English father.