Smart’s childhood was marked by pain and loss. He grew up in a tough Dallas neighborhood that he described as a “war zone.” His older brother Michael Smart was involved with the Bloods street gang, sold drugs, and was pimping. Smart tried to convince his brother to leave the world of gangs and drugs, and he waited up all night to ensure his brother got home safely. The family moved to Flower Mound, a suburb of Dallas, away from the gangs and violence, and Smart began taking anger management classes.

In addition to his brother's troubles, Smart’s oldest brother Todd Westbrook, a talented basketball player, died of cancer when Smart was just nine years old. Westbrook was seen as a second father figure to Smart and was an amazing basketball star who could have made it in the NBA. Westbrook's battle with cancer taught Smart about perseverance and never giving up.

Smart's mother, Camellia Smart, played a significant role in his life. She was always his biggest fan and critic. He learned to be a team player from her, and she always encouraged him to play hard. After Smart made it to the NBA, he bought his mother a new house in the Dallas suburbs, fulfilling a promise he had made to her.

Camellia Smart battled bone marrow cancer and passed away in September 2018. Smart revealed that her death was peaceful and that he was grateful he was able to be with her in her final moments. During the Celtics’ postseason run in 2022, Smart dyed his hair green in honor of his mother, who had always loved his hair and different hairstyles.

In summary, Smart's childhood was marked by pain and loss, but he overcame the challenges he faced to become an NBA star. He learned about perseverance from his oldest brother, and his mother played a significant role in his life. Although she is no longer with him, he continues to honor her memory.