The words have never drifted far from Bettye Kearse’s mind. She heard them over and over from her mother and grandfather. They evoke strength and hope, an unwavering pride in family and resilience. They will forever be a part of who she is: “Always remember—you’re a Madison. You come from African slaves and a president.”

Now, in her new book, The Other Madisons: The Lost History of a President’s Black Family (Houghton Mifflin Harcourt, March 24), Kearse examines her place in the story as the family griotte, a traditional oral storyteller who passes her family’s history on to the next generation.

The Other Madisons is a unique blend of history, mystery, fiction, and memoir following Kearse as she travels the world to connect with her ancestors and seeks DNA proof of a connection to James Madison Jr., the fourth president of the United States.

But that relationship, so revered by her elders, resonates differently to a pediatrician with a Ph.D. in biology who came of age during the civil rights movement. Previous generations focused on pride. Kearse refuses to look away from the fact that a future president fathering a child with a slave inevitably meant rape.

“My mother never questioned anything that had been passed down to her,” Kearse says by phone from her home in New Mexico. “She was just very happy to be able to say she was a descendant of a president. She never thought what that meant….She probably expected me to continue to emphasize the pride woven into the credo and not go out there and start talking about negative things.”

Kearse, who has a brother, never expected to become the griotte until 1990, when her mother passed along the family’s precious box of photographs, letters, deeds, birth certificates, and other papers. Her first emotional connection to the story was Mandy, a young African girl stolen into slavery and sold to the Madison plantation in Virginia.

Mandy grew so large in Kearse’s imagination that she has included fictional passages from Mandy’s point of view in The Other Madisons, an unusual choice for a work of nonfiction.

“Those are my favorite parts of the book,” she says. “It allows my ancestors to come alive.”

Kearse follows the trail of the African slave trade to Lagos, Ghana, and the Madison plantation. There, walking the path Mandy’s daughter Coreen walked, she felt an undeniable connection.

“It was just an incredible experience to walk in an ancestor’s footsteps, to literally walk where she had walked,” Kearse says. “Going to the slaves’ cemetery was also nearly overwhelming, in a good way. I just knew that’s where Mandy was. It was such a strong feeling. I can’t even really explain it. It reassured me about all the stories I had heard about Mandy and some of my other enslaved ancestors. They were real people who contributed toward making me who I am.”

Unlike the descendants of Sally Hemings, who proved that President Thomas Jefferson was part of their family tree via DNA, Kearse was unable to genetically link James Madison to her family. A Madison descendant refused a DNA test, leaving Kearse with a dilemma: Was there a book if there was no genetic proof?

“As a writer, it was disappointing,” she admits. “I was well aware if I had proof of DNA, publishers would be tracking me down, and the public would’ve been enthralled. But as someone who came to believe so much in these slaves, who are part of who I am and part of what this country became, the DNA didn’t come to matter as much….The most important things are the values my ancestors passed down.”

Kearse’s mother told her she shouldn’t think badly of the Madisons, but after her years of research, Kearse can’t keep that promise.

“Yes, it was part of the culture of the time, but they knew slavery was wrong,” she says. “Especially James Madison Jr. The president wrote that it was wrong, yet he never freed a single slave.”

Still, confronting the horrors her ancestors survived has left Kearse with a sense of gratitude. “I came to terms with it by really looking at the strength these enslaved people had,” Kearse says. “Not only did they have inner strength, they had a sense of hope that enables them to recognize themselves as being more than property. They were people with many gifts. They gave me values and love of family, but I think probably the most important thing they passed down is hope.”

Connie Ogle is a writer in Florida.