The 41-year-old recently stepped off the set of her new film Alexander and the Terrible Horrible No Good Very Bad Day to speak about what went wrong the first time.
"We didn't have a shot," Jennifer told Allure magazine.
"Oh, he's a great guy. We were full-on grown-ups, but looking back I'm aware we did not know what hit us," she said of the break-up with her previous husband of four years.
Jennifer and Scott worked together on TV series Felicity in 1998. They married in an October ceremony two years later.
In March 2003, however, they separated, finalising their divorce one year later.
"We didn't have a shot. He's a really good guy, and we just imploded," said the actress, who months later got together with Ben thanks to working together on the set of Daredevil.
The handsome Hollywood couple tied the knot in June of 2005 and now have three beautiful children Violet, seven, Seraphina, four, and Samuel who is 18 months old.
Scott has also gone on to find love again, marrying fellow actress Marika Dominczyk in 2007 and having two children Malina, three, and Keller, 16 months, together.
Golden Globe winner Jennifer also opened up about what she has learned from her current marriage to Ben - to be cautious about attending Hollywood events together.
She said that it can be hard for the wives of famous men as they tend to lose their identity when they become "the wife of".
Ben moved millions of viewers across the world when he spoke openly to his wife of their marriage in his touching Best Picture acceptance speech at the Oscars awards in February.
After paying tribute to the stars and cast of his winning film Argo, he turned to look at his actress wife Jennifer Garner in the audience.
"Thank you for working on our marriage," he said. "It is work – but it's the best kind of work," he quipped. "There is no-one else I would ever work with."