But the couple had some sad news to share too – they lost the little boy Kayte was also expecting.
"Early this morning Kayte gave birth to Faith Evangeline Elisa Grammer," the couple said in a joint statement on Friday.
"We are thrilled. She was 6lbs 2oz when she entered the world at 1am on the 13th of July in the year 2012. Mother and child are in excellent health."
"We were ecstatic earlier this year, when we announced that Kayte was carrying twins. Tragically we lost the little boy shortly thereafter.
"This was not something we cared to make known publicly at the time.
"It was unspeakably painful and we know that people will understand our desire to keep the news private then, as we know they will respect our privacy in this matter now. A glorious birth with a lingering sadness is ours today.
"We choose to celebrate the life that has been given us. We proudly introduce our Faith to the world today looking forward to the days ahead and the children yet to come."
Frasier actor Kelsey and former air stewardess Kayte married in February 2011.
They tragically lost another baby in 2010 when 32-year-old Kayte suffered a miscarriage.
Fifty-seven-year-old Kelsey has four other children from previous relationships.