It seems 4 is the limit for Rob Lowe, who's just announced that he's leaving Brothers & Sisters at the end of this season.
Citing some crap about being "underutilized", Lowe is currently in talks with ABC for his own show, on which he will have more of a central role.
Ensemble work is clearly not Lowe's forte since he also departed The West Wing at the end of its 4th season because he was not playing a large enough role.
The character of Robert is expected to be written out rather than killed off so that Lowe can continue to guest star, which means there must be trouble ahead for Kitty and Robert, the couple that's only just renewed their wedding vows and declared themselves unshakable. Word on the street is that this means the return of my beloved (my tv award nominee) Matt Letscher as Kitty's crush, which I'm sure will soften the blow.
Nevertheless, Lowe's departure is sure to leave a hole in the Walker family dynamic. Luckily, the show has such a strong ensemble that the other (dare I say it, superior) in-laws can step up to fill the screen time in his place (here's lookin' at you Scotty!) and Robert was a late character addition so is not actually integral to the balance of the series.
I doubt Lowe's new project will make it far but if it does pop up somewhere down the line in a pilot season I'll be curious to see if he can make it, someday, to a season 5.