We have been trying to pare down the show recently to very rarely be more than three people. (Special cases like someone being in from out of town for GDC or whatever are exceptions.) This isn't detail I was planning on sharing, but whatever, it's not really that big a deal: Idle Thumbs lost a LOT of listeners when the cast of the show grew too big and we turned into sort of a weekly game of musical chairs. I am strongly convinced that this is not because of the presence or lack of any particular combination of people, but rather because the changing combination of people made the show unpredictable and inconsistent from week to week. Not necessarily inconsistent in terms of end quality of each individual episode, but inconsistent in terms of tone and listener expectation. I don't imagine this hurt the show much for people here on the forums, since you folks know us all really well relatively speaking, but I think the larger listening audience, especially people who haven't been listening as long, would just kind of lose interest over time when it seemed like they didn't know which version of the show they were going to get week by week.
We also think it's conversationally manageable with three people, and gets really tough with four (and especially five). We have such an unstructured style, with lots of interruptions and cross-talk, that when you go above three, things get dicey--again, probably not as big a deal for the longtime hardcore listeners, but I think often very frustrating and difficult to parse for many. Once you have four people, you're in the zone when you can have two different parallel small conversations, and that is the absolute worst. It also kind of sucked for me to have to deal with scheduling a different group of people each week, where people who weren't me and Jake could basically decide arbitrarily whether they were going to be on or not; that's just inevitably how free-time projects end up if you don't have a really strong requirement that a specific group of people always show up. So it has made things much easier to just assume "Okay, this group of three is going to be on every week, and if someone absolutely cannot make it, we'll figure out a sub." I am completely certain that Sean will still end up on episodes. In terms of the default week to week configuration it has just made it a way less hectic show to have a consistent group.