For the Series X, do you need the over priced memory cards to take advantage of the fast load times? I thought any other external drive would degrade the performance.
Yes, you do. The external cards are expansion SSDs, but I wouldn't bother using them for backlog titles from prior generations. The loading benefit is tangible, but varies from title to title and still isn't so impactful that it's a must-have. I am not entirely clear how that impacts swapping between games.
Many new releases are going to require they be installed on the SSD before you launch the application. You can put them into cold storage on an external HDD, meaning you can keep all the files locally without having to redownload them, if you prefer (which is what I do for certain games), but before you can play the title it will have to be moved to an SSD, whether internal or external.