I think when it comes to Diffculty, people forget that Gen 3 was a cryptic game. They hardly told you what to do or where to go. Hence the endless surfing to find where to dive or what routes to take.
Gen was really an adventure game, hardly any interruptions and a vast world to explorer even if it was by sea.
The games got way easier after that those in my opinion
I agree, but I don't think it was that cryptic. My kid self was able to traverse through the dive spots to find treasure, and the path to Sootopolis without much struggle. All I did was read the map and tried figuring out which dive spots led to which areas. It truly felt like an adventure game as you mentioned.
Sometimes, I wonder if frustration and lack of patience to use given resources to travel affects people's view of Gen III's difficulty.
And at this point, what can you do to make the AI better without cheating (aka knowing exactly what you'll switch into)? The only way to make a good pokemon single player battle is to have the trainers run a single strategy to be countered by the player and have 6 pokemon, AI changes would simply have to be either in pursuit of the strategy or hitting a weakness that the player would have the sense to counter in some way. That's it, and it would be wholly satisfying to the player, I assure you. Routes and caves would be designed to wear down the player and the gyms test individual strategies associated with type. I rather enjoyed the Sinnoh gyms because of this idea.
On the topic of AI, It took decades to get chess AI to a competent level and even more years to get them to beat grandmasters and chess had hundreds of years to get all the metagame knowledge it needed. Now Pokemon is definitely not as in depth as Chess but the metagames and mind aspects of out thinking the other person are similar aspects. It's just not a reasonable expectation from a single player turn based series to ask AI of that level to run prediction algorithms and be an enjoyable experience in single player. (Not regarding the fact that most "AI" in games is actually complex Scripting to begin with)
My point wasn't about making a cheating AI system, it was about Set not being that effective in terms of introducing challenge. It's merely an illusion of challenge that one gets until you realize you can still game the system so to speak.
That being said, there are many things you could do. First, it starts with not making stupid Pokemon choices for certain trainers. There are 721 available Pokemon, there's really no need to make rosters consisting of 6 Magikarps/etc. Additionally, create a strategy that you want the trainer to have, for instance, do you want the trainer to stall? Be Hyperoffensive? etc. And then you distribute movesets and so on. You can create an effective and challenging combat without resorting to cheating AI by exploring these fundamental gameplay issues. It's just a question of whether GameFreak wants to.