Even games with better AI - Halo, FEAR, Half Life to name a few FPS examples - are still fairly simple for the most part, with very learn-able patterns to their behavior.
If you look at the Bungie presentations, you'll see that these simple patterns were deliberately dialled back or were replacing more complex systems. "Better" AI was possible 15 years ago, but it just didn't playtest well. The "smarter" enemies were no fun to fight and players would feel like they were interacting with random number generator.
The key factor is transparency. If there's a complex system driving the behaviour of a character, but that system is opaque to the player, then it's no fun to interact with. When a character makes a decision, it has to be clear to the player why that decision was made.
Consider these two scenarios:
A) When you kill the Elite, the Grunts get a reduction to their Morale stat, which is also dependent on ammo remaining, situational advantage, and current health. If it falls below a certain level, trigger the "panic" behaviour.
B) When you kill the Elite, the Grunts panic.
With A, you have no idea ahead of time whether it's worth killing the Elite. Sometimes it makes the Grunts panic, sometimes it doesn't. The Grunts aren't running around with Morale meters over their heads, so it feels random.
With B, you have a predictable behaviour that you can take advantage of. When combined with the other simple systems (grenades make Elites dive. Melee attacks from behind are one-hit kills etc.) there's enough complexity to be fun, but not so much complexity that it's indistinguishable from randomness.