Hmm english...
1. Idiomatic weirdness.
Many common verbs used in phrases easily decoded by the native speaker but whose meaning can't be easily discerned from the meaning of the constituent words.
2. Nightmarish orthography.
For various reasons, (borrowed words never respelled, pronunciation changes never reflected in writing, more vowel sounds than letters in latin alphabet) english spelling is extremely incosistent.
The 'ough' in the words below are all said differently and you could probably come up with more.
Though
Through
Thorough
Rough
Cough
In contrast, spanish has a consistent orthography. Probably one of the most consistent in common use. If you know spanish but encounter an unfamilar word while reading, your chance of correctly saying it from its written representation is very high.
3. Some common english sounds are somewhat uncommon in many other languages.
The two different sounds represented by 'th' (one in thing, another in those) are not common in many widely spoken languages, even those more closely related to English such as German (thing: Dinge, with: mit). In fact, even some dialects of enlgish and english derived languages don't have it (replaced by 'd' or 't').
However in standard english it appears in a disconcerting number of very common words: eg the, that, those, them, with, theirs, thank, three, sixths.
There are other sounds that give problems depending on what the leaners native tongue. Some might find 'l' and 'r' difficult to differentiate or 'b' and 'v'.
4. Profusion of homophones.
There are tons of english words that sound the same (sometimes are written differently) but mean different things. Makes it great for puns and double-entendres but can certainly caause unintended confusion, even among native speakers.
5. Grammar.
The grammatical features of the english language aren't as numerous as some other languages (minimal grammatical gender, words don't inflect as much) but there are a singificant number of special cases and forms that you just have to know. For example, negation is strange compared to many other languags. The common modern english way is to say 'I do not dance', (introducing another verb) rather than the equivalent of 'I dance not' or 'I not dance' common in many other languages (the negator simply precedes or follows the verb).
I'd say the difficulty of learning english depends on the language you're coming from and maybe how many languages you've learned previously. Nonetheless, I think the argument is mostly moot. The language of the most dominant nation tends to be the most widely used. There was a time when it was Greek, Latin or French.
And this is the advantage of english, it was the language of the British empire (Britain itself is just a few islands like Japan). As a result it was already far flung and dispersed spoken from the Americas through Africa and Asia.