[...] speaking Arabic is what makes someone an Arab, not being from Saudi Arabia.
Not really, the other Northern African countries (Morocco, Algeria, Tunisia and Libya) generally are not considered to be Arab, but Berber. Arabic also is a very wide-spread language among many Muslim countries, since it is used as the main language of the faith.
The term Arab has to do with the culture, which also was cemented historically and politically at this point, rather than based purely on language.
Arabs from Saudi Arabia are specifically referred to as Saudi Arabian or Saudis.
At least, to my knowledge.