So. There's two different things you can do to "mod" your 3DS. You can get homebrew, and you can get custom firmware (CFW). You NEED homebrew to get CFW, so the first step to getting CFW is to install homebrew. If you just want to know if you can install CFW on your 3DS or not, skip the next paragraph (you'll have to do those steps if you do decide to get CFW, but hombrew is not the limiting factor here).
You can get homebrew on pretty much any system on any firmware. Most people use browserhax to get into hombrew. You go to a URL (with an internet connection) and homebrew gets loaded. As far as I'm aware (someone correct me if I'm wrong), this method does NOT work on 11.0 and up Old 3DS's, since there's no way on those to bypass Nintendo asking to update your 3DS (if you have a New 3DS, follow
this guide). There are still a
multitude of methods to getting to homebrew if browserhax doesn't work for you, but you might have to spend money to get a game. Once you're in the homebrew menu for the first time, I'd suggest installing menuhax so you can load homebrew by booting your system and pressing L (unless you want to use ctr-httpwn).
At this point, you should have a readily-accessible homebrew. Can you get CFW? If your system is...
Firmware 10.7 and down: Yes.
Firmware 11.0 and 11.1 (NOT 11.2): Yes, but you need to have Fieldrunners, Legends of Exidia, Guitar Rock Tour, or The Legend of Zelda: Four Swords (Anniversary Edition). You can't get these from the eshop anymore; you need to have had them already physically installed on your 3DS.
Firmware 11.0, 11.1, and 11.2, but you don't have any of the four games listed: It's possible, but you either need to hardmod (as in actually solder things inside your 3DS) or a second 3DS that already has CFW on it. The latter methods requires you to do a system transfer from the CFW 3DS to your not-CFW 3DS, and you can't system transfer from New 3DS to Old 3DS/2DS. But you can system transfer from New 3DS to New 3DS and Old 3DS/2DS to anything.
One note: There's a step in the CFW installation that apparently causes a lot of Old Ambassador 3DS's to brick. It's still unlikely that you will brick, but you should keep that in mind if you want to CFW one of those 3DS's.
There's also a sudohax floating around but I haven't looked into that.
For actual instructions on how to homebrew/CFW your 3DS, go to
https://3ds.guide/