You're don't start out as female (I mean for the beginning you're some combination of X and Y chromosomes) but development is the same up to a point which I why men also have nipples.
They both develop from the same area.
They're are kind of analogous in a way. Keep in mind my information is from a college Human Reproduction class so I'm not an expert.
Hopefully this image is okay to post.