When did you learn to distinguish accents?

I grew up amongst Spanish speakers yet I still can't tell one accent from the other. I'm wondering if it's just too much for the average Spanish-as-a-second-language speaker.