You can only use "me" if you are saying "someone" spoke spanish to me. You don't always use a subject in spanish. So if in a conversation someone is talking about another person you can say "he/she spoke spanish to me." That would translate to "habló en español " but it's a past tense. othwrwise you wouldn't use it.
