While learning grammar structure, vocabulary, and kanji are okay for me (I'm Chinese, so I probably know more Kanji than a Japanese person... MEANING-wise that is), I find that listening to native speakers speak the language is very hard. The words just speed by so fast... and I can't pick up anything at all.