I think the hardest thing is counters!!!! My friend atsuko and i saw a plane in the sky the other day, and I said, "Do you use dai to count planes, like you do to count cars and machines?" At first she said yes. Then a second later she said, "Wait, actually you use 'ki' as in hikouKI." Argh! It never ends! I related this story to another friend Naomi-chan, and she said "I'm Japanese, but I don't get counters! I just use 'ko' for everything! Movies? Ko. Airplanes? Ko. It's just easier that way."
Not to mention that you have several ways to pronouce each number (ikkai, hitotsu, ichi-en are all from 1) and there's no logic to which one you're supposed to use.
Of course, Kanji is hard. But after I got past a certain point learning kanji (after I could read 600 or 700) i realized that they make the language EASIER not harder, because if you understand new vocab via kanji you already know from other words, theya re easier to remember, and also, if you understand the structure and logic of kanji, a lot of things about japanese that seem arbitrary start to make sense.
that's my ni-en. :)