Lexicon, vocabulary, 語彙! And not because they are so foreign (although that has something to do with it), I just hate learning vocabulary (I was the same with Spanish). I like learning grammar, conjugations, inflections, etc., etc., more generally I like learning how things work.

What's easy is the lack of irregular verbs and straightforwardness of the grammar, once you get used to the SOV structure. One thing about learning Japanese: it makes learning languages like Spanish, Italian, etc. a hell of a lot easier (save the irregularities in conjugations), and it really makes learning Chinese grammar easy!

Edit:
I almost forgot about probably the most difficult thing for any Westerner to grasp in learning Japanese: the topic/negation marker "wa/ha" and how it is used. The idea of marking anything in English is already foreign enough, but marking topics as opposed to focuses or marking negated words/phrases is even more out there. I've read a lot about the topic (subject? lol), and I still can't always use it correctly.