I've thought about this for some time and finally come to my decision, which doesn't come up on that list.
The most difficult part of Japanese is the names of people and places. First of all its hard to remember names in your own language, if the names are unfamiliar in the first place, even more so.

But more importantly than this is that Japanese insist in writing their names in Kanji. And not just normal Kanji, they are often obscure or only used for names and have unpredictable readings.

Then there is that any given name like yosiko or keiko could be written in any number of ways, and there is no way to tell without asking or seeing the Kanji.

And so I have come to consider names to be the most difficult part of Japanese, followed by keigo and writing Kanji.