For me, it's mainly particles and keigo. I suck at particles, and even native Japanese speakers find keigo difficult, so what hope do I have?

Grammar can be difficult too. The different ordering of sentences doesn't throw me, but knowing the nuances between similar grammars is what's truly confusing.