Well, it’s not elitest 😉
But aside from that, there are a few serious disadvantages. And these are ones you already know about: there’s lots of (potentially-vulnerable) PHP running, which is a security risk and also makes WordPress … slow.
Also, WordPress makes microblogging, or “notes” in IndieWeb parlance, not very easy. I want to publish my own microblog on my site and publish it elsewhere, but this will take futher investigation.
WordPress, also, has a reputation. It’s a bit like Walmart (or Asda in the UK). It’s a great, hulking CMS that everyone knows. It’s everywhere. Everyone uses it. Which means there’s less that’s “special” about it. And that’s a shame, because for all of that it’s really quite brilliant.