I think the breaking changes between Swift versions has been the main issue. However, they've finally settled on ABI stability with Swift 5, so things may turn around. Swift is still in a lot of Top 10 language lists, and also ranks highly on "next language I intended to learn" lists. The association with Apple is tricky, of course, but mostly because people love to hate Apple, not because of anything in the development of the language itself (apart from the slow path to ABI stability, of course).

I think the bigger problem is one of perception (shaped, in part, by articles like this). Swift is not so much the "new kid" anymore, and it has had the version-related growing pains you mention. The big problem there is that people who suffered those pains had to decide on other languages in the meantime, and once they started building stuff they weren't incentivized to switch. A stronger push on the cross-platform capabilities of the language would help a lot, but there's no real incentive for Apple to do that, so it would have to come from elsewhere.