Not at all. If you created a detailed backstory for him and a host of other characters for him to interact with, that version of him is as much your character as he is a fandom character. You can easily make alterations to him, re-writing or excising the parts that identify him as a fandom character, and then complete your story as an original piece, if you so choose.
And there’s no real shame in starting an idea from a fannish place. Fanfiction in general has existed for a lot longer than people think. Virgil’s Aeneid (written from 29 to 19 BCE) is basically a POV fic of a minor character in Homer’s Illiad (circa 8th century BCE).
Also, villains are awesome! Of course, they’re gonna be “problematic”! They’re usually the source of the hero’s issues and give everybody nothing but trouble!
Long story short: Do your own thing, honey, and let me know how it goes. From what you told me, it sounds like it’ll be awesome. 🙂