I played X/Y and found it pretty but lacking in challenge
I played Pocket Morteys and it scratched my itch, until the fucking thing crashed and ate my Morteys.
Now i want a real Pokey Mans game, that is recent and an actual challenge.
Digimon if you got a PS4 or Vita for Cyber Sleuths...or the Japanese 3ds if you prefer raising one individual at a time
Dragon Quest Monsters
Medabots (if you got a Japanese 3DS)
Moco Moco Friends
Fossil Fighters 3DS
Yo-kai Watch, notably 2
Final Fantasy Tactics
Jade Cocoon 2 if you're into old stuff, along with Monster Rancher.
There's probably still more out there
I also forgot Pocket Mortys on the list too, but then I remembered OP actually played the game.
So after this is likely Monster Strike on the 3DS (japan) and Ni no Kuni for PS3.
While it is more challenging than Pokemon, it is far more like a Pokemon clone than the others plotwise and mechanic wise. It also suffers from recolors adore, even topping YW's recolor of that fucking cat.
It's niche as fuck. Since there's no real advertising, not many people heard of it.
Unless you can search the name in Japanese, gameplay videos are hard to find