When im reading a book which is full of references, I must read

Yes. And you have to read the books those books reference before that, and the books those books reference before that. Not just explicit references either, you have to read the writings that inspired the authors, as well as experiencing their experiences which did the same.

Colin Wilson is actually an incredible resource for this sort of thing. The Outsider is a fantastic book. He wrote a couple other in the same "cycle":
https://www.amazon.com/Colin-Wilsons-Outsider-Cycle-students-ebook/dp/B007UQ478I

...for which he was panned, badly, by the British intelligentsia, after briefly being a cause celebre due to The Outsider. Also worth reading are his The Occult and Beyond the Occult, if you're into that sort of thing.

But most likely you'll only want to read The Outsider, unless you specifically enjoy Wilson's thought or esoterica in general.

On what you're asking: It's a bit of both. You're basically going to go through several years where you're drowning in names without "anchors" that make them meaningful, until one day, you'll notice that this tendency has reversed dramatically, and you recognise more than you don't recognise. In order for that to happen, you just have to tolerate a bit of tedious slog for those first few years, using the Internet to look up various figures or at least making mental notes of them. I promise you, they eventually link up and form webs of relevance that are smaller and less intimidating than they seem at first.

But you definitely don't have to READ all those figures. Wilson is a great example of a trend among intellectuals, of having the "gist" of another thinker and responding to that. The knowledge of Coleridge necessary to understand a Wilsonian reference to Coleridge is probably about three or four talking-points that you will eventually have in your mind by reading things like Wilson for long enough, even if you never read Coleridge. Wilson's understanding of French philosophy, and many things actually, is not deep - he was an intellectual adventurer with his own vision and he travelled through many different scientific, philosophical, literary, esoteric circles picking up ideas which he liked. That's why he has been a gateway and a touchstone to so many of those things, for so many people. But he certainly didn't have a systematic appreciation of them.

Short answer: Nope.