The living entities are not without spiritual senses; every living being in his original, spiritual form has all the senses, which are now materially manifested, being covered by the material body and mind. The activities of the material senses are perverted reflections of the activities of the original, spiritual senses. In his diseased condition, the spirit soul engages in material activities under the material covering. Real sense enjoyment is possible only when the disease of materialism is removed. In our pure spiritual form, free from all material contamination, real enjoyment of the senses is possible. A patient must regain his health before he can truly enjoy sense pleasure again. Thus the aim of human life should not be to enjoy perverted sense enjoyment but to cure the material disease.
(Sri Isopanishad).

These so-called yogīs are actually enjoyers of the world garbed as yogīs. Bhagavad-gītā, however, recommends that if one wants to enjoy life, then it cannot be with these gross senses. A patient is advised by the experienced physician to refrain from ordinary enjoyment while in the diseased condition. A diseased person cannot enjoy anything; he has to restrain his enjoyment in order to get rid of the disease. Similarly, our material condition is a diseased condition. If one wants to enjoy real sense enjoyment, then one must get free of the entanglement of material existence. In spiritual life we can enjoy sense enjoyment which has no end. The difference between material and spiritual enjoyment is that material enjoyment is limited. Even if a man engages in material sex enjoyment, he cannot enjoy it for long. But when the sex enjoyment is given up, then one can enter spiritual life, which is unending. In the Bhāgavatam (5.5.1) it is stated that brahma-saukhya, spiritual happiness, is ananta, unending. Foolish creatures are enamored by the beauty of matter and think that the enjoyment it offers is real, but actually that is not real enjoyment.
(Srimad Bhagavatam-------3:20:35-------purport).