One should be extremely careful about associating with persons who are not respectful to the higher authorities. It is instructed, therefore, in the Vedic literature that one should always be free from the association of atheists and nondevotees and should try to associate with devotees, for by the association of a devotee one can be elevated to the platform of self-realization. This injunction is stressed in many places in Srīmad-Bhāgavatam; if one wants to be liberated from the clutches of material existence, then one has to associate with great souls, and if one wants to continue one's material existential life, then one may associate with persons who are materialistic. The materialistic way of life is based on sex life. Thus both becoming addicted to sex life and associating with persons who are addicted to sex life are condemned in the Vedic literature because such association will simply interfere with one's spiritual progress. However, association with great personalities, devotees who are great souls, will elevate one to the spiritual platform.
(Srimad Bhagavatam--------4:4:26-------purport).

There are two natures, spiritual and material. Materially, the inclination of sex enjoyment and eating meat—āmiṣa, āmiṣa means eating meat, flesh and fish, like that. That is called āmiṣa. Nonvegetarian means nirāmiṣa. So āmiṣa and mada and vyavāya. Vyavāya means sex. Loke vyavāya āmiṣa mada-sevā. Sex indulgence and eating meat, flesh, eggs, and drinking wine. Mada. Mada means liquor. Nityasya jantuḥ. Jantu. When one is in the material world he is called jantu. Jantu means animal. Although he's living entity, he's not called jīva soul. He's called jantu. Jantur dehopapattaye. Jantu. This material body is developing for the jantu, animal. Anyone who is devoid of spiritual knowledge, he's called jantu, or animal.
(Lecture on Bhagavad-Gita--------London, 1973).

Biphale means uselessly, and janama means birth, and goṅāinu means "I have passed." He's representing a common man, as every one of us is simply spoiling our life. They do not know that they are spoiling their life. They are thinking that "I've got very nice apartment, very nice car, very nice wife, very nice income, very nice social position." So many things. These are the material attractions. Gṛha-kṣetra-sutāpta-vittair. (child crying) (aside:) Stop it. Oh, that's all... The attraction, material attraction, is stated in the Śrīmad-Bhāgavatam in many ways. In one place it is very nicely summarized what is this material attraction. The basic principle of material attraction is sex. (child crying) Pumsaḥ striya mithuni-bhavam etam (S. B. 5:5:8). A man is hankering after another woman, and the woman is hankering after another man. This is the basic principle of material life. Puṁsaḥ striyā mithunī-bhāvam etam. Mithunī-bhāvam means sex. Tayor mitho hrdaya-granthim. And when they actually come into sexual life, each one of them becomes too much attracted. Hṛdaya-granthim. Granthi means very hard knot: "I cannot leave you." He says, "I cannot leave you. You are my life and soul," and she says, "You are my life and soul." For a few days. (laughs) And then again divorce. You see? But the beginning is there. Basic principle of material attraction is this sex life. This is general. Those who have organized the sex life in social convention in so many ways... The marriage is also another convention to give a very finishing nice touch than the animal. That's all. Just like sometimes it is said marriage is legalized prostitution. So for social convention the marriage is a license, but it is also based on the sex life. But for keep up social life, one has to accept some regulative principle. Therefore sex life like animals and sex life in marriage, there is difference. It is better. That is accepted in civilized world.
(Lecture-----Los Angeles, 1968).

All the rules and regulations apply equally to the householder and the sannyāsī, the member of the renounced order of life. The gṛhastha, however, is given permission by the spiritual master to indulge in sex during the period favorable for procreation.
It is sometimes misunderstood that a gṛhastha, a householder, is permitted to indulge in sex at any time. This is a wrong conception of gṛhastha life. In spiritual life, whether one is a gṛhastha, vānaprastha, sannyāsī or brahmacārī, everyone is under the control of the spiritual master. For brahmacārīs and sannyāsīs there are strong restrictions on sexual indulgence. Similarly, there are strong restrictions for gṛhasthas. Gṛhasthas should indulge in sex life only in accordance with the order of the guru. Therefore it is mentioned here that one must follow the orders of the spiritual master (guru-vṛttir vikalpena). When the spiritual master orders, the gṛhastha may accept sex life. This is confirmed in Bhagavad-gītā (7.11). Dharmāviruddho bhūteṣu kāmo 'smi: indulgence in sex life without disobedience to the religious rules and regulations constitutes a religious principle. The gṛhastha is allowed to indulge in sex life during the period favorable for procreation and in accordance with the spiritual master's order. If the spiritual master's orders allow a gṛhastha to engage in sex life at a particular time, then the gṛhastha may do so; otherwise, if the spiritual master orders against it, the gṛhastha should abstain. The gṛhastha must obtain permission from the spiritual master to observe the ritualistic ceremony of garbhādhāna-saṁskāra. Then he may approach his wife to beget children, otherwise not. A brāhmaṇa generally remains a brahmacārī throughout his entire life, but although some brāhmaṇas become gṛhasthas and indulge in sex life, they do so under the complete control of the spiritual master. The kṣatriya is allowed to marry more than one wife, but this also must be in accordance with the instructions of the spiritual master. It is not that because one is a gṛhastha he may marry as many times as he likes and indulge in sex life as he likes. This is not spiritual life. In spiritual life, one must conduct one's whole life under the guidance of the guru. Only one who executes his spiritual life under the direction of the spiritual master can achieve the mercy of Kṛṣṇa. Yasya prasādād bhagavat-prasādaḥ **. If one desires to advance in spiritual life but he acts whimsically, not following the orders of the spiritual master, he has no shelter. Yasyāprasādān na gatiḥ kuto 'pi. Without the spiritual master's order, even the gṛhastha should not indulge in sex life.

(Srimad Bhagavatam-------7:12:11-------translation and purport).

Regarding your question about Lord Siva and Thakura Haridasa, and their different response to sex agitation: This does not mean that Lord Siva has become degraded from the devotional platform. A devotee even if he is sometimes found that he is attracted by sex life, that does not mean he is degraded. That is by chance. Because Lord Siva is Grhastha, so by his past habits if he appeared to have been attracted by sex life, that does not mean he has been degraded from his position. You should never think of any devotee like that. There are many other stories also, but such apparent falling is without any influence to very very advanced devotees. A neophyte should always be careful. One should not discuss about such great devotees' apparent fall-down. Just like one should not discuss about the sun who evaporates urine from the earth; it is possible for the sun to do it, and still remain the sun, but for ordinary man if he lives in a filthy place he will be infected. So Lord Siva or Lord Brahma, they are highly elevated devotees, and we should not try to criticize about their behavior even though it appears against the rules. These things are very nicely explained in Srimad-Bhagavatam in course of discussion between Maharaja Pariksit and Sukadeva Goswami, in the 10th canto.

(Letter to Himavati------Hawaii, 1969).