Although Haridāsa Ṭhākura was such a highly exalted Vaiṣṇava that he was addressed as Haridāsa Gosvāmī, he still did not like to disturb the common sense of the general populace. Haridāsa Ṭhākura was so exalted that he was addressed as ṭhākura and gosāñi, and these titles are offered to the most advanced Vaiṣṇavas. The spiritual master is generally called gosāñi, and ṭhākura is used to address the paramahaṁsas, those in the topmost rank of spirituality. Nonetheless, Haridāsa Ṭhākura did not want to go near the temple, although he was called there by Śrī Caitanya Mahāprabhu Himself. The Jagannātha temple still accepts only those Hindus who are in the varṇāśrama order. Other castes, especially those who are not Hindu, are not allowed to enter the temple. This is a long-standing regulation, and thus Haridāsa Ṭhākura, although certainly competent and qualified to enter the temple, did not want even to go near it. This is called Vaiṣṇava humility.
(Sri Caitanya Caritamrta---2:11:165---purport).

We should always remember that it was Śrī Caitanya Mahāprabhu Himself who wanted such a place, either for Himself or Haridāsa Ṭhākura. No one can suddenly attain the level of Haridāsa Ṭhākura and sit down in a solitary place to chant the Hare Kṛṣṇa mahā-mantra and remember the lotus feet of the Lord. Only an exalted person likeHaridāsa Ṭhākura or Śrī Caitanya Mahāprabhu, who is personally exhibiting the proper behavior for an ācārya, can engage in such a practice.
(Sri Caitanya Caritamrta---2:11:176---purport).

Even at Jagannātha Purī the Lord chanted the Hare Kṛṣṇa mahā-mantra in great meetings at the Jagannātha temple. The point is that one should not try to imitate Haridāsa Ṭhākura at the beginning of one's transcendental life. One must first become very mature in devotion and thus receive the approval of Śrī Caitanya Mahāprabhu. Only at such a time may one actually sit down peacefully in a solitary place to chant the Hare Kṛṣṇa mahā-mantra and remember the lotus feet of the Lord. The senses are very strong, and if a neophyte devotee imitates Haridāsa Ṭhākura, his enemies (kāma, krodha, lobha, moha, mada and mātsarya) will disturb and fatigue him. Instead of chanting the Hare Kṛṣṇa mahā-mantra, the neophyte will simply sleep soundly.
(Sri Caitanya Caritamrta---2:11:176---purport).

Śrī Caitanya Mahāprabhu once asked Haridāsa Ṭhākura how trees and plants could be delivered, and Haridāsa Ṭhākura replied that the loud chanting of the Hare Kṛṣṇa mahā-mantra would benefit not only trees and plants but insects and all other living beings. One should therefore not be disturbed by the loud chanting of Hare Kṛṣṇa, for it is beneficial not only to the chanter but to everyone who gets an opportunity to hear.
(Sri Caitanya Caritamrta---2:17:45---purport).

To live with devotees or to live in a temple means to associate with the śravaṇa-kīrtana process. Sometimes neophyte devotees think that they can continue the śravaṇa-kīrtana process without worshiping the Deity, but the execution of such śravaṇa-kīrtana is meant for highly developed devotees like Haridāsa Ṭhākura, who engaged in the śravaṇa-kīrtana process without worshiping the Deity. However, one should not falsely imitate Haridāsa Ṭhākura and abandon Deity worship just to try to engage in śravaṇa-kīrtana. This is not possible for neophyte devotees.
(Sri Caitanya Caritamrta---2:19;152---purport).