Yes, it will come under the prescribed 32 misdeeds (apacarams) when we do not observe purity during the thriumanjanam of Salagrama deities and Acarya padukai-s. As we must touch the deity to perform thirumanjanam, we should maintain physical cleanliness. It is good to know that you are striving to follow the prescribed observance and please maintain cleanliness to your utmost ability.