This is termed as Kraya Shuddhi. When we buy silk from commercial establishments, we pay for it and thereby we do not accrue sin. Hence, it is fine to buy silk in this manner to offer to Perumal, Bhagavathas as well as for ourselves and this will not accrue any taint.
However if one is very particular in not causing harm for other lives, it is better torefrain form wearing silk. Few of our elders had avoided wearing silk.
NOTES
Buying silk by paying money will nullify the taint due to kraya shuddhi. As regards the sin that may accrue to the manufacturer or the seller, this will get nullified through specific atonements prescribed in the shastras for people in such professions