For a balanced view on ethical stances, we need to start with some presuppositions:
1) There is no ethical perfection in this world;
2) Thus it is not perfection for which we are striving, it’s “direction.”
3) Whatever ethical stances we take, they should be grounded in our relationship with Krishna and Krishna Bhakti.
4) Not everyone’s “standards” are going to be the same; if Yudhiṣṭhira were in the chariot with Krishna, there would be a very different “Gītā” discussion between them than what took place between Arjuna and Krishna.
The fact is that Krishna accepted milk from the cows he so loved, and from cows who so loved Krishna. These cows were hardly factory farm cows, ultimately destined for slaughter after their milking usefulness. So, abstaining from the use of factory farm milk for some devotees is a way of honoring that relationship of love that is so powerfully and poignantly represented by Krishna accepting the milk from his cows.
Each devotee has to decide carefully what ethical standards they should embrace. And once each devotee does, they can go in a direction that incrementally strives for higher standards, as is practical. Do not compare yourself with anyone; rather, dig deep within yourself to understand where you are today, and how you may increase your standards gradually tomorrow.