Not for me, if style or fashion is the issue in the pastor’s mind. The text must be understood as prohibiting drag, or the erotic blurring of gender differences. Ancient Israel was told not to permit the parading of drag queens in society.
On the other hand, the pastor is right in warning against some modern fashions. Many men believe, as he evidently does, that the pants young women wear are deliberately provocative. Of course, dresses and skirts and mini-skirts, and everything else young women wear or don’t wear, can be so designed as to catch a man’s eye and stir his manly passions. Nobody believes that women whose clothing is not modest are truly ignorant of the achieved effect (see 1 Tim. 2:9–10).
Back to the question of whether it’s a sin to wear pants. The answer is, it depends on the modesty of the pants.