Righteous living by a church’s members does not ensure numerical growth. Our Lord addressed His disciples as a “little flock” (Luke 12:32). Of course, that was before Pentecost when thousands poured into the new church in Jerusalem. Even so, most if not all churches in New Testament times seem to have been small groups of people.
Both large and small churches often include people whose lives are disgraceful. Why most churches remain numerically stagnant whereas a few explode numerically is a question ventilated in many “church-growth” books. You may find helpful tips in such books. Most importantly, though, remember that there are different kinds of growth, including coveted growth in the knowledge of God and resultant spiritual maturity.
You need not feel guilty if your church is small—unless of course, you are small because you are not friendly or are lazy or locked into sectarian traditions that have little to offer ordinary believers in search of a church home for themselves and their children.