This is because there are only two ways to get your villagers. You can either do it by curing zombie villagers that spawn near your base or kidnapping villagers that already live in a village.