At least three houses belonging to suspected kidnappers were on Friday evening torched by Suneka residents.
They are suspected of kidnapping Pastor Esther Bochaberi Oyugi.
Residents blamed security officers for doing little to help find the pastor who was abducted from her home three weeks ago.
They blocked the busy Kisii-Isebania highway for the better part of the day, disrupting transport.
Tension gripped the area as police supervised efforts to trace the missing woman whom residents suspect may have been killed and buried in one of the suspects’ land.
Kisii South pastors had given the Government seven days to find their colleague.
The pastors and worshippers had threatened to engage in a peaceful protest.
Pastor Bochaberi is said to have been abducted from her house in Bokeire Village on the night of March 11, 2018. She was alone in the house.
Blood spots were found on her door step.
In a statement to journalists, the pastors, led by Chairman Lawrence Nyanuga Omambia, said their colleague was involved in a land tussle with her neighbour.
"Her misery started when people forcibly trespassed into her land and maliciously damaged her crops, trees, and even erected a structure on her parcel," said Rev Omambia.
The more than 50 pastors said no suspect has been arrested so far despite the fact that those who invaded her farm are well known and could be responsible for her abduction.
"We want justice for our colleague. We want her dead or alive. We are living in fear of being attacked," said Bishop Robert Atandi.
Mr Joseph Oyugi, the woman’s husband, said they have searched for her but not found her.
Mr Oyugi lives in Nairobi with their six children.
Kisii South police boss David Mburukua said it was unfortunate that residents took the law into their own hands instead of waiting for investigations to be concluded.
Five suspects, among them a chief, were arrested and charged at Kisii law courts in connection to the kidnapping.