Evaluating the Revolution at the Grassroots
Chairman, Quality Council of India
IN RESPONSE TO the Prime Minister’s push for an Open Defection Free (ODF) India, the largest behaviour change programme in the world was launched in 2014, under the aegis of the Ministry of Drinking Water and Sanitation (MDWS): the Swachh Bharat Mission (SBM). As the movement became stronger, it looked as though India would make significant progress in a short time. In just about five years, access to individual household toilets has increased from nearly 38 per cent in 2014 to 99 per cent in 2019.
But a common question, and sometimes critique, has been – ‘Is this really happening on the ground? Are toilets actually being built and then used by people?’ In 2016, MDWS asked the Quality Council of India (QCI) to create a system for measuring this in the top 75 districts of India, and then across the country in 2017, in a statistically significant and nationally representative way. The idea on our end was to be 100 per cent efficient and scan every state and district in India within 90 days!
The QCI is an autonomous organization, and its mandate is improving quality in all spheres of life in India. Improving Government services is an important part of the mandate, and QCI has done several large-scale studies to measure the on-ground progress and impact of Government programmes.
MDWS has developed an online monitoring system for SBM-G, an Integrated Management Information System (IMIS). Household-level data with respect to sanitation facilities of all gram panchayats in the country are fed into the IMIS by states on the basis of the Baseline Survey 2012–13, and updated as need be. Each toilet built is geo-tagged and uploaded through the SBM geo-tagging app to a central database maintained by MDWS.
Likewise, MDWS has an annual third-party monitoring mechanism that tracks the progress of the scheme on a real-time basis which helps in course correction. In addition to QCI, independent verification agencies like the National Sample Survey Organization (NSSO) and National Annual Rural Sanitation Survey (NARSS) have audited the claims mentioned in the IMIS from time to time through on-ground surveys of households, schools, public health centres, and various other community congressional places.
The first such survey to check the access and usage of the toilets was conducted by NSSO in 26 states and Union Territories in 2015 followed by QCI in 2016 and 2017. It is a matter of pride for our organization to have been a part of Swachh Survekshan Grameen 2016 and Swachh Survekshan Grameen 2017.
Swachh Survekshan Grameen 2016 was conducted in the 75 best-performing districts (53 in the Plains and 22 in North Eastern and Special Category) which were identified on the basis of IMIS data available with the Ministry. The household survey covering 70,000 households was carried out in these districts from 21 May 2016 to 20 June 2016. A team of 150 qualified assessors was deployed to collect the required data with the help of a mobile application. The villages were selected based on proper sampling wherein each district was divided into four zones and then villages were randomly chosen from each zone. A minimum of 1,050 households were selected from each district in the Plains and a minimum of 350 households were selected from each district in the North East and Special Category. Mandi, Himachal Pradesh, and Sindhudurg, Maharashtra, respectively, were the frontrunners in the North East and Special and the Plains categories.
In Swachh Survekshan Grameen 2017, the survey was conducted in 4,626 villages from May to June 2017. The villages were selected based on the Probability Proportional to Size technique, subject to a minimum of 20 villages in each district and 30 households in each village chosen through Simple Random Sampling. A team of 300 qualified assessors was deployed. The study was divided into two categories: Household Survey and Direct Observation. The Household Survey captured parameters like Access to Toilet, Usage of the Toilet and Solid and Liquid Waste Management at the Household Level. Direct Observation collected information regarding access to toilet and cleanliness at common places in a village, i.e. school, anganwadi and public health centre. To validate the sampling methodology, QCI approached experts in the field from various organizations including the NSSO, Indian Statistical Institute, and various universities in India and the USA.
The overall results indicated that rural sanitation coverage had increased from 38.70 per cent in 2014 to 62.45 per cent in May–June 2017, as compared to 63.73 per cent according to the SBM-G IMIS for the same period – a testament to its robustness. The best performing states were Haryana, Kerala, Himachal Pradesh, Manipur, Tripura, Sikkim and Nagaland.
Behaviour change takes time and our surveys made for some interesting findings, beyond access and usage of safe sanitation practices. One common phenomenon was what we now call ‘Respondent bias’. Given that it is now common knowledge in rural India that the Government supports economically weaker households with a financial incentive of Rs 12,000 for building and using their toilets, our surveyors found respondents taking that into account during the survey. For instance, in Uttar Pradesh, some households would deny having toilets when our assessors visited them, hoping to become eligible for another round of incentives. This became an operational challenge for us since it affected the sampling across many villages. To overcome such challenges, we went for resampling of the households that were to be visited by an assessor.
Our execution strategy involved ‘three levels of defense’ to maintain a robust trail of on-ground data, which are as follows:
The design parameters were such that we were receiving rapid responses while we were performing a quality check of real-time data. The hit rate of the number of villages went up to 200 per day covering 6,000 households. The operational assessment planning entailed a lot of brainstorming to progressively collect the data and maintain its sanctity at the same time. We initiated a pilot to understand the nitty-gritty of the subject on the ground before expanding the scope of the survey. At some stages, we had to check the geographical spread of responses so that statistically significant conclusions could be drawn from the sample.
One of the challenges that we faced during the assessment was collecting data in difficult terrain where access to public transport was limited. In some villages in Arunachal Pradesh the assessors had to cross fluttering ropeway bridges to carry out the survey! Poor network connectivity in certain parts of the country was another big issue as it made it difficult for the assessors to upload their data in real time. The vagaries of the weather in different parts of the country made our task even more challenging. In Siddipet, Telangana, our assessor fainted in the jungle due to extreme heat. His mobile phone was stolen and we could not establish any contact with him for a few hours. Thankfully some local people took him to a nearby clinic and he recovered there. In another instance, in the Andaman and Nicobar Islands, torrential rainfall made it nearly impossible to conduct the survey. Even after the survey had been conducted, the telecom networks were jammed and the data had to be sent on CDs all the way to Delhi.
Through our survey, we found that 91.29 per cent of the households with access to toilets were also using them regularly, indicating a paradigm shift in the attitude of rural households towards open defecation. One cannot help but admire the sensational change that SBM has brought about even in these remotest of villages.
Of course, there has been criticism from some NGOs and media, often based on anecdotal experiences in a few villages. But the data speaks for itself. The surveys conducted by QCI looked at over 1,40,000 households and are representative of what is happening in every district, and hence a validation of the success of Swachh Bharat to date.
The impact on lives also speaks for itself. The QCI teams found many powerful examples of communities’ lives being transformed as a result of the progress made by SBM. In East Sikkim, one of the first districts in the country to be declared ODF, the attendance of girl students in schools improved after separate toilets were constructed for them, and many girls who had previously dropped out returned to their classes. In West Bengal, the unsafe waste disposal from kachcha toilets was causing many health issues. But once pucca toilets were built at homes under SBM, usage went up as there was no longer any foul odour around the toilet. In Arunachal Pradesh, we came across moving stories of people saying that they were ‘not compelled to defecate with pigs any more’. Before SBM, households were compelled to defecate in the open which attracted pigs in that area. An elder from the family had to always accompany children when they went to defecate in order to ensure their safety and security. Once safe toilets were constructed under SBM, life became safer for the people, especially kids.
The above is just a taste. Our assessors captured hundreds of such heart-warming stories of how toilets have improved the lives of individuals and families, across communities on video … but that’s for another day, and perhaps another essay.