AN ASSESSMENT OF THE POLICY AND REGULATORY OUTCOME BY THE TELECOM SERVICES USERS: THE EMERGING ECONOMY STUDY

How to cite this paper: Mishra, P., Singh, N. P., & Farooq, A. (2022). An assessment of the policy and regulatory outcome by the telecom services users: The Outcome-based policy evaluation is an established practice in the distributive and redistributive public policies. Such practices are not evident for competitive regulatory policies of telecom, especially in India. This study bridges this research gap by carrying out an outcome-based evaluation of telecom policy and highlighting the importance of such evaluation. Using the methodological pluralism model from Schalock (2002), the outcome of India’s telecom policies was evaluated. Outcome measures from the vision statement of telecom policy were appraised by telecom users by responding to a structured questionnaire-based survey. Factor analysis confirmed that our survey instrument measured the identified policy outcomes. Regression analysis confirmed that users’ appraisal was based on their experiences of telecom services. Against five policy outcome measures, the survey respondents agreed on the achievement of affordability of services : 68.9% of the respondents found telecom services not secure; 74.7% of the survey respondents indicated an issue with quality; 55.6% of the respondents did not agree that the services are available anytime, anywhere. Outcome measures like telephone density (teledensity) as adopted by Telecom Regulatory Authority of India (TRAI) and Department of Telecommunications (DoT) are not the true representative of policy outcome. A multistakeholder policy evaluation will reveal the actual policy outcomes. International Telecommunications Union (ITU) should establish a standardized framework for outcome-based policy evaluation to address such issues.


INTRODUCTION
Public policy analysis and policy outcome evaluation are well-established practices in distributive and redistributive public policies (Oliver & Parolin, 2018).

Telecommunications
policies fall under a competitive regulatory process due to licensing and spectrum management functions (Birkland, 2019). Most countries have telecom sector regulators and regulatory impact analysis (RIA) is more prevalent.
As Organisation for Economic Co-operation and Development (Organisation for Economic Co-operation and Development OECD, 2014) noted, of all the OECD members only two countries did an ex-post analysis of policy and regulation in 2014. International Telecommunications Union (ITU) posited that RIA cannot substitute the policy analysis function and should be included within the policy analysis cycle for holistic benefit (ITU, 2014).
India is the world's second most populous country with a democratic government and any public policy affects a considerable number of households. In telecom, the government arm of the Department of Telecommunications (DoT) is vested with policy-making, licensing, and spectrum management function (an ex-ante aspect of policy-making is in DoT's control). The Telecom Regulatory Authority of India (TRAI) is the regulator that ensures a level playing field, compliance to license conditions, quality of service (QoS), interconnection, and tariff (ex-post activities related to policy). In such a scenario, policies formed by DoT have a more far-reaching impact than the ex-post actions of TRAI. This situation mandates that in addition to RIA, a public policy outcome evaluation should be carried out to ensure that policy-making and its outcome is aligned to the public good.
With 1189.15 million subscribers and a teledensity of 86.89%, India is the second-largest telecom market in the world (India Brand Equity Foundation [IBEF], 2021). The telecom sector has been a success story for India. In the first decade of the 21st century, the number of subscribers grew by 33% annually (Competition Commission of India, [CCI], 2021). It continues to grow rapidly and by the year 2025, out of the 200 million new subscribers added in Asia-Pacific, over half of them will be from India (Groupe Speciale Mobile Association [GSMA], 2021). The sector contributes 6.5% to India's GDP and attracted $29.8 billion of foreign direct investments (FDI) in the last two decades. It directly employs 2.2 million people and is among the highest contributors to the exchequer (Cellular Operators Association of India [COAI], 2021). In the National Telecom Policy 2012 (NTP-2012), the Indian policy makers stated their -vision‖ and -mission‖ for the Indian telecommunication sector (DoT, 2012). The policy makers strived -to provide secure, reliable, affordable and high-quality converged telecommunication services anytime, anywhere for an accelerated inclusive socioeconomic development‖ (DoT, 2012, p. 4). The operators, regulators, and policy makers have contributed significantly to the success of this vision.
However, in the last decade, India witnessed major issues like 2G license cancellation, adjusted gross revenue (AGR) dispute, Unified Access Service License (UASL), and Wireless in Local Loop with Limited Mobility (WLL-LM) service. It was estimated that $12 billion was stuck in disputes and litigation involving license fees, spectrum usage charges, and one-time spectrum fees (-DoT seeks to reduce litigations‖, 2018). In 2019, the industry was under a debt of $105 billion (Parbat, 2018). The average revenue per user (ARPU) fell significantly due to the tariff war that intensified post-entry of Reliance Jio (RJio) in 2016. ARPU (access services) dropped from ~$3 to ~$2.4 between September 2016 and December 2020 (TRAI, 2016; TRAI, 2020). Bharti Airtel posted its first quarterly loss in 14 years with losses standing at $433 million for Q1'19 (-Q1 results: Bharti Airtel's biggest loss‖, 2019). The loss for Vodafone Idea Limited stood at $732 million in Q2' 19. With an estimated loss of $2.13 billion for 2019 fiscal year Bharat Sanchar Nigam Limited (BSNL) struggled to pay salaries to the employees (-BSNL FY19 loss‖, 2019).
The Cellular Operator Association of India (COAI) has often highlighted the requirement of policy reforms in the taxation regime, the definition of adjusted gross revenue, and spectrum management (Press Trust of India [PTI], 2021; Singh, 2021). The telecom operators Vodafone Idea and Bharti Airtel alleged that frequent policy changes and unreasonable demand from the Indian government have damaged the operators (Sanjai & Saxena, 2019;Pandey, 2019). Users, on the other hand, suffer from poor call quality, frequent call drops, and slow data speeds (Singh, 2020;Tech Desk, 2019). In 2019, more than half of the surveyed users experienced issues in call connection and had frequent call disruptions (-53% citizens facing severe call‖, 2019). The concern for the security of mobile networks was highlighted by the honorable prime minister of India himself (Bhardwaj, 2020).
Even after twenty-five years of structured reforms, huge growth, a mature regulatory framework, and a highly competitive market, such a situation questions the outcome of telecom policies. If operators are raising grievances and customers complain about attributes of telecom services, the attainment of policy vision is questionable. There is hence a pressing need to perform a policy outcome analysis related to telecom. Such assessment will not only help in highlighting the improvement areas but will also trigger research in the area of public policy making. Using the methodological pluralism model and policy evaluation model from Schalock (2002) the authors carried out an outcome-based evaluation of telecom policy in India. The authors adopted the individual appraisal mode of evaluation using a structured questionnaire survey with a focus on the following objectives: 1) carry out an appraisal by the end-user of attributes of telecom services mentioned in the vision and mission statement for telecom in India; 2) find out the appraisal of the end-users about the telecom policy outcome in terms of achievement of the vision for telecom services by the Indian policy makers; and 3) analyze if there is a relationship between the end user's appraisal of attributes of telecom services and their verdict on achievement of the vision of telecom by the Indian policy makers.
The authors could not find literature where the outcome-based evaluation of the public policy related to telecom was carried out in India. By carrying out the individual appraisal-based outcome evaluation, this study provides otherwise unavailable visibility on policy outcomes to the telecom operators, regulators, policy makers, and industry bodies. The study highlights the gaps and improvement areas in the domain of telecom policy, which the Indian policymakers must address. Additionally, the study determines if users' perceptions of telecom services drive their judgement on policy outcomes. This novel study is likely to trigger further research in the area of telecom policy making and public policy evaluation in India and other emerging markets.
This paper provides a background and key literary work on the subject matter of this study in Section 2.
Section 3 details the research methodology. This is followed by Section 4 which presents the results and findings of the study. The discussion on the findings is covered in Section 5. The paper concludes by presenting the conclusion in Section 6.

BACKGROUND AND LITERATURE REVIEW
The telecom policies announced by DoT are the primary vehicles for shaping the Indian telecommunications industry. The first telecom policy (National Telecom Policy 1994 (NTP-1994)) was announced in 1994 and aimed at ensuring the availability of world-class telecom services. The New Telecom Policy (NTP-1999) introduced in 1999 emphasized affordable and world-class telecom services. It highlighted the need for telecom services in remote, rural, hilly, and tribal areas. The next telecom policy NTP-2012 stated a formal -vision‖ and -mission‖ for the Indian telecommunication sector as -to provide secure, reliable, affordable and high-quality converged telecommunication services anytime, anywhere for an accelerated inclusive socio-economic development‖ (DoT, 2012, p. 4). The vision statement incorporated objectives from NTP-1994 and NTP-1999 and added the requirement of secure and reliable telecom services.
The National Digital Communications Policy 2018 (NDPC-2018) was approved by the Indian Cabinet on 26th September 2018. The policy makers moved from the objective of availability of telecom services to the need of availability of broadband for all. The vision was to fulfill the information and communication needs through the establishment of an ubiquitous resilient, secure, accessible, and affordable digital communications infrastructure and services.

Assessing telecom policy outcomes
Policy analysis and policy outcome analysis is well-established practice in social science research. However, as Rossi, Lipsey, and Henry (2019) point out not every policy irrespective of its good intent will lead to a better outcome. A systematic policy evaluation is hence necessary to assess the effectiveness of social programs and identify the factors that drive their effectiveness. Policy evaluation can help course correct, create better programs and replace an otherwise ineffective policy with a better one.
Policy evaluation can be ex-ante or ex-post. The ex-ante methods typically are the cost-benefit analysis, cost-efficiency analysis, social marginal cost of funds analysis, and data envelopment analysis (Shah, 2020). There are multiple-criteria evaluation (MCE) and multiple-objectives evaluation (MOE) that can be ex-post or ex-ante. Under MCE and MOE, formative evaluation, which consists of diagnostic and process evaluation related to policy cycle, is ex-ante. Whereas summative evaluation consists of outcome and impact assessment (Shah, 2020). A variant of summative evaluation is conclusive outcome evaluation, which provides an overall judgment on a policy program (Chen, 2015).
Outcome evaluations are experimental or non-experimental. The widely used non-experimental methods are difference-in-differences, fixed effects regression, reflexive control designs, matched control designs, regression discontinuity design, and instrumental variables approach (Shah, 2020). Evaluation models like goal-attainment model, sideeffects model, relevance model, client-oriented model, stakeholder model, and collegial models are also identified in the literature (Vedung, 2013). Retrospective process evaluation can be cross-sectional and longitudinal. Longitudinal studies evaluate changes in the outcomes over a period of time, whereas cross-sectional studies, evaluate outcomes at one point in time (Dunn, 2018). Dunn (2018) highlights the importance of multidimensional evaluation to consider the perspective of all stakeholders affected by a certain policy. The multiattribute utility analysis model proposed by Dunn (2018) is based on the subjective judgments from various stakeholders about policy outcomes. On a similar theme, a multidimensional methodological pluralism model is proposed by Schalock (2002). The methodological pluralism model incorporates the components of multiattribute utility analysis of Dunn (2018), systematic approach framework of Rossi et al. (2019), and logic model. Schalock's (2002) model allows for summative evaluation and more specifically a conclusive outcome evaluation. The first of the five step of the model consists of describing the intent of the policy. This is followed by analyzing the policy's anticipated outcomes. In the third step the focus of the evaluation in terms of individual or organization, the evaluative standards (performance versus value assessment), and outcome indicators are decided. In the fourth step the status of the anticipated outcomes is evaluated. This step involves use of quantitative or qualitative methods to analyze the status of the measured policy outcome. In the fifth and final step of the process a feedback is provided to the policy makers to improve public policy process. Schalock's model (2002) can be adapted for a single stakeholder. It is based on personal value and personal appraisal of the selected stakeholder. Schalock (2002) also demonstrated the suitability of survey-based evaluations and individual data sets for the evaluation of public policy outcomes. The authors found this model comprehensive and suited the objective of the study. The majority of the literary work highlighted one common requirement of identification of correct outcome measures and then identifying the appropriate methods of measuring them.
varied approaches in assessing telecom sector outcomes. The sectoral reports by telecom regulators around the world focus on specific indicators like teledensity, tariff, usage, and so on. Such reports are published by regulators like: Office of Communications (Ofcom), the UK; Federal Communications Commission (FCC), the USA; Australian Communications and Media Authority (ACMA), and so on. The use of operational reports, customer surveys, and reports on technical parameters to assess QoS is common. Countries like Brazil, China, and Mexico, use customer surveys to assess the quality of the regulatory framework (ITU, 2018a). Argentina, Japan, Portugal, and Switzerland use customer surveys to validate the QoS reports submitted by the telecom operators (ITU, 2018b).
Survey-based assessments of telecom service attributes are also common. Intergovernmental agencies like ITU and OECD do quality surveys at the country level. OECD has a comprehensive framework of RIA and policy evaluation. ITU and OECD regularly monitor the implementation of RIA in their respective member countries. Ofcom, FCC, ACMA, and TRAI survey telecom users with respect to quality of service evaluation. Ofcom drives an agenda of better policy making, but it is similar to RIA. The objectives are identified ex-post and impact assessments are done against them (-Better policy making‖, 2010).
The authors could find limited examples where certain aspects of systematic outcome evaluation were used. Guadalinfo project in the Andalusia region on Spain employed ex-post evaluation of the initiative. The initiative is intended to improve access to ICT services. A detailed list of indicators was established to ensure quantitative and qualitative analysis of the level of achievement of the policies. Internet penetration, percentage of users creating digital content were some of the indicators set up. These indicators were eventually found to be critical in evaluating the program (Abreu, 2012). The New Mexico State University conducted a customer appraisal-based ICT outcome assessment using a stakeholder survey. The goal attainment model of the evaluation was adopted and considerable insight on process and outcome was obtained (New Mexico State University, 2015). While monitoring of specific telecom-related parameters and status indicators on overall telecom and ICT sectors are common, a systematic policy evaluation could not be found by the authors.

Outcome-based policy evaluation of India's telecom policy
Status of the Indian telecommunication industry in the form of annual reports, study reports, telecom reforms reports, etc. are published by TRAI 1 and DoT 2 . TRAI conducts surveys through third parties for audit and validation of the QoS reports provided by the telecom operators (TRAI, 2017). The QoS reports covers parameters like call drops, data speed, activation, BTS downtimes, billing, latency, throughput, and call success rate. SERVQUAL, SERVPERF, and Likert scale-based user surveys have been typically used. The surveys cover the aspects of availability, reliability (technical parameters), and quality of telecom services (technical parameters).
The private sector and academia have also done various studies related to telecommunications services in India. Indian Institute of Technology Delhi and Consumer Unity & Trust Society (CUTS) did an Internet usage study based on a survey (5-point Likert scale) and secondary data on QoS related to mobile services (CUTS International & IIT Delhi, 2016). Based on the response from 800 users, SG Analytics in its survey in 2016 found 50% of the respondents were unhappy with their present telecom operator (SG Analytics, 2016). Mishra, Singh, and Farooq (2020) conducted users' surveys to assess the awareness and affordability of telecom services. In a survey of 408 telecom users, half of the respondents were not aware of DoT.
While there are fragmented studies in silos on various aspects of telecom, the authors could not find a systematic policy evaluation related to telecom. NTP-1994, NTP-1999, NTP-2012 published by DoT captures quantitative, as well as qualitative objectives, a systematic evaluation of such policy is not available from either DoT or TRAI. While NTP-1994, NTP-1999, NTP-2012, and NDPC-2018 lay out various objectives related to different areas of telecom, the vision statement in NTP-2012 lay out the specific outcome indicators of telecom policy (based on guidance from Rossi et al., 2019). These outcome indicators are availability, affordability, reliability, quality, and security of telecom services. These individual components can be measured based on various approaches in the literature.

Approaches
to measuring availability, affordability, reliability, quality, and security of telecom services The literature is divided into a technology-oriented definition and a user-based definition and measurement of these parameters. Measures like mean time to failure (MTTF), mean time between failure (MTBF) are technology-based measures for availability and reliability. These metrics may be misleading, as these do not represent the perspective of usability of the services (Lehr, Bauer, Heikkinen, & Clark, 2011). The quality of a telecommunications network is a sum of the application's performance and network quality. User perception is linked with the QoS and quality of experience (QoE) measures. Their perception of services is altered based on changes in these metrics (Al-Shehri, Loskot, Numanoglu, & Mert, 2017). When it comes to end-users, perceived quality is the preferred metric (Zeithaml, 1988). Reliability and robustness are usually measured at the infrastructure level. The quality of recovery (QoR) metric is used at the network level. Network connectivity, node connectivity, cascading failure assessment are commonly used to evaluate reliability in telecommunication networks (Al-Shehri et al., 2017).
Return on investment (ROI) on security investments and financial losses due to security breaches are some of the metrics related to the security of services. Other methods like exploit dependency graph (Bhattacharya & Ghosh, 2012) and common vulnerability scoring system (CVSS) (Frigault, Wang, Jajodia, & Singhal, 2017) are also used. Serrelis and Alexandris (2010) converted disperse measurement of security into a single metric. They combined percentage conformity with the legal and regulatory framework, availability of service, profits from the particular service, size of economic losses, and price of the stock.
The literature establishes that when it comes to attributes of services, mostly the measures the technical. End-user metrics are required as the users' perception is the preferred measure to evaluate the outcome. There are fragmented studies on various aspects of telecom, but a systematic policy evaluation related to the telecom policies of India was not found. The authors could not find literary work where the outcome measures of availability, affordability, security, reliability, and quality of telecom services are accessed as a single study. The authors attempt to bridge this research gap by carrying out a systematic evaluation of India's telecom policy based on the stated vision in NTP-2012.

RESEARCH METHODOLOGY
The authors used the methodological pluralism model from Schalock (2002) and adapted it to a policy evaluation model for a single stakeholder in the Indian telecommunications industry. The end-user of telecom services in India was considered as the stakeholder. The assessment of the availability, affordability, reliability, security, and reliability of telecom services by the end-users was considered as the outcome measures. The following Table 1 details the adaptation of Schalock's (2002) outcome-based policy evaluation for this study.
In the first step of the analytical model as shown in Table 1, the intent of the policy outcome evaluation was established. The authors intended to find if the Indian policy maker achieved the vision for telecom as stated in NTP-2012. The anticipated outcomes were derived from the vision statement as mentioned in NTP-2012 (Rossi et al., 2019). These were the availability of affordable, secure, reliable, and high-quality telecom services anytime and anywhere.

Table 1. The analytical model of telecom policy outcome evaluation
Step 1 Question asked

Step 2 Anticipated outcome
Step 3 Focus of the evaluation

Step 4 Measurement approach and analysis
Step 5 Did the Indian policy maker achieve its vision for telecom as stated in NTP-2012 In the third step, the focus of the evaluation was identified. Effectiveness was identified as the dimension of policy evaluation. The focus of evaluation was the individual telecom users and the value they associated with the telecom services. To assess the user's evaluation of the policy outcome, data collection through a structured questionnaire was planned. This method enabled the authors to gather individual-level data sets. The collected data set was subjected to factors analysis to confirm that users' responses were aligned to five outcome variables that were picked up from the vision statement for measurement and appraisal. Regression analysis was carried out to find out, what drove the user's assessment on a particular outcome related to the policy makers' vision. The outcome of regression analysis would ensure that users' appraisal of the outcome of the policy is based on their actual experiences rather than unrelated factors. The information thus arrived was used to provide feedback to the policy maker in this study.

Instrument
Primary data was collected through a survey of mobile services users using a structured questionnaire. The questionnaire was designed to gather data on the perception of telecom services and the user's verdicts on the outcome of telecom policies. The authors opted for closed-end questions to put minimal cognitive burden on the respondents. It also ensured ease in answering, easy quantification, and the possibility of asking more questions in a given length of time (Krosnick & Presser, 2010).

Measurement scale
Five-point Likert scale with the middle option was chosen due to its validity and reliability (Revilla, Saris, & Krosnick, 2014). Likert-type scale data can be treated as interval data and subjected to factor analysis (Boateng, Neilands, Frongillo, Melgar-Quiñonez, & Young, 2018). The authors used multiple questions related to individual aspects of availability, reliability, quality, affordability, and availability of telecom services from the end-user. It was done to avoid the instability caused by any particular item, emphasis, or mood changes.

Pre-test
The questionnaire was reviewed by experts in the field of telecom to ensure relevance (-face validity‖) and completeness (-content validity‖). After incorporating their feedback, the questionnaire was administered to select subscribers of telecom services. Their comments in terms of understanding the objective of the questionnaire, clarity of questions, and difficulty (willingness) in answering the questions were incorporated.
The questionnaire initially contained 38 questions. Eight questions were related to demographics, 25 questions were related to Indian telecom users' perception of telecom services and five questions were related to users' verdict on policy outcome. Based on the pre-test, four questions on service perception were dropped. The modified questionnaire was then administered to the wider audience in the final step.

Sampling and sample size Sampling
The sampling frame consisted of all the users of telecommunications services in India and the sampling unit was the individual telecom subscriber. To cover users from different locations of India, authors used convenience/purposive sampling with multistage. In the first stage, telecom users in the immediate contacts of the authors were selected. It was ensured that individuals across the North, South, West, and East regions of India were included. In the second stage, each of the telecom users selected in the first stage was requested to cascade the survey to users in their immediate contact list. Users selected in the first stage used their convenience to select samples for the second stage.

Sample size
The total population of Indian mobile users as of September 2021 was 1189.15 million (IBEF, 2021). For calculating the sample size for this population, the authors assumed maximum variability (which is equal to 50% (p = 0.5, q = 0.5) at a 95% confidence level with ±5% precision. Based on Cochran (1977, p. 75), the representative sample of 384 was arrived at for a finite population of 1189.15 million subscribers. This method has been used and found suitable in many studies including Adam (2020) and Memon et al. (2020).

Execution of data collection
The questionnaire was circulated online and was hosted on a popular survey portal. The questionnaire was self-administered to avoid the interviewer's bias and social pressure (Krosnick & Presser, 2010). The online questionnaire link was distributed through email and WhatsApp messaging to more than 600 subscribers. This was done to take care of non-responses and rejection due to inconsistencies in responses. In all 411 responses were received, but three were discarded as their submission on demographics was inconsistent. Hence, 408 responses were available for further analysis, which was more than the required sample size of 384. The data collected were subjected to the descriptive statistics, t-test, analysis of variance (ANOVA), factor analysis, and regression analysis.

Descriptive statistics
Descriptive statistics in the form of mean, mode, median of perception scores, percentage agreement of the responses of the users about attributes of telecom services, and user's verdicts on achievement of vision by Indian policy makers was carried out. Results and their interpretations are presented in the following sections.
4.1.1. User perception on availability, affordability, security, reliability, and quality of telecom services in India The survey respondents were asked multiple questions related to the attributes of availability, quality, security, reliability, and affordability of telecom services in India. The following Table 2 and Table 3 detail the statistics of responses. As evident from Table 2, more than 70% of the survey respondents agreed that telecom services are available on-demand and to everyone. However, only 34% of the respondents agreed that telecom facilities are adequate in remote, hilly, and tribal areas of India. More than 77% of the respondents found telecom service available any time of the day without any issues. More than 80% of the respondents agreed that telecom services are available to all income segments. Above 70% agreed that they can use telecom services without compromising on any other household expenditure. However, less than 55% of the respondents agreed that they are charged for their telecom service usage clearly and transparently. 65% of the respondents confirmed that they can rely on the telecom service to communicate during an emergency. 79% of the respondents expressed that they can rely on telecom services for office and business needs. 52% of the respondents felt safe in their transactions with their mobile service provider. 23% of the respondents agreed that their personal information is safe with their service provider. 47% of the survey respondents agreed that the banking and e-commerce done on the network of their telecom service provider is protected. More than 70% of the respondents felt that their voice calls and data transactions can be intercepted. 60% of the respondents disagreed that DoT and TRAI have established effective measures to protect the security and privacy of consumer information. 82% of the respondents did not get the same QoS across all regions of India. More than 80% of the respondents faced interruption in the telecom services. More than 72% mentioned that they do not get the same quality of services while using voice and data services. Nearly 85% of respondents did not get high quality of network coverage. 67% of the respondents felt that TRAI and DoT have not been successful in ensuring compliance of the prescribed performance standards and QoS parameters by the telecom service providers. It is evident from Table 3 that the highest mean perception score is for P4, i.e., availability of telecom services to all income groups, followed by P8, i.e., no impact on household income. The mean perception scores are low on quality-related parameters. This is also evident from the values of median and mode. The respondents did not agree that DoT and TRAI have been successful in ensuring the security and QoS of the telecom services in India (P12).

Users'
verdict on policy outcome (achievement of the vision for telecom) Survey respondents were asked to provide their verdict (belief) on the policy outcome measures mentioned in the vision statement by the policy makers in NTP-2012. The following Table 4 and  Table 5 capture the summary statistics of responses.  It is evident from the statistics given in Table 4 that the mean verdict score is maximum on affordability followed by availability. It is the least for quality.
Based on the data that was analyzed (Table 5), it was inferred that 85.8% of the respondents agreed (cumulative of agree and strongly agree responses) that telecom services are affordable. 50% of respondents agreed (cumulative of agree and strongly agree responses) that telecom services are reliable. 68.9% of the respondents did not agree (cumulative of disagree and strongly disagree responses) that the telecom services are secure. 74.7% of the survey respondents did not agree that telecom services are of high quality. 55.6% of the respondents did not agree (cumulative of disagree and strongly disagree responses) that telecom services are available anytime, anywhere.

Inferential statistics
The data from survey response was subjected to ANOVA to ascertain variation in mean perception scores across various demographics of the telecom subscribers. The following set of null hypotheses was tested for four demographics, namely: H10: There is no significant difference between the mean perception score of males and females.
H2 0 : There is no significant difference between the mean perception score for different age groups.
H3 0 : There is no significant difference between the mean perception score for different educational qualifications.
H4 0 : There is no significant difference between the mean perception score for different occupations.

Results of hypotheses testing of user mean perception scores across demographics
The following Table 6 details the results of hypotheses testing of mean perception score of telecom service attributes across the demographics of the respondents. As evident from Table 6, no statistically significant difference across the demographics of the respondents was observed in the responses for the availability of telecom services. A statistically significant difference was found between the responses of males and females on the aspect of quality. While 15% of male respondents agreed that mobile network is of high quality, only 6% of female agreed with that statement. Responses between males and females statistically varied on a question related to taking a loan for meeting telecom expenses. The responses on affordability showed no statistically significant difference across age groups. However, students in the lower age group mentioned that they may have to forego some other expenses to use telecom. On the aspect of getting the same QoS across all regions of India, no significant difference in responses was found. The responses on availability, reliability, privacy, and quality were similar across the education levels. The responses on availability, affordability, quality, and reliability of telecom services were similar irrespective of the respondent's occupation.

Results of hypotheses testing of user mean perception scores of statements related to the outcome of telecom policy across demographics
Analysis of data in the form of p-values for testing the hypothesis of mean score of responses for the statements relating to outcome across four demographics is given in Table 7. It is evident from the results that hypotheses are rejected in four cases at a 5% level of significance. Three in the case of gender and one in the case of educational qualification. There is a significant difference between the mean perception score among males and females for three policy outcome statements, i.e., O1, O2, and O4. For educational qualification mean perception score for the statement O1 is significantly different. For the rest demographic, there is no significant difference in mean perception scores.

Dimension reduction (factor analysis)
As mentioned in research methodology, data were subjected to reliability measures, KMO, Bartlett test, and factor analysis. The results of the analysis are discussed in the following sections.

Cronbach's alpha
Likert scale data need reliability checks before subjecting it to factor analysis. Cronbach's alpha for the reliability of the final questionnaire was > 0.7. With the sample size greater than 384, this Cronbach's alpha indicated an acceptable level of reliability in measuring the same latent variable (Mohamad, Evi, & Nur, 2018). Cronbach's alpha was also acceptable to ensure internal consistency as it was used across all items (Gharaibeh, Al-Smadi, Ashour, & Slater, 2018). The values of Cronbach's alpha are given in Table 8. It can be seen that the KMO statistics value for service perception variables was 0.885 and for verdict variables 0.794. As per the criteria specified by Kaiser (1974), these values fall between -meritorious‖ to -marvelous‖ for doing factor analysis. The p-value from the Bartlett test of sphericity for service and verdict parameters was less than 0.001. This endorsed the suitability of data for the factor analysis (Knekta, Runyon, & Eddy, 2019). Table 9 details the result of the factor analysis of the 21 service perception variables. The responses to questions related to user perception of telecom services were divided into five factors. Factors with communality greater than 0.5 or nearly equal to 0.5 due to higher explanatory power were picked up (de Barros Ahrens, da Silva Lirani, & de Francisco, 2020). P7 was the only item related to the aspect of loan, and for prudence, it was retained even if the communality was less than 0.5.

Factor analysis of the response data on the perception of attributes of telecom services
As evident from Table 9, responses on qualityrelated questions loaded onto Factor 1 (Quality Factor). Responses related to questions on security loaded onto Factor 2 (Security Factor), and responses to questions on availability loaded onto Factor 3 (Availability Factor). The responses on questions related to reliability loaded onto Factor 4 (Reliability Factor) and responses on affordability question loaded onto Factor 5 (Affordability Factor). The five factors put together were designated as Service Perception Factors. These factors together explained more than 60% of the variance in factor analysis. The five factors were named as per the details are given in Table 10 and their mean scores and other statistics are given in Table 11.

Factor analysis of the response data on user verdict on telecom policy outcome
The following Table 12 details the result of factor analysis of all the responses related to the question on user's verdict on outcome of telecom policy. The response data on verdict-related questions loaded into two factors. The responses on reliability, security, availability, and quality are summarized into Factor 1 (Service Attribute Verdict) and responses on affordability  in Factor 2 (Affordability Verdict) as given in Table 12. These two factors together explained more than 71% of the variance in factor analysis. The descriptive statistics of these factors are shown in Table 13.

Do service perception factors drive the users' verdict on telecom policy outcomes?
The authors found a significant correlation between the service perception factors and verdict factors. As evident in Table 14, Service Attribute Verdict (Factor 1) had a significant correlation (at 1% level) with Quality Factor, Security Factor, Availability Factor, and Reliability Factor. Affordability Verdict showed a significant correlation (at 1% level) with Availability Factor, Reliability Factor, and Affordability Factor. This indicated a statistical association and relationship between the verdict and perception factor. The linear relation between variables was evident in the respective scatter plots. ). The authors hence analyzed the interplay between service perception factors and verdict factors (on policy outcome). This relationship establishment was important to ensure that user verdicts on policy outcomes are related to their experience of telecom services and not driven by some other factor. Based on guidance from the studies of Kim and Lee (2013), Sharma (2019), as well as Alzoubi, Alshuridehb, Kurdi, and Inairat (2020), the relationship between verdict factors and service perception factors was conceptualized in the equation (1).
The following regression equation (2) was formulated for the analysis. An ordinary least square (OLS) method was used for the estimation of parameters. (1) The output of regression between Service Attribute Verdict as a dependent variable and Service Perception Factors as the independent variables are detailed in Table 15 (Model 2). The outcome of regression between Affordability Verdict as a dependent variable and Service Perception Factors as an independent variable is detailed in Model 1 (Table 15). F-statistics is statistically significant for both Model 1 and Model 2, confirming model fit. Variance inflation factor (VIF) and tolerance statistics of 1 indicated no issue of multi-collinearity. All the residuals were normally distributed confirming the robustness of the model.

Affordability Verdict as a dependent variable
As shown in Table 15, the regression between Affordability Verdict as a dependent variable and Service Perception Factors as the independent variables had an adjusted R 2 value of 0.287. Quality Factor, Security Factor, Availability Factor, Reliability Factor, and Affordability Factor jointly explained nearly 30% of the variance in Affordability Verdict of the respondents. The unstandardized coefficients were largest for Affordability Factor (0.392), followed by Availability Factor (0.276) and Reliability Factor (0.224). These were significant at a 1% confidence level. A one-unit change in affordability perception, availability perception, and reliability perception positively improved Affordability Verdict by 0.392, 0.276, and 0.224 units respectively. Security Factor and Quality Factor had unstandardized coefficient values of 0.090 and -0.085 respectively at a 5% confidence level. Users' verdict on policy outcomes related to affordability was found to be based on their actual experiences captured under the perception factor.

Service Attribute Verdict as a dependent variable
The regression model with Service Attribute Verdict as a dependent variable and Service Perception Factors as the independent variables (Model 2, Table 15) had an adjusted R 2 value of 0.643. Affordability Factor, Reliability Factor, Availability Factor, Security Factor, and Quality Factor together explained 64% of the variation in the value of Service Attributable Verdict. The unstandardized coefficients were largest for Quality Factor (0.533), followed by Security Factor (0.511), Reliability Factor (0.259), and Availability Factor (0.184). A one-unit change in Quality Factor, Security Factor, Reliability Factor, and Availability Factor positively improved Service Attribute Verdict by 0.533, 0.511, 0.259, and 0.184 units respectively. Users' verdict on policy outcome related to availability, reliability, security, and quality was found to be based on their actual experiences captured under the perception factor.

DISCUSSION
The policy evaluation models in the literature highlighted the importance of choosing the correct outcome measures and their impact on the outcome evaluation. In this study, the authors used the outcome measures which were mentioned in the vision statement for the telecom sector. It did not quantify the outcomes of availability, reliability, security, quality, and affordability of telecom services. A user-based appraisal was hence suited for the measurement of the actual status of the outcome parameters.
The measured policy outcome when compared with the vision statement in the NTP-2012 policy document indicates that the policy is not effective. This is supported by the finding that only 34% of the respondents agreed that telecom facilities are adequate in remote, hilly, and tribal areas of India. Popli and Madan (2013) had similar findings in their empirical research where the availability of telecom services in rural areas was found to be low. Even when the telecom services are affordable for population (based on a sample mean of 4.04 and standard error (SE) of 0.039, at a 95% confidence interval) the charging for such services is not perceived as transparent. The population (based on a sample mean of 3.24 and SE of 0.049 at 95% confidence level) of telecom users do not agree that telecom services are reliable.
Surveyed users expressed issues of security while transacting on a cellular mobile network. They are concerned that their personal information is not safe with their service provider. Poor confidence of surveyed users on doing banking and e-commerce on the telecom network is a jolt to the digital transformation ambition of Indian policy makers. When the findings are extended to the population of telecom users (based on sample means of responses and standard error at 95% confidence), the statistics are similar. With improved technology and an active regulator that monitors QoS, it was not expected that more than 80% of the survey respondents complained about QoS. The statistics based on the sample mean and SE is similar for the population. Non-uniform QoS across geographic regions, interruptions in services, differential QoS on voice and data are major concerns. The QoS reports from TRAI do not reflect this experience of the endusers. These findings of users getting the inferior quality of telecom services was also noted by Kushwah and Bhargav (2014) in their study of Indian telecom users.
With 89% teledensity, the availability of telecom services can be considered high. The responses from surveyed users however showed different statistics. 55.6% of the respondents did not agree that telecom services are available anytime and anywhere. The statistics for the population was similar as sample mean was 3.06 and with SE of 0.055 the population mean mapped to neither agree and nor disagree. This confirms that teledensity is not the true measure of usability of service. When 89% population has access and only 55% can use it on demand, the teledensity metric used by DoT and TRAI for sector outcome is not objective.
To summarize, out of the five policy outcome measures the survey respondents (and the population-based on sample mean and SE) agreed only to the outcome of affordability. This outcome is puzzling as India does not figure in the top fifty countries when affordability is measured in terms of telecom expense as a percentage of gross national income (ITU, 2020). The impact of this metric from ITU on affordability is visible in the difference of opinion between male and female respondents. Female survey respondents mentioned issues with affordability and compromising on other spendings for using telecom services. These results are consistent with the findings of GSMA (2018) and Barboni et al. (2018) who raised the issue of the access gap between males and females in India. When India has one of the lowest ARPU in the world, the results are concerning as the supply-side metric of ARPU is competitive, but the demand-side outcome of a gender gap is wide.
Factor analysis of responses on attributes of telecom services loaded into the five policy outcome measures that were defined based on vision sated in the telecom policy document. However, when respondents' verdicts on these five outcome measures were analyzed, they were divided into two distinct factors. The first was related to Service Attribute Verdict and the second  to Affordability Verdict. This indicates that policy measures need to be adopted separately for these two factors to improve the overall policy outcome. Service attributes are directly under the influence of DoT and TRAI, but the affordability factor depends on macroeconomic factors and the overall economic development of India. The supply-side dynamics of affordability in terms of tariff can be improved only to a certain extent based on competition. India, having one of the lowest ARPU in the world, indicates that supply-side actions on affordability are already at play. Actions to improve affordability from the demand-side may need to be developed by the government of India in consultation with DoT.
The regression between Affordability Verdict as a dependent variable and Service Perception Factors as the independent variables indicated that affordability perception, availability perception, and reliability perception positively improve Affordability Verdict. If efforts are made to improve these parameters, user verdict on the affordability of services can be improved. Quality Factor was negatively driving survey respondents' verdict on affordability.
Survey respondents perceived high-quality services as less affordable. Literary work on the price-quality relationship supports such behavior (Lai, Yuen, & Chong, 2020;Ni & Li, 2018).
The regression model with Service Attribute Verdict as a dependent variable and Service Perception Factors as the independent variables indicated that perception of quality and security of telecom services are the biggest drivers of users verdict on service attributes. If quality, security, reliability, and availability of services can be improved, the overall user verdict on service attributes of telecom can be significantly improved. These results are consistent with findings from Gautam (2015) who found similar results in a survey of Indian mobile customers. The service providers must improve network coverage and reduce call drops.
More than half of surveyed respondents did not agree that DoT and TRAI have established effective measures to protect the security and privacy of consumer information. A similar percentage of respondents also opined that TRAI and DoT have not been successful in ensuring compliance with the prescribed performance standards and quality of service parameters. TRAI must revisit the QoS parameters it mandates for the service providers as the on-ground situation is concerning. DoT must also start a customer outreach program like TRAI to understand the on-ground status of policy outcomes. Targeted programs to improve QoS, security, reliability, and availability of service needs to be designed in collaboration with operators, academia, and intergovernmental organization like ITU if the policy outcomes are to be improved.

CONCLUSION
Outcome-based policy evaluation is an established practice in distributive and redistributive public policies. However, such practices are not well-known for evaluating competitive regulatory policies in telecom, especially in India. Since the telecom policy NTP-94, the regulators, policy makers, and operators have worked in tandem to make India the second-largest telecom market in the world. In NTP-2012 the policy makers envisioned -to provide secure, reliable, affordable and highquality converged telecommunication services anytime and anywhere for an accelerated inclusive socio-economic development‖ (DoT, 2012, p. 4). However, even after twenty-five years of the first telecom policy and announcement of four telecom policies, there are ongoing grievances registered by telecom operators on the policy front. Moreover, the end-users continue to face issues in availability, quality, reliability, and security of telecom services. While fragmented studies on the outcome of the telecom sector are done by DoT, TRAI, and academia, there is no systematic policy evaluation that is evident. The authors bridge this research gap in this novel study by carrying out a systematic outcome-based evaluation of India's telecom policy. To the best of the knowledge of the authors, this is the first of its kind study for the Indian telecom sector and is expected to generate valuable insight to improve policy making and to measure policy outcomes.
The authors adopted the methodological pluralism model and policy evaluation model from Schalock (2002) for the outcome-based evaluation of telecom policy in India. The study carried out an appraisal by the end-user of the attributes of telecom services mentioned in the vision statement for telecom in the NTP-2012 policy. The policy outcome measures of availability, affordability, security, reliability, and quality of telecom services were adopted from the vision statement of DoT. The effectiveness and goal attainment dimension of policy evaluation was used. Data on the individual appraisal of policy outcomes was collected through a structured questionnaire. To provide specific guidance to the policy makers the relationship between the end user's appraisal of attributes of telecom services and their verdict on policy outcomes was ascertained.
Against the policy outcome measures of availability, quality, security, reliability, and affordability, 85.8% of the respondents found telecom services affordable. 50% of respondents agreed that the services are reliable. 68.9% of the respondents did not agree that telecom services are secure. 74.7% of the survey respondents indicated that telecom services being provided are not of high quality. 55.6% of the respondents did not agree that telecom services are available anytime and anywhere. Taking into account the sample mean and SE, at a 95% confidence level, a similar conclusion can be drawn for the population of mobile users in India. Ironically 60% of the respondents disagreed that DoT and TRAI have established effective measures to protect the security and privacy of consumer information. 67% of the respondents felt that TRAI and DoT have not been successful in ensuring compliance of QoS parameters by the telecom service providers. Out of the five policy outcome measures, only affordability is agreed by the users as an achieved outcome.
The factor analysis of the responses from the users' on service attributes is divided into five outcome measures of availability, affordability, reliability, quality, and security of telecom services. The regression analysis between Service Perception Factors and Service Attribute Verdict on policy outcome revelated that users' verdicts were based on experience and were not based on some other isolated factors. Regression analysis also revealed that user verdict on policy outcome can be significantly improved if users' experience of affordability, availability, and reliability of services can be improved. The telecom users value usability of service and not teledensity which was reflected from the fact that on a teledensity of 89.9% only 55% of the respondents agreed on the on-demand availability of telecom services. Quality of services negatively drove the verdict on affordability. The high quality of services was perceived as more expensive by users. This outcome arguably reflects that either the operators' offerings are expensive for high-quality services or the price-sensitive nature of telecom users is affecting this view.
This study demonstrates the gap that exists in doing a systematic evaluation of telecom policy outcomes in India. Telecom sectors reports may boast of high teledensity and minutes of usage, but telecom users continue to face issues in availability, reliability, quality, and reliability of services. Quality, security, and reliability of services are more valued by telecom users than mere access and availability. The definition of availability needs to include quality, security, and reliability as additional dimensions of access. The Indian telecom market is highly competitive, there is no retail price regulation and there are mandated QoS requirements. Still, users do not get the entitled services and basis metric associated with world-class telecom services. This indicates that the outcome measures adopted by TRAI and DoT are not the same as outcome measures mentioned in the vision for telecom in the policy documents. The policy makers and regulators must go beyond the teledensity figures and affordable services to ascertain the achievement of the vision for the Indian telecommunications industry. They need to evaluate the measures that can be adopted to improve the services. The telecom users' perception that DoT and TRAI can not ensure security and high quality of service further strengthens the need for such measures.
The time is ripe to carry out a systematic evaluation of policy outcomes by TRAI, and the academia for the Indian market. Fragmented reports on outcome like teledensity, minutes of usage, data speed are not a true representative of the policy outcome. A multistakeholder evaluation is needed as operators and end-users are not aligned with the policy makers' view on achievement in telecom. Before every new telecom policy is announced, the Indian policy makers should publish the outcome of systematic policy evaluation of previous policies. Feedback from all the stakeholders on such outcome evaluation should be taken to improve transparency and policy-making cycle. Intergovernmental organizations like ITU publish regulatory trackers but have not standardized a policy evaluation framework. OECD limits its recommendation on policy evaluation to member countries. It is time that intergovernmental organizations realize that sector-specific outcomes and policy outcomes are different and a standardized framework for policy evaluation for telecom must be established. The public policy of telecom has a far-reaching impact on various dependent services in other sectors, which mandates such action.
The authors adopted a retrospective policy outcome analysis using cross-section data. A cross-section study might be influenced by a telecom user's recent experience of services. In such cases, their verdict on policy outcome suffers from recency effect. The study measured the outcome level and not outcome change due to same reasons. Future research based on longitudinal studies should be done to measure outcome change. Such studies may provide further visibility on the reasons affecting policy outcomes. This study used the policy outcome mentioned in the vision statement of the policy makers. The vision statement by nature does not quantify the policy outcome. Alternate outcome measures in agreement with the policy makers can help in detailed outcome analysis. This paper is based on an appraisal by a single stakeholder. Further research can adopt appraisal by operators and telecom vendors for a holistic evaluation.