Quantitative evaluation of two-way referral policies based on PMC index model

Abstract

Objective

Perfect policy system support is the key to effective work, and policy evaluation can provide theoretical basis for policy revision. This study aims to quantitatively evaluate the two-way referral policy in China and thus improve the policy system of two-way referral.

Methods

This research focuses on 14 two-way referral policies published from 2007 to 2021. Basing on the focus of the two-way referral policy identified by text-mining methods, we constructed a PMC index model containing 9 primary and 43 secondary variables to conduct a quantitative evaluation of the two-way referral policy.

Results

All document types are work notices. The two-way referral policy issued by relevant departments have gradually increased since 2015.The average PMC index of the 14 documents was 5.84, and the policy with the highest PMC index score was the policy P14 issued by Jiangsu province. There are 6 excellent policies, 8 good policies, and no qualified grade policy. The difference in PMC index scores mainly shows the three dimensions of policy clarity, policy content and policy coverage.

Conclusions

This study indicates that the quality of two-way referral policy is generally good, which reveals the factors affecting the policy operability and contributes to expand the research perspective of two-way referral policy evaluation.We also analyzed the advantages and disadvantages of the existing policy of two-way referral through policy evaluation, and provided guidance for subsequent policy revisions.

Introduction

The reform of China’ s health system focuses on the hierarchical diagnosis and treatment system, the modern hospital management system, the universal medical insurance system, the drug supply guarantee system and the comprehensive supervision system, among which the hierarchical diagnosis and treatment system is the first of China’ s “five basic medical systems”. The basic mode of hierarchical diagnosis and treatment is first diagnosis at primary medical institutions, two-way referral, classification and treatment of acute and chronic conditions, and linkage between higher level and subordinate medical institutions [1]. Among which two-way referral is an important embodiment of the operation effect of hierarchical diagnosis and treatment [2]. Patient referral rates in other countries are generally between 10 and 30%, but less than 5% in China [3].

China’ s public medical institutions are divided into first-level hospitals, second-level hospitals and third-level hospitals. The higher the level of hospitals, the higher the difficulty of the diseases that can be treated. As tertiary hospitals have high-quality medical resources such as doctors with superb medical technology and advanced medical equipment, patients will go to tertiary hospitals directly as long as they get sick [4]. To relieve the pressure on tertiary hospitals, China further optimized the referral policy in 2015. According to the types of diseases, the reimbursement rate of medical insurance is differentiated from primary hospitals to tertiary hospitals, so as to restrict patients’ medical treatment behavior. The essence of two-way referral is to guide patients to different levels of medical institutions according to the severity of patients’ disease [5], and to reasonably divert patients who were concentrated in tertiary medical institutions before.

A sound policy document can point out the direction of action for health management reform, while policy evaluation can evaluate the advantages and disadvantages of existing policies and provide guidance for subsequent policy revisions [6]. The domestic and foreign studies on two-way referral focus on the aspects of two-way referral mode, implementation status, problem countermeasures, and efficiency of referral [7]. The analysis of two-way referral policy mainly focuses on literature review [8], and there are few studies on the analysis of policy quantification. The more commonly used tool in policy evaluation is PMC index model, which can scientifically and intuitively compare multiple policies [9]. In this study, we established the PMC index model, quantified the policies related to two-way referral, and explored the potential improvement points in its policy system, which provided a theoretical basis for the construction of a reasonable and effective two-way referral mode.

Methods

Data source

We took “two-way referral” and “referral” as the theme words, combined with the authority of policy sources, and searched in the CNKI government document database, the official website of the health commission of national and provincial (including autonomous regions and municipalities), and the Peking University Law database. Since the official establishment of the two-way referral system was 1997, the search publication time was limited to January 1, 1997-December 31, 2023. The inclusion criteria for the document are as following: the document type is one of the notice, opinion, method and scheme; the document issuing agencies are at the central government, provinces, municipalities directly under the Central Government and special economic zones; the subject of the document is the two-way referral. As illustrated in Tables 1 and 14 documents are included in this study, and the policy text is coded according to P1-P14.

Table 1 Basic information of the two-way referral policy text in China

Policy text mining and variable selection

We used ROST CM 6.0 to clarify the high-frequency words in the two-way referral policy document through text mining, eliminate the words that lack statistical significance such as “two-way referral”, “following” and “according”, and finally determined 60 high-frequency words with high frequency and reflect the focus of two-way referral (Table 2). As shown in Table 3, based on text mining and combined with other literature, 9 first level variables and 43 secondary variables were selected in this study.

Table 2 The top 60 high-frequency words in two-way referral policy texts

Table 3 Evaluation indicators and sources of the two-way referral policy

Table 4 Multi-input-output matrix of the two-way referral policy

The PMC index model

The PMC index model, proposed by Ruiz Estrada [16], is a common method to evaluate policy documents. It can present the advantages and disadvantages of various policies from multiple dimensions, and put forward a more scientific and effective path for policy improvement [17]. The PMC index is calculated in four steps [12]. Firstly, Multi-input-output tables were constructed using primary and secondary variables, as shown in Table 4; Secondly, the secondary variable is assigned. If the index reflected by the secondary variable is involved in the policy text, the secondary variable is assigned to 1, otherwise, the value is 0. Thirdly, the primary variable score is calculated, which is equal to the mean of all its secondary variables combined. Finally, the sum of the primary variable scores was calculated for each policy. Taking into account the relevant policy quality classification standard [18], the policy is divided into four grades: perfect policy (8–9 points), excellent policy (6-7.99 points), acceptable policy (4-5.99 points) and qualified policy (0-3.99 points).

Results

Publishing characteristics of the two-way referral policy

According to the included policy text, all document types are work notices. As illustrated in Fig. 1, the first document in line with the theme of two-way referral was released in 2007, and the number of two-way referral policy releases has gradually increased since 2015. 2017 was the year with the most publications, followed by 2015 and 2021, both with two policy releases. The core issuing agency of the two-way referral documents was the Health Commission, and 12 out of the 14 documents were issued by the health commission of each province and city. Only one document is joint publication, while the rest are independent publication, lacking the cooperation of multiple departments.

Fig. 1

The issuing year and numbers of two-way referral policy texts

Network analysis of the two-way referral policy

The social semantic network analysis of the 14 included policy texts was conducted by ROSTCM.6 software. As shown in Fig. 2, the two-way referral policy follows the two subjects of medical institutions and patients. The types of two-way referral patients can be seen from the high-frequency words such as “rehabilitation”, “chronic disease”, “frequently-occurring disease” and “complicated”. From the words “clear diagnosis”, “principle”, “condition” and “evaluation”, we can know that the key to two-way referral lies in confirmed patients. From the words “higher level”, “subordinate” and “up and down linkage”, we can see the linkage working mechanism of two-way referral. From the words “process”, “sign a contract” and “make an appointment”, we can find that the two-way referral emphasizes the standardization of referral.

Fig. 2

Social network analysis of two-way referral policy

Overall evaluation of the PMC index for two-way referral policy

As shown in Table 5, the average PMC index of 14 policies is 5.84. Six policies are rated as excellent, eight policies are acceptable, and no qualified policies. The design of two-way referral policy is generally reasonable. The highest PMC index score is policy P14, and the lowest score is policy P2. As indicated in Fig. 3, through the comparative analysis of all policies through the radar chart, it can be found that the difference of PMC index scores mainly shows three dimensions: policy clarity, policy content and policy coverage. From the perspective of policy evaluation indicators, among the 9 primary variable PMC index scores, the policy evaluation and policy clarity scores are high; the policy coverage score is low. Among the regional distribution, the two-way referral policies issued by Jiangsu, Yunnan and Hainan provinces had high scores, while the policies issued by Xinjiang Uygur Autonomous Region had the lowest scores.

Table 5 PMC index scores and grade evaluation of two-way referral policy

Fig. 3

Radar chart of two-way referral policy

Discussion

The development process of two-way referral policy

In 1997, relevant departments proposed the establishment of two-way referral system for the first time [19]. This research found that the relatively standard two-way referral document was only released in 2007, with a stagnation period of nearly 10 years, which is consistent with the research conclusions of other scholar [20]. Most of the studies on hierarchical diagnosis and treatment in China were after the new health care reform in 2009 [21]. In 2015, The General Office of the State Council issued the Guiding Opinions on Promoting the Construction of the Hierarchical Diagnosis and Treatment System, requiring it to gradually improve the hierarchical diagnosis and treatment system by 2017, and comprehensively improve the hierarchical diagnosis and treatment capacity by 2020, which is of positive significance to the implementation of two-way referral in various regions [22]. As illustrated in Fig. 1, the release of two-way referral policy texts increased significantly since 2015. Although a series of two-way referral documents was developed in response to national policies, the 14 policy texts included in this study show that their PMC index ranges from 4.99 to 6.86 and there is room for improvement.

Focus of two-way referral policy

As shown in Fig. 2, the smooth flow of two-way referral involves communities, medical institutions, medical staff, patients and other stakeholders. The policy formulation needs to comprehensively consider “willing to referral”, “whether to referral” and “how to referral” [23]. At present, the included policy texts mainly focus on the issue of “how to transfer”. According to the PMC index score, most policies define the scope of referral implementation and have clear referral procedures. Meanwhile, the division of rights and responsibilities among medical institutions and health departments at all levels is also standardized. For example, the Hainan Provincial Policy (P1) mentions: the Health Bureau organizes and implements two-way referral, secondary medical institutions undertake professional training tasks, and the community is responsible for popularizing the concept of scientific medical treatment to the public. The policies included in this study have had relatively little discussion on the issue of “ability to referral”, with only 43% of policies mentioning personnel training. However, general practitioners act as the gatekeepers of primary diagnosis and two-way referral whose service ability is closely related to the effect of hierarchical diagnosis and treatment. Therefore, the future policy formulation are supposed to emphasize this content.

Factors affecting the feasibility of two-way referral policy

Among the regional distribution, the two-way referral policies issued by Xinjiang Uygur Autonomous Region had the lowest scores, which may be related to the level of economic development and special geographical location of Xinjiang Uygur Autonomous Region [24]. There are many landforms, such as mountains, gobi and desert in Xinjiang Uygur Autonomous Region, and the winter snow season is long there, which causes some difficulties for the implementation of two-way referral work. As a result, the two-way referral policy formulated by relevant agency only stipulates some basic problems. Establishing a specific and continuous assessment mechanism is essential for providing continuous referral services [3]. Lack of clear referral guidance is an important reason affecting the hospital’s referral work [8]. As shown in Fig. 3, it can be found that the difference of PMC index scores mainly shows three dimensions: policy clarity, policy content and policy coverage. For the evaluation of the effectiveness of two-way referral work, there are few quantitative indicators in the policy, and each indicator lacks quantitative provisions, which reduces the feasibility in the policy assessment. In the two-way referral policy, the lowest PMC index scores were found in P2 and P3. Due to the lack of instructions on referral principles, standards, and protocols, P2 scored poorly in the policy content dimension. Due to the lack of policy publicity, system improvement and personnel training content, P3 scored low in the policy guarantee dimension. SMART principle (specific, measurable, attainable, relevant, time-based) was often used for designing practical indicators. When developing two-way referral policies in the future, we could consider adding indicators including the referral rate, referral speed, referral service quality, health care cost service efficiency and referral residents’ perception. Besides, the policy should also reflect the assessment period, indicators, methods and effectiveness significance of two-way referral work, which is also of great significance for promoting the development of two-way referral work.

The relationship between long-term policies and short-term policies

From the perspective of policy timeliness (X2), most policies belong to long-term policies, and only two documents are short-term policies (P9 and P11). This may be because long-term policies, as a macro guidance document, have the characteristics of overall, universal and comprehensive, which is of great significance for promoting the development of healthcare undertakings. However, promoting two-way referral should not only rely on long-term policies, but also need short-term policies. The validity period of the short-term policies is often within 1–3 years, with good flexibility. Short-term policies can be improved under the guidance of the long-term policies, as the basis for testing the phased completion effect of the long-term policy goals, which is more practical. Thus, detailed and executable implementation plans should be designed under the guidance of long-term policies [25].

Policy coverage and two-way referral

The PMC index of policy coverage score (X9) was low, indicating that less attention was paid to the hierarchical diagnosis and treatment system in the two-way referral document. In 2016, WHO proposed the framework of integrated medical and health service system. The integrated medical and health service system has reshaped the functional orientation of communities, rehabilitation nursing institutions and medical institutions, providing patients with integrated life cycle services such as health promotion, disease prevention and treatment. It is necessary to build a patient-centered and community-based integrated medical service system to improve the accessibility and satisfaction of residents’ medical treatment. Some scholars point out that taking the medical alliances as the carrier, expanding the coverage of remote collaboration network, signing up family doctors, and strengthening community construction can effectively promote two-way referral.

Conclusion

There is no quantitative study of two-way referral policy before. Basing on text mining and PMC index model, this study quantitatively evaluated 14 two-way referral policies from 2007 to 2021 issued by the Chinese government. We also explored the potential improvement points in two-way referral policies, which provided a theoretical basis for the construction of a reasonable and effective two-way referral mode. The two-way referral policy is overall reasonable which mainly revolves around patients and medical institutions. In the process of policy formulation, relevant departments tend to focus more on the foundation, content, rights and obligations of the objects involved, and clarity of policies, while paying less attention to the timeliness, coverage and feedback of policies. To optimize the two-way referral policy, we suggest increasing the short-term policy with a time span of less than 3 years, and clarifying the applicable institutional level and evaluation indicators when formulating the policy, such as: referral rate, referral speed, referral service quality, and views of referral residents, so as to enrich the policy content. Furthermore, enhancing cross-departmental collaboration through the establishment of referral centers is also a notable issue because referral involves different levels of medical institutions.