A research team from the Hong Kong University of Science and Technology (HKUST) has demonstrated how to measure vaccine effectiveness (VE) in the absence of adequate public health data, offering an easy-to-implement alternative to measuring VE and keeping data-deficient countries informed when formulating and adjusting their vaccination and immunization policies.

The development of effective vaccines offers us the best hope in strengthening our global effort to end the COVID-19 pandemic.  Taking advantage of a well-established medical recordkeeping system and a wealth of public health datasets, many developed countries are already measuring VE for their populations using test-negative design (TND) – the prevailing VE measurement which compares the vaccination status of persons receiving either positive or negative results in the same test.

But measuring VE could be difficult for low- and middle-income countries.  This is because while the health authorities in these countries may know how many people in the total population were vaccinated, infected, or hospitalized, they often don’t know whether a particular person who tested positive was vaccinated or not. Without such knowledge of personal vaccination status, the standard TND methodology could not be used.

As a result, current VE study efforts tend to be skewed toward the world’s more affluent countries.  For example, VIEW-hub, a comprehensive database of 249 COVID VE studies (as of early May 2022), contains only 16 large-scale studies (50,000+ subjects) from low- and middle-income countries – of which half are from Brazil.  Much of Asia and Africa is left out in the database, and such underrepresentation may create an information void for public policy in some of the most populous countries in the world.

Now, a research team led by Prof. Abhiroop MUKHERJEE and Prof. George PANAYOTOV from HKUST and Prof. Rik SEN from the University of New South Wales has demonstrated how to estimate VE even in the absence of such personal data.  The team exploits the fact that many governments applied age-based eligibility cut-offs when rolling out their vaccination programs – meaning that at some point citizens above the cut-off age are eligible for vaccination, whereas those below are not.

Using a simple statistical model called the Regression Discontinuity Design (RDD), the team then compared the various COVID outcomes (positive cases, positive cases with high CT value, hospitalizations, deaths) of groups of individuals above and below the cut-off age. These groups are very close in age, and hence assumed identical in terms of health conditions or their exposure to disease. This allows one to calculate VE with less data than required by standard VE methods.  A key input for RDD-based VE measure is the proportional drop in COVID outcomes at the cut-off age, which is factored into the calculation along other inputs, including the vaccination rates corresponding to the same age groups.

In their application of the above RDD method and an extended measure called breakthrough-based RDD to public health data in India, the team has found that even one dose of the vaccines used were 55% effective against infection and 80% against hospitalizations during the Delta wave in early 2021.  The data contain over 8 million observations on COVID-19 vaccinations and testing, as well as 140,000+ hospitalizations in West Bengal, an eastern state in India, where a 45-year age cut-off was imposed from March to April 2021.

The research findings were recently published in Science Advances.

“VE could differ between populations as individual countries’ pre-existing exposure to COVID-19 could vary. Helping individual countries to understand whether and how well vaccines are working for their populations can help ascertain vaccination policy.  For example, if a country finds that the one-dose VE for its population is high enough, it might make sense for them to provide first doses to more people, rather than insisting on two doses for fewer people – if vaccines are scarce,” said Prof. Mukherjee, also the Liwei Huang Associate Professor of Business at HKUST.  “Having accurate VE estimates can also help health authorities decide whether vaccinated populations will be spared severe outcomes, which is critical for deciding whether to ‘live with the virus’.”

Prof. Mukherjee added that almost all countries used – and are still using – age eligibility criteria for COVID vaccines, so the RDD measures are widely applicable, and will be particularly helpful in data-deficient countries where public health data infrastructure is not good enough to apply standard VE measurements.