A Look at Illinois' County and School Metrics

18 Sep 2020

The Illinois Department of Public Health (DPH) provides COVID-19 tracking Metrics and guidance on how policy makers can use them at the regional and county level. They also provide metrics tailored for schools to use as they deliberate in-person learning, remote learning, and everything in between. Their aim is to keep local health officials and policy makers informed about the current level of virus spread in their community and provide some guidance on how to act for different levels of spread. If you’re familiar with my reporting, then you’re likely familiar with some of the county and regional metrics. I refer to them often and have been reporting the regional positive test rate along side the county’s rate for some time now.

The Monmouth-Roseville school district is currently entirely remote. I have two elementary school kids in this district and it’s going pretty well for us. I’m really impressed with how the teachers and administrators in our district are handling things thus far but my family’s experience is not shared by all the families in our community. Just 16 days into the school year, there are already calls to return to in-person instruction. As a result, the district administrators and board of education are considering when, why, and how such a return will take place. This has pushed me to spend a lot of time considering the state’s metrics, what they look like when applied to Warren County, and how local school officials might use them to inform their decisions during the coming school year. As a parent in the district, I want to see evidence that the board and school administrators are using these metrics to make informed decisions and the extent to which the state’s reporting reflects the ground truth in Warren County.

Warren County

If we want to understand how to interpret these metrics in the context of Warren County, then we need to know a few things about Warren County. According to county population data from usafacts.org, the county is home to 16,844 people. I couldn’t find it posted anywhere, but based on the DPH case numbers and cases per 100000 posted by the DPH, the population of Warren county is 16,981. I usually use the usafacts.org number when comparing ourselves to other counties, but I’m going to stick to 16,981 because we’re looking at data calculated by the state using that as our population. The impact of the size of our population on the calculation and interpretation of state metrics will be a recurring theme in this post.

The state of Illinois groups counties into 11 Emergency Medical Services (EMS) regions. These regions are used by the state when tracking the availability of medical services during the pandemic. They originally served as the pandemic recovery regions. They’ve since made some tweaks, probably to more evenly distribute medical services based on case statistics, but the EMS regions are still the basis for the recovery regions. Warren county is in EMS region 2. EMS region 2 is entirely contained in recovery region 2 but recovery region 2 also includes Kendal and Grundy counties. When, in my weekly reports, I look at the regional trends I am looking at recovery region 2 and a few counties from Iowa. If you’ve been following that analysis, then you have a pretty good feel for what its like in recovery region 2 and by extension EMS region 2.

The Metrics

We now turn our attention to the metrics chosen by the state for evaluating the spread of COVID19 and the availability of medical resources for handling those cases. My emphasis is on metrics is directed at those meant for school and county officials. The state provides four metrics that school officials should use to gauge their county’s level of community spread of COVID19:

1. Cases per 100,000 people in one week
2. Count of actual cases in one week
3. Count of youth (under the age of 20) cases in one week
4. Test positivity rate for one week

These metrics are then combined with the eight metrics used to assess county-level risk:

1. Cases per 100,000 people for one week
2. Test positivity rate for one week
3. Number of Deaths in one week
4. Tests performed in one week
5. Percent of reported COVID cases that occurred in a cluster.
6. Percent of Emergency department (ED) visits by adults for COVID-like illnesses (CLI) in one week
7. Number of hospital admissions for CLI in one week
8. Percent of ICU bed available for the week

I’ll cover the details of each of these below. For now it’s important to know two things: the state calculates these numbers on a week by week basis and reports on these numbers with a one week lag time. At the time of writing this post the state lists a report for the week of Sunday, August 30 to Saturday, September 5 on their website. These numbers were posted on Friday, September 11. Every Friday the DPH posts the numbers for the prior week. Counties get placed on a high risk warning list orange) on Fridays because of what happened six to twelve days earlier. It’s old but very useful data. Let’s explore why the state would want to work in increments of one week and why and how counties and schools might act upon data that is one week old.

Daily values can vary a lot. In Warren County we’ve gone from a day with no new cases to a day with seven new cases and a very high positive test rate. We’ve had a day with 20-30 tests performed followed by two to three times that number of tests the next day. Day-to-day measures tell you what happened on that day but do not really provide insight on continuing trends until you look a several days in a row. Jumping form day-to-day up to week-to-week just makes sense from a human perspective. It’s how we split time. When looking at seven days worth of data you can choose to average the daily values, this is a seven-day rolling average or a rolling statistic, or you can choose to combine the seven days into a single data point, this is re-sampling your data on weekly time intervals instead of daily time intervals. The state is doing the later. They are re-sampling daily data points on a one week basis. What’s important here is that both approaches meet the goal of providing insight into trends and weekly re-sampling is a perfectly acceptable thing to do in this situation. My positive test rate graphics use a seven day rolling average. You can see how that value is a good indicator of long term trends and jumps much less than the day to day numbers. A good or bad week is a meaningful measure of the pandemic’s current trajectory in the county. It smooths out the day-to-day variability that makes planning things like whether or not it’s a good time to deliver remote or in-person instruction difficult.

Things can and do vary from week to week and this might be one reason the stat lags behind on its reporting. The state’s weekly report tells decision makers what’s been happening and how much risk there was a week ago. They can then work with the local health department (LHD) to asses more recent data and decide if last’s weeks trend is continuing into this week. From there they can consult their crystal ball to see if it’s going to continue into the future. Another reason for the lag time is to verify and validate the data. Testing services and hospitals report to a central database and the state then draws their data from there. They need a day or two of lag to ensure that there aren’t errors in the numbers. Lagging behind on reporting provides stable, accurate historical data and higher degrees of accuracy in the newest data points. Ideally, that data is then combined with on the ground knowledge in the schools and from the LHD to contribute to an informed decision making process. What’s key is that local leaders take the time to contextualize state assessments based on what they know to be happening in their area.

We now turn our attention to the metrics themselves. The state has the problem of providing data to 102 counties and doing so in a way that is uniformly useful. It’s an attempt at one size fits all reporting when not all counties are alike. They come in all shapes and sizes and their citizens respond in different ways to the recommendations and directives of state and local health officials. The challenge for individual counties and schools districts is understanding how these metrics are influenced by and subject to the features of their local population. In my analysis I will share some historical data from Warren County. I’ve computed these statistics using the Warren County Health Department’s daily numbers. The state’s reporting and the counties reporting don’t quite align. The state attributes data to the date the DPH receives it but then posts it publicly the next day. I tend to attribute data to the date that the Health Department posts it which I believe is the day the state makes it public. We get it one day late. I’ve accounted for this but the number still don’t match perfectly with the state’s. They’re not off by much and the analysis I’m going to provide still holds true, but if you’re going to compare my historical data to something on the state’s reporting sites, then keep this discrepancy in mind.

Cases per 100,000 people

There are 102 counties in the state of Illinois. The median population is 23,491 with an average population of 123,027. All counties but Cook county have populations under 1,000,000. There are 82 counties with a population at or under 100,000. This leaves 20 counties with populations above 100,000. Warren county is the 41st smallest county in the state. We are the median county for counties with a population with a population of 100,000 people or less. We’re right in the middle of that pack and on the small side overall. This greatly influences how we should interpret the state’s risk assessment metrics.

The state cannot set risk benchmarks based on the actual number of cases reported in a week. There are over 5 million people in Cook County. If they reported 15 new cases in a week, then they’d have cause to celebrate. If Warren County sees 15 new cases in one week we should be a bit more concerned. To account for differences in population the state, and many other agencies tracking COVID, will normalize case counts to a population of 100,000 people so they can then compare county to county numbers on equal footing and set specific benchmarks for number of cases per 100,000 people. To compute cases per 100000 people you do the following:

\[ \frac{Actual Cases}{County Population} \times 100,000 \]

If your county population is near 100,000, then cases per 100,000 and actual cases are nearly identical. If, like Warren county, your population is noticeably less than 100,000, then a single actual case translates into multiple cases per 100,000. At the other end of the spectrum you have counties with populations well above 100,000. One case in these counties becomes less than one case per 100,000. At the county and regional level, the state sets a target of 50 cases per 100,000 or less. Going above 50 can result in a place on the orange warning list. For the purposes of school safety monitoring, the state has set its benchmarks at 50 and 100 cases per 100,000 per week: below 50 is a sign of minimal community spread, between 50 and 100 is moderate community spread, and above 100 is substantial community spread. There’s a catch though. If your county reports fewer than 10 actual cases, then the state does not compute this value and uses actual cases instead. Whether or not this matters is completely dependent on your population. Let’s look at Warren County and a few other counties in the state. The table below shows you a selection of counties in Illinois, their population, what one and ten actual cases translates to in cases per 100,000, and what the state’s cutoffs translate to in terms of actual cases.

What I want to draw your attention to in the table above is that larger counties have a cleaner separation between the moderate risk cutoff of 50 cases per 100,000 people and the state’s threshold for only computing the statistic when you reach 10 actual cases. There’s some good reason to have this threshold. Look at Henderson county. Each actual case they see is equivalent to 15 cases per 100,000 people. At three or four cases they’d be in the moderate warning zone. At 10 cases they immediately jump to substantial community spread. It might make sense form them to look at fewer than 10 cases based on the circumstances of that week. On the other hand, they will only ever be listed as minimal spread or substantial spread. There’s no in between. It’s up to their local leadership to monitor that gray zone. The state doesn’t flag it.

In Warren county we will cross the 50 cases per 100,000 threshold at 9 actual cases. When this happens, the state reports our cases per 100,000 as 9 and not the computed 53.46. It’s deemed a sign of minimal community spread not moderate community spread. If the next week we see 10 cases, then the state will report that as 59.37 cases per 100k. To the untrained eye, that’s a week-to-week jump of nearly 50 cases. The same can happen on the way down from 10 cases to 9 cases. There’s no smooth transition and it can be a bit of whiplash. These kinds of transitions are not unheard of. You can see that we’ve been around that threshold several times in the past 12 weeks.

Cases per 100,000 people is still one of my favorite statistics because it lets us level the field a bit when comparing different sized populations. Policy makers in counties with populations noticeably under 100,000 need to be aware of the state’s ten case threshold and the effect that can have on the state’s risk assessment. If you’re seeing numbers in and around your gray area, that’s 8-10 in Warren county, then you may want to self-assess the community spread and the circumstances surrounding those cases. You may also want to ensure that the community understands that the state’s evaluation of cases per 100,000 can produce these jumps in numbers and week-to-week variability of their risk assessments.

Test Positivity Rate and Tests Performed

The test positivity rate is an oft cited statistic. It’s computed as follows:

\[ \frac{Total Positive Tests}{Total Tests Administered}\]

At the county and regional level, the state has a target of 8% or less. For school risk assessment they attribute minimal community spread to a rate of 5% or less, moderate spread for 5% to 8%, and substantial spread for more than 8%. The tricky part of this statistic is that it’s dependent on test administration practices in the county which in turn is probably greatly influenced by population. Smaller populations are less likely to generate large numbers of tests and therefore more likely to see bigger swings in the positive rate. What’s more, I’m not aware of any county that is proactively trying to test some fixed number of citizens per week. This means the total tests administered varies from week to week. This is probably why the state will flag you for not testing enough if your positivity rate is above 8%. It’s not based on the actual number of tests but on the positivity rate. The county-level warning system puts counties on warning if the miss 3 or more targets. If you miss the positivity rate target, then you’ve also missed the testing target and you’re very likely to be on the orange warning list the next week.

The table below shows you new cases, total tests, the positivity rate, and the state’s county spread assessment for Warren county over the past 12 weeks.

Warren county has been averaging around 200 test per week over the past 12 weeks. At 200 tests administered, one positive case is a 0.5% difference in positivity rate and the the state’s 5% and 8% thresholds occur at 10 and 16 positive tests respectively. The percents will vary more than you might see in a larger population with more tests administered. On the other hand, at a volume of 200 tests per week, the moderate community spread indicators for cases per 100,000 people and positivity rate aren’t too far apart. This is both appealing and frustrating. It means that signs of community spread are in agreement but also means that the state is more likely to issue warnings for both of them or neither of them with no in between. If, like the state recommends, you’re looking for multiple indicators of community spread to trigger before modifying your current practices, then in places like Warren County, you might find that these two metrics frequently move in tandem. Once again, in smaller populations, closer attention to the stories behind this statistics will be needed. You can’t rely on the state’s reporting alone.

The other interesting, and difficult to navigate for policy makers, feature of the state’s advice to schools is the inclusion of the 5% to 8% warning zone. If you’ve been watching county level reporting, then you’re probably more accustomed to the 8% threshold and not looking for the 5% to 8% zone that Warren county regularly sit in and around. If you’re going to open and close a school based in part on postivity rate, then you need to make sure people are aware that 5% or less is the new target and rates above 5% might merit extra caution and risk mitigation measures even though the state is not flagging the county or region for rates in that range.

New Cases and Youth Cases

The state provides the total number cases and the total number of cases where the infected person is under 20 years old but the warning levels are based not on the actual numbers but the week-to-week percent change. It applies the same rules to both total cases and youth cases so I’ll talk about them all at once. I find their methodology for assessing risk in this case to be more than a little under-specified. What’s clear is that they look at a two week span and see if there were increased case numbers (positive percent change each week) each week. This make sense. If you have 25 cases one week, 35 the week after that, and 40 the week after then case numbers are increase and the virus is certainly spreading. Where things get uncertain is how they evaluate minimal, moderate and substantial spread. The state determines community spread levels based on weekly increases of 5% to 10%, 10% to 20%, and 20% or more. Minimal community spread is when two weeks in a row fall in the first bracket, moderate occurs when they fall in the second bracket, and substantial spread occurs when they both fall in the final 20% or more bracket. They say nothing about how they treat a situation where one week is at 8% and the next at 15%. I’m assuming they call any situation with less than 5% change or where one week sees a decrease in case number minimal community spread. Let’s see how this looks when applied to Warren county’s case numbers. When possible, I applied the state’s risk assessments but added comments for cases where I’m guessing or don’t know how the state would react.

As you would expect in smaller populations, the actual case numbers are relatively small and this means the percent change from week to week can be huge. If you didn’t see that 375% change, then go back and look at July. Larger populations with larger case numbers are not going to see these huge jumps unless something very, very bad is happening. It seems to me that any time we see back to back increases in new cases we should do some investigation and consider raising some concerns if the increase is tied back to situations that might lead to continued increases. In smaller populations, the state assessment just doesn’t seem to handle small cases numbers very well and seems to only be good for flagging obvious substantial community spread. It gets worse when we apply this to youth cases only.

You cannot calculate the percent change of going from zero cases to some number of cases. It makes no sense. Some that did not exist came into being. That’s not a rate change. In larger populations, re-sampling on a week-to-week basis might correct this problem but here we aren’t seeing at least 1 youth case per week. This is good. We do not want kids getting COVID just to make the metrics work. If we’re relying on state warning to keep us on our toes when it comes to youth cases, then we’re likely to miss anything but the extremes. If you go back to the table above, you’ll see that the only time when the state’s risk assessment is clear was during late August. In terms of total number of cases, August was the worst month of the pandemic in Warren County so far. I suspect that small counties like Warren county are going to have to evaluate every new youth case as it comes and largely ignore the state’s risk assessment. If we’re talking about managing risk in the schools, then any week-to-week increase in youth cases merits some investigation. Where are they in school? Is that school in person or remote? What are their close contacts? I just don’t see how we can rely on state-level reporting on this one. It’s extremely all or nothing and doesn’t seem to tell you about a problem until it’s clearly getting out of hand.

Hospital Capacity Metrics: ED visits and Hospital Admissions for CLI and ICU Capacity

I don’t have data on the hospital metrics so this one will be short. If you’re still with me, then you’re probably thankful for that. These three metrics and their associated targets are used for county and regional monitoring. The state uses them to determine if regions and counties should enact efforts to reduce viral spread. They play a role in the school risk assessment process only in that school officials should be aware of the risk and warning levels of their county and might want to enact mitigation measures like remote learning if the county is showing signs of substantial community spread.

The Warren County Health department does not share these statistics and I haven’t been collecting them from the state’s reporting site like I have regional positivity rate. I had not looked too hard at these metrics until I started digging into the DPH school recommendations this past week. I have been assuming that Warren county, like many rural counties, has fairly minimal hospital resources and state tracking measures probably didn’t apply well or at all. We’ve never missed the state targets for these metrics. When I looked at the DPH school metrics site I found some historical data for these metrics. It’s clear there’s something being tracked and I wanted to understand it better.

Emergency department (ED) visits and hospital admissions for Covid-like illness (CLI) seem to account for hospitals within individual counties. The historical data looks different for Warren county and Knox county and if the state were going to pool the data for multiple counties it would do it by EMS region. You probably know what’s coming next. The state will not calculate the ED visit percentage if there are fewer than 5 visits for CLI. Similarly, admissions are not reported if there are fewer than 5 admissions for CLI. Our historical data looks largely all or nothing kind. We saw some non-zero values in August but nothing before then. Once again, waiting on the state warning system to tell us there’s a problem in the hospitals seems to mean waiting for a situation where we probably don’t need the state to tell us there’s a problem. There’s no early warning signs.

The state clearly says on the DPH site that they assess ICU capacity based on the the entire EMS region and not at a county level. I’m not sure we even have an ICU in Warren county. If they’re looking at ICUs from a regional perspective, then I suspect the target they’ve set is only going to be missed if we’re seeing the kind of spread that will put us in lock down. I have not been monitoring this metric closely, but I don’t recall ever seeing a region miss this target since they started reporting it on their website. Again, no early warning signs to be had here.

The Utility of State Metrics in Warren County

The state’s assessment of community spread in Warren county should not be ignored. These metrics, and to some extent the state’s assessment based on these metrics, provide valuable and important insight into what’s happening with the virus in our community. The metrics themselves provide insight into weekly trends and this is vital to risk assessment and planning. You cannot make informed decisions based on day to day numbers alone and totals for the whole pandemic are no longer informative for future planning. Weekly statistics strike a good balance between past and present data. Every county should taking these numbers and assessments seriously but needs to contextualize them with what they know to be happening in their area. If you read the DPH recommendations and guidelines for schools you’ll see that the state agrees. This local contextualization seems especially important in the less populated counties due to the the way the metrics behave and are calculated in the face of smaller case numbers. The good news is that its probably easier to investigate the story behind the numbers when the population is small and the case counts are relatively small. Counties like Warren county can and should be utilizing the state’s data but it’s likely going to take more work and coordination with local resources to use them effectively than it would in a more populated county.

Positivity rate and cases per 100k are worth keeping a very close eye on but require that school leaders know about and keep tabs on the grey zones and the resultant swings in the state’s assessments. The new case and youth case risk assessments done by the state are less helpful but the idea that two consecutive weeks of increases in the numbers merits scrutiny is a good one. In small populations small and modest increases are worth watching. Watch the actual numbers more closely than the assessments. The hospital metrics might be the least useful of the bunch. It might be better to get exact counts from the local health department or the hospitals themselves rather than rely on the state reporting and their state wide assessments.

The most difficult part of using the assessments is that the state isn’t likely to flag warning levels in situations where they might be warranted because the metric variability can easily skip over the moderate risk levels. This problem is compounded by the fact that the hospital utilization targets seem hard to miss unless something really serious is happening. For county-level warnings, we’re pretty much just talking about cases per 100,000 and positivity rates missing their targets. When the positivity rate misses, the testing number target is missed as well and you’re one warning away from the orange list. This might lead to regular trips on and off the orange list when you’re experiencing moderately high case numbers. I could imagine school administrators and local officials wanting to enact mitigation measures in cases where the state says there is minimal spread but the ground truth in town clearly indicates some worrying but moderate viral spread. It’s also tempting to take the state’s side and say that small case loads aren’t worth reporting. I really can’t disagree enough with this sentiment. Small case counts are relative to the population because smaller populations lack the resources of the larger counties. Besides the fact that this virus impacts individuals in a variety of ways and each infection runs the risk of long-term health problems or death, populations the size of Warren county and the schools that serve these communities aren’t exactly rich in human resources.

One of the chief reasons I chose to teach my classes remotely is the reality that if I get sick and did not bounce back quickly the college has nobody to cover for me and my class will run the risk of ending prematurely. Our roster of computer scientists is not exactly deep and there are not qualified adjuncts available in the area. I’m chair of a large, multi-disciplinary department. I lose sleep over what will happen if one of my in-person instructors gets sick and needs to miss several weeks of class or more. It’s not at all clear that we have a way to salvage a class in that situation. We can power through a day or two but a week or more becomes really, really problematic. I’m not under the impression that the local schools have long lines of substitutes waiting in the wings. What do you do when you cannot find someone to teach a class for weeks on end? Is whatever band-aid solution you come up with going to result in an educational experience that is actually worse than a fully remote semester? I suspect this problem weights heavily on local school administrators because it certainly weighs heavily on me. For this same reason I worry about the state’s assessment of hospital utilization and ED visits. Having 3-4 people visit the hospital with a CLI in a single week could be a valuable warning sign here but the state’s reporting does not cover this scenario. They don’t report until at least 5 people show up for such a visit. You have to rely on the local health officials and hospitals sharing this information if they have it.

Any decision making process informed by data and computed statistics needs to understand the source and context of that data and the way in which is was computed. The numbers don’t tell the whole truth but the do help you to see it. The state of Illinois has given us what I believe to be a useful set of metrics for tracking the state of the pandemic. It’s important that we not only know what the numbers mean in general and understand their targets, but that we understand how the circumstances in Warren county shape and mold those values in ways that differ from other counties, especially those with significantly different populations. I started tracking the local data for this very reason. I wanted to make informed decisions for my family. I share that data with others so that they can do the same. As we move into discussion about re-opening or even closing the schools, it’s important that we understand the strengths and limits of the data and how to critically interpret the state’s metrics and assessments in our little corner of Illinois.