AUDIO START: [0:00:00]
Good afternoon, and welcome to Gray's webinar on Understanding Student Demand. This is Bob Atkins, I'm CEO here at Gray. And before I get started, let's take care of a few housekeeping items. During the course of the presentation, you may well have questions, feel free to enter those in the chat screen to the left on your monitor, and we'll take those probably at the end, but time allowing, we'll take them as we go along, as well. So, feel free to enter them, and we'll answer them as we can. After the webinar, downloads will be available for you, so you don’t have to take too many notes on this. The other thing I want to mention is that this is the first of a six-part series of what we're calling Master Classes, which may be a bit of an aspirational name on your part, but that's what we hope we're giving, is a pretty advanced lesson on how to do analytics around programs.
This first session as we have discussed is Understanding Student Demand. The second will be on Determining Employer Needs, also a key part of program analysis. Session #3 will evaluate Program Intensity, the competitive intensity of a program. In #4, we will look at How To Calculate Program Econonics.
In #5, we will look at How To Inform And Engage The Constituencies. Data isn't enough in these processes, you have to involving the right people at the right time, or you really don’t make much progress at all.
And finally, Part #6, we'll talk about how to integrate this kind of data into your everyday life, if you will, into the regular review of programs and sustain a data informed programmatic decision making process over time. We're going to do those webinars once a month for the next five months, and at the end of the presentation, I'll share with you the dates for all those; when you registered, you probably saw them, as well.
So, with that, let's get started looking at Student demand.
Goals for Today's Session
I'll walk you through a few key areas. First, what's the problem here? What's the challenge that you all face? Second, we'll give you a simple framework that you can use to understand the data that you have to collect on programs in order to do a good evaluation. We'll walk through each of those data sources and talk about what their uses are, and their limitations, so you understand how to, if you will, use the data safely. But to do that, you have to decide what data you're pulling, so we'll talk about how you define your market, then finally we'll share an integrated approach that pulls all of that together.
Let's start off talking a little bit about the challenge you face when you get into this business of program evaluation.
The first is that there is just a bazillion program options out there. IPEDS keeps track of about 1,400 academic programs in what are called CIP Codes, Classification of Instructional Programs, and somebody, somewhere is going to want to look at darn near every one of those at some point. So there's just a proliferation of programs and if you have to analyze them all, it becomes very, very time consuming.
The second problem is that everyone has their own favorite data source. So while it might be relatively simple to go out, for example, and grab data from the Bureau of Labor Statistics and analyze it to understand the job options out there, somebody is going to bring up, when you do that, that there is data on job postings that you can get from Burning Glass and other sources, and you should look at that. And then they're going to mention, oh well, we should be looking at IPEDS too, and it goes on and on.
You've got demographic you might want to keep track of, financial data on your old program, and so forth, and as an individual analyst, it can be a bit overwhelming. These are all very large datasets, out of which you generally only need a small portion that is for your geography and actually cutting it that way is not trivial, and then aligning the databases is also challenging and trying to show the linkage, for example, between job postings and academic programs. So all of that can make it cumbersome and difficult, and quite error prone, for an individual analyst, and we’ll come to that when we talk about the employment, it really can throw you, you can get the answer very significantly wrong if you're not careful in how you align programs and jobs.
Another issue here is that you've just got a whole bunch of stakeholders in this process, and it's critical to involve them, but also quite a challenge, given how many there are, representing every aspect to the university, from admissions to buildings and grounds, when you're thinking about capacities, and of course your faculty and deans. So, corralling that group is quite a challenge, and making sure that whatever data that they believe is important about, for example, student demand, is either included, or you've thought through why it has not been included.
A Framework for Evaluating Markets for Programs
One thing I want to say at the outset here, is we have a perspective on a well-run process for this and it starts with the premise that visions and opinions are not data. Crystal balls, I think we're now at a stage where there are better ways to do forecasting than that, and there are better fact bases on which to form your decisions. So, there is really no need to rely on somebody's personal opinion about what a good program is. The market for programs is pretty heavily saturated and intensely competitive. So the odds of your guessing right about a new program are not what they once were, and frankly when you look at the number of small programs at most institutions, there is considerable evidence that our ability to guess is never all that good.
As an example, 48% of all programs offered in the United States have less than 10 completions a year, and that 48% of programs only produces 7% of all graduates. So, we have had a lot of thoughts about what a good program would be, and the result has been many, many, many small programs that are probably not at critical mass, and may actually be losing money. Now, that's not to say that a small program is necessarily bad. Some are mission critical, some are very important to a discipline, but nonetheless, as I said, the odds of success if you're guessing and relying on opinion or a trustee's personal point of view on which program you should offer, that's a risky way to go and it's just no longer necessary.
So, let's talk about the framework for evaluating the student demand in a market, well, actually for evaluating an overall market, and then we'll talk about student demands in particular.
Market Evaluation Framework Four Factors
So, we see four key dimensions to evaluating a program market. The first is student demand, The second is employment. Are there jobs for graduates, and much will I get paid? The third is has that student demand and employment opportunity already been addressed by the competition? And how competitively intense is this space? And the last question is degree fit; what level of degree do you have to have in order to succeed in this marketplace, both from an educational standpoint and for your graduate to succeed in the workforce?
So, today we're going to focus in on student demand in particular and talk about the data sources, the uses of the data, and the limitations to the data for student demand.
Data Sources, Uses, and Limitations
So, fundamentally, when you're trying to understand more than one program at a time, things like student surveys become somewhat irrelevant. It's just very difficult to get enough people to respond across the 1,400 programs you have, to really understand what the demand might be for an individual program. But, fortunately in today's world there are other ways people express their interest in a program, and there are three that I would suggest you keep track of.
First is Google, and Google is very current, you can find data on almost all programs in there, in terms of search volumes, and it's a little tricky to get out, and we can talk a little bit about how to do that, but certainly when you are looking at 1 to 10 programs, you can't actually do that work by hand.
Second is GrayReports. This is data we keep, it's on inquiries, we'll talk about that in more depth. Like Google, it is very current, we update it once a quarter, we actually get it every month, and it's a pretty extensive dataset. We get about 700,000 new inquiries a month, and since we've started collecting this, we now have a database of over 50 million inquiries going back in time, so a pretty good sense of the change in demand, as well as the demand by program.
And finally IPEDS. And of these three datasets, IPEDS is the most complete, however, it's also the oldest, and that's a bit of a challenge. Let's assume for a moment that you're a four-year college and the most recent IPEDS is for 2017, and that's true. Well, that means that people made their decision about their majors at least three years before 2017 if they graduated in 2017, and if their graduating as is more normal these days at 150% of normal time, then they made that decision as far ago as maybe 2012, about what major they would be in, and then they probably changed their mind once or twice, as well, so that IPEDS data is really looking at marketplace decisions about programs that were made back in the early part of the decade, so it's getting pretty stale at this point.
And in the nature of IPEDS, it will always be that way, because a decision about what program to take is made years before you graduate, and the only information at IPEDS by program is in fact completion, so they can't see that student until they're graduating, and then there is just the normal lag between when IPEDS collected that data and when it gets published, and in between publications. So, you're dealing with something that's about five years old. So if you're relaying on IPEDS for market demand data on its own, you're really playing a significant risk. It's good data if you want to know roughly how big a program, is it a large or small one, it's good for long-term trends, it is not good for understanding what students want right now.
Google: Trends and Keyword Search Volumes
Let's dive into Google a little bit. Everybody has used this, the question is how do you use it to understand the demand for an academic program? There are a couple of different elements in Google that you can look at. The first is Google Trends, and this is actually a really fun database to look at in general, you can see what the hottest searches are in the internet right now, and when you get done with that, you can actually drill in and look at the programs you're interested in. You can enter those, you may have two or three different search for them, and you can see what the relative volume is for those, one search term compared to another, and you can also see the trend.
And one of the problems in the higher education market right now, if you look at this data, this is a little bit old, but you can see that there has been a downward trend in Google searches for MBAs for some time, and that has continued. And obviously, that puts a little bit of pressure on what is the largest graduate program in the US. So, that's been an underlying demand characteristic. But one thing you will notice here if you look closely, is that this data is not actually the number of searches, it's an index of search volume. So, it takes the highest point on this chart, calls it 1.0, you can see it just a little to the right of the first bump there in the data, so you go all the way to the left on the chart and come a little bit to the right, and you can see the highest point, that's 1.0, or 100%, and then every other data point on the chart is the search volume to that 1.0. So, it's great for understanding what the trend is, it's useless in terms of understanding total search volume.
However, there is a way to work around that, and that is take a search term where you know your volume, and enter that, and then compare to whatever program you're looking for, and then you can actually pull the index data off, and you will be able to see how much bigger or smaller the program you're looking for is, compared to a program that you know. So, if the index was 100 for the program you're looking for, and 50 for the program you have and that you understand, you'd know the program you're considering is twice as large in terms of Google search volumes as the one you have. So, that's the way to be able to use the trend data and get at the volume of searches, as well as the trend in searches. But to do that you have to have your own internal benchmark program where you know the volumes so you can do that comparison.
There is another problem, however, with the Google trend data, which is it taps out geographically, and frankly the trend for the US is interesting, but for most colleges and universities, their market is not the US. Even if they're online typically, they're going to be serving an area 50 to 100 miles around their campus. And once you get into those narrower geographic areas, the data in Google often taps out. When you want to know searches for medical assisting in Topeka, there may not be sufficient data in Google to show that to you, so you're going to get a chart like this, which is not all that helpful.
Now, there is a solution to this, which is to go into Google AdWords, and here you get more particular data that's specifically about individual search terms. So for example, here we're looking at registered nursing, and we've used the term, "registered nurse" and we can actually see the search volume for "registered nurse" down there in the area I circled in red, and we can see it's gone up from 480 to 590 from one year to the next, so it's got good growth. This also breaks that search volume down by geography, in this case we've used Massachusetts, where I'm based, and we're looking at a beautiful snowy day today, and you can see where that search volume is coming from to understand whether your market is relatively more attractive, and also where you might consider advertising, especially if you're on line, other markets where there may be significant volume for that program.
A couple caveats here, the most important is that the results that you're going to get are a function of the keywords you use. And so the first challenge is that the keywords that you may be looking for is incomplete. And then you just have to be very sure that it's not confounded with some other reason for searching. Let me take an example with "registered nurse." In this case, somebody may be looking for a registered nursing program, or they could be looking for a registered nurse, they need help, they want to hire a nurse, in which case you're actually looking at two different phenomena here, only one of which pertains to your course or program you're considering.
So, be very careful with the search terms, and make sure that they're unambiguous and that you're not accidentally including data about phenomena that you don’t want confused with your program. And here you can actually see a time series on that, which is very helpful to understand if the search volumes are going up, and if they're going up consistently over time. This chart will also show you seasonality which does vary a bit from program to program, so you can understand for example when might be most advantageous to launch, or if the current monthly volume is just a blip, as opposed to a persistent trend.
Another issue with Google that you've got to watch out for, when they start to give you this data, they're not really giving you their raw data. They're manipulating, manipulating is not the right word, they clean up that data, and one of the things they do is they smooth it. And you can see here, in this particular month, March and April, the search volume that came to us from Google was exactly the same, well, that's never happened in the real world. What they've done is gone in here and smoothed out the data, and you just have to be aware that from month to month the volume changes may not be completely real. They are a combination of real data and Google's effort to communicate that data in a way that it's statistically reliable, for which sometimes they smooth. So, that's Google. It's a great source, it's largely free. There is one challenged here that I haven't mentioned, and that is if you really want to look at a lot of programs or a lot of markets, it's very time consuming to get this data.
Now, many of you may know some Python, and it's not very difficult to go in and write some code, or have one of your students who has taken that course come in and help you out and write some code, that will repeat that search over and over again, changing the geography. I can tell you, we did try that, it doesn't work. As soon as Google notices that it's a machine doing it, it will shut down your searches, not permanently, but it will shut you down. So, you can't automate it, and as I say, if you're looking at more than one market or more than one program, gathering this data can be very tedious. We found in API that we can use the full data out of Google in bulk, that they do allow, so we can get at it. We actually track Google Search volumes for the top 200 programs for some 25 keywords each, in each of 3,000 counties in the US. So we have a lot of granularity, but you can't really replicate that without finding the API and writing a bunch of code to talk to Google and pull that data out. But you can go in and do one or two programs at a time without any trouble.
Now, let’s talk about a different dataset, Student Inquiries. The beauty of this dataset is it's very current, it tells you what program the student is looking for, whether they wanted to take that program online on on-ground, where that student is physically located, and a few other attributes, veteran status, and years since high school, so if you're aiming for the older adult, you can actually see that data if you participate in these inquiry streams. Now many of you may ask what exactly a Student Inquiry is, and it's actually a really good question. It's not actually somebody calling up your school, it's people who are online and are searching, and they get picked up in one of several places. In some cases these are coming directly from people who hit a website for a school; as you can see here the example would be UNC, a fictional example.
They could also be an aggregator, and these you'll see if you went online right now and searched on MBA, there will be sites that come up and say what can an MBA do for you, as shown here on the bottom left, and they'll show you any number of different MBA programs that you might be interested, and also happen to be their sponsors -- this is an advertising medium, after all. And then they will capture information about that person, including their name, their address, their phone number, their email, their age, and so forth, and we get a subset of that data sent to us, in particular their address so we can geo-code it, and the program of interest. The other people who submit these inquiries to us are agencies, and they're doing a similar thing out on the web where they're advertising for programs and then selling those leads back to academic institutions. So, that's where inquiry data comes from.
The only place that I know of that you can get it in bulk is from us, in something we call GrayReports, and we aggregate this data by academic program, we also take it all the down to census track level, so we can create any geographic market you want. And we do keep track of this month over month, by program, and actually if you all aren't webinarred out by the end of the day, we do a webinar every month to share the highlights from this data set and from Google about which programs are hot, and which are not, and you're welcome to join us.
You can also call us, and we can share this kind of data with you, we have it on a subscription basis. All of that can be geo-coded, so you can see which areas of country have relatively more inquiries than others. We can do that for the Google data, as well. Where it gets really interesting for a program, especially if you've got any question about where to market a program or where to locate a campus facility, you can get this all the way to the census track level and physically see where on that map people are looking for a school, and more than a school, a specific program within that.
So you can see how many people in St. Petersburg, or how frequently people in St. Petersburg are inquiring a medical assisting program or a history degree, in the unlikely event that you've got any other prospective history majors in Tampa, like me. And this data is really, really helpful when you're trying to figure out where to advertise, and obviously you're thinking about where to locate a facility, you can literally see where the demand for that program might come from.
Completions: Program Size and Long-Term Trends
Now, let's turn to IPEDS, and for most of you, this is going to be the most familiar database. This is the Integrated Postsecondary Education Data System. The data is kept by the Department of Education and all Title IV schools, that is, all schools that accept Title IV funding, have to submit data to IPEDS every year. The most current year in IPEDS is Spring 2018, we haven't got the 2018 data yet, we probably will get that in the next few months.
Inside here, what you can see at the program specific level is completions. As I mentioned, they don’t keep track of students by program, because students are changing their majors. So, you can see completions, and this allows you to understand who you're competing against in your local market, how big they are, and how fast they've been growing. So, it's a wonderful source for that. It's also a reasonable way to understand the degree to which your local market may be saturated for an individual program.
Now, the last part, though, in order to understand saturation, you actually have to be able to compare your local market to other markets, so keep that in mind. And what we do at Gray is we actually keep track of the top 100 markets in the United States, and we look at completions per capita as a measure of saturation. Without that comparator, it's very difficult to know if 100 completions in your local market is a lot or a little. And that's really true for most of the data that you will collect. I was literally on the phone with somebody earlier today who asked us to do an individual program analysis and he loved the analysis, but he said, "At the end of the day I found having looked at one program was really a challenge, because when people asked me if that was a good program, or not, I couldn’t actually tell them because I didn't have anything to compare it to." And with this data, you will find that could be an issue, and keep it in mind with almost all the raw data you might collect on student demand, if you don’t have comparators, it's very difficult to interpret.
So, for example, in this particular case, we found that there were 100,000 inquiries for a given program in a certain market in the US and I awkwardly had a client once look at me, when we were in the very first days about 10 years ago, doing this, and say, "Well, is that a lot?" And I honestly couldn’t tell him whether that was a lot compared to anything else. So, all this raw data without context is extremely difficult to interpret.
So, how do you do it? Well, in our view what you need to do is put your data into desk files for your market. So you can understand one program compared to what we do is all other programs, over 1,000 IPEDS programs, and for example, that 100,000 number I was talking about, if you look in the left hand column here where the past inquiries are, you see the red arrow, 100,000 is at the 99th percentile, more or less. So, that's a very large program, it's the top 1% of all programs in this market.
Now, you may not want to go out and collect data on every program in your market. What you can do is collect data for your own programs, so if you look at your current programs or a new program, you can see how the data compare for your current program to the one you're considering, and that will give you a sense of its relative size. And we suggest you do that on each metric you collect, so that you have a good sense of the relative value of the numbers that you're going to get back.
Now, once you do that, you've got to give us non-quant guys a break. I mentioned I'm a History major, and we started out with all these charts with just masses of numbers of them, and I found it daunting, so what we decided to do is color code them. What we do is we define the percentiles for each color, and generally we use colors that have meaning. So, for example, green is typically an indicator of good, and red is typically basically, so we run a green to red scale for this, where the dark green means that individual program is in the 90th percentile or better. And all of a sudden you can start to glance at a page full of numbers and get a very quick sense of whether it's mostly green or mostly red, and with that, understanding if it's potentially attractive.
The second part is being able to look at the individual numbers and understand how those stack up, and this case we can see, for example, that inquiries are in the top 10%, the change is in the top 10%, and completions are in the top 10%. So, when my inquiry data and my completion data agree, that makes me pretty comfortable, that this is at least a very large program, one of the top 10% of all programs in this market. You will also notice that there is a disparity in the growth numbers. The inquiry data is growing, but the completions are shrinking. And that is often the case, when you get two, and we keep actually three sources on student demand, because we have Google, as well, the data may not agree, and that's why we collect more than one source, because it's often difficult to know exactly what's going on and you need to know if there is uncertainty around the numbers.
But in this case I would say it's not completely clear whether this is growing. It's clear it's large, because both my size metrics are very high, but the growth rate is in conflict between the completion data and the inquiry data, so I worry that I don’t know exactly how fast this market is growing. If you go back to the sources, though, and think about what each of them is, inquiries is very current, so that 9.4% is probably reflecting what's going on in the market now. Those completions are in large part a function of how many people signed up for this particular program two to five years ago. So it's very possible that this program was in decline four or five years ago, fewer people were starting it, and that's now coming through as completions, and a decline in completions.
So, what I would tend to say here is, it's possible, not certain, that a turnaround has taken place and a program that was in decline has now resumed its growth. The other thing I would take away is that this is uncertain. And if I were evaluating this particular program, I would want to go look at other indicators to make sure that I could be comfortable, whether it's growing or shrinking.
To do that, however, I need to decide what market I'm actually evaluating. What we would suggest is to map your students, this is fairly easy today in any number of applications. It used to be very difficult, but you need to get addresses in these, we geo-code those to census track, and then we use charts like this to be able to visualize where students are coming from, and that's the first step in deciding how to define your market.
And as I look at this, if you look at the Chicago Loop there, it looks to me like we could pick up most of our students with a radius that extended out somewhat west, north, and south, but there is not a lot of incremental students if we just go due west on this chart. We do have another issue down in River Oaks, that's the second campus, and you could imagine that you might want to define a second market down there, as well.
Once you've got all that geo-coded, we then calculate the distance from those students to the campus, and that allows us to create charts like this, so we can see where the curve flattens out, that is, as we move further and further from the campus, we pick up a higher percentage of students, but that rate of growth slows down. So, between 5 and 10 miles the total number of students doubles, as we get out to 15, we're up to 51% of all students, by the time we get to 30 miles, we're up to 80%, and each 5-mile increment after that doesn't pick up very many incremental students. So, I'd probably pick 30 miles as a reasonable definition of the primary market for this particular program.
And then we can go in, once we've defined that market, and pull data for that particular market. So then we pull the Google data, we'll pull our inquiry data, and we'll pull IPEDS data for all the folks who are competing in that market and all the students who have raised their hand in that market to express interest in a program.
Now, that's not easy stuff to do in practice. We've actually built a system to do this, and we pull all that together in the system, so we can then tell you what the data is, not just for one program, but actually for all 1,400 programs in IPEDS.
So, let me summarize. When you're evaluating a market, we like to start with student demand, but by itself, that is incomplete. To understand the market you need to do the work we described today, but you also need to understand whether there are jobs, what the overall employment atmosphere is, how competitively intense that market is, and what degree level to offer. And over the next few weeks, we'll share with you how to do that, as well as how to run a good process.
So, let me review the times if you would like to join us. We'll be taking about Employer Needs and to evaluate them on March 7, all of these will be at 2:00 PM, and these are all Thursday afternoons. We will evaluate Competitive Intensity on April 11. We'll talk about how to Calculate Program Economics on May 16, how to Inform and Engage Constituencies on June 13, and how to Integrate and Sustain this process over time on July 18.
I hope you all have enjoyed today's webinar, and that you will be joining us in the future. Now, what I'm going to do is turn it over for questions, and also remember that you can download the materials from our website after the webinar.
There are no questions this afternoon, so I'm going to hope that everybody understood everything and enjoyed the webinar. We welcome your feedback. Please feel free to shoot me an email with any thoughts you have on the material or how we might have presented it better. You can reach me at firstname.lastname@example.org. Thank you very much.
AUDIO END: [0:32:09]