Contradictions and Cognitive Dissonance: The (Kevin) Hall Effect
Have you seen the guy on the smart phone video popping up all over the internet saying he has disproven the carbohydrate-insulin hypothesis of obesity? His name is Kevin Hall and he must be experiencing some real cognitive dissonance. Or I certainly am, given the sharp contrast between what his study results actually show and what he’s saying they show on the video. As we all know, cognitive dissonance occurs when we try to hold two opposing views in our mind at the same time, when we experience a disconnect between what we believe and what reality serves us up. And Hall certainly appears to have a disconnect.
Or maybe it’s me. Maybe my confirmation bias is blinding me to the truth. Because when I look at Hall’s results, what we can see of them, they imply something entirely different than what he’s saying. Let’s try to figure this out. Let’s delve into the recent study Hall just presented at the World Obesity Federation meeting in Vancouver and compare it to the video now making the rounds, and his explanation of what it all means. Maybe we can figure out who’s off base here: Hall or me.
While the interview with Hall suggests this is his study, it was actually a collaboration of researchers, including very senior folks at Columbia University and Pennington in Baton Rouge and TRI in Orlando. The study Hall is talking about also seems to have been a pilot study, and the word “pilot” implies that the researchers knew there would be inherent limitations when they started it. So right there I’m wondering why Hall has such confidence about what he thinks he’s proven.
The results are in, but as yet unpublished. Hall just presented the data in poster format at a big international obesity conference. Nothing out of the ordinary until he was approached to do a video interview. Hall then did something extremely bizarre and this is the cognitive dissonance: He spent over ten minutes in this video interview completely misrepresenting his own work, so I’m wondering also if maybe he’s going rogue here. Maybe he’s taking the opportunity to say something publicly that his older and presumably wiser colleagues wouldn’t agree with. I’d like to think so, because what he’s saying makes very little sense. This is why journals don’t like scientists to talk about results before they publish the paper. Without the paper, none of us can tell if the scientist is spinning the data to satisfy some pet agenda. Gullible folks on the internet and in the press can then take it and run, which is what happened here.
A few years ago, I had a conversation with Gary Taubes, one of the founders of NuSI, about who was going to do the research NuSI funded. I assumed they were going to fund Jeff Volek or one of the other prominent low-carb researchers currently working. No, Gary told me, they were going to fund researchers who were not pro low-carb, but who were, if anything, low-carb naysayers. Influential folks who had spent their careers insisting that a calorie is a calorie. That way the results of the studies wouldn’t seem tainted because pretty much everyone knows that researchers can see what they want to see. They are afflicted with the confirmation bias themselves. So if nonbelievers did the study, Gary said, no one would question the results. Kind of like an organization of atheists hiring a bunch of cardinals from the Vatican to do a study they think will demonstrate that God doesn’t exist. Sounds like a good idea, but you can see how it could go wrong.
Feel free to accuse me of indulging in the hindsight bias right now, but at the time Gary told me this, I was a little worried. Just as researchers can see what they want to see, they can also not see what they don’t want to see. So I was concerned that researchers who weren’t at least neutral on the low-carb issue might not come up with entirely accurate results. Think about it. If you spent 15 years of your academic career insisting that a calorie is a calorie, would you like to publish a study with strong data more or less negating what you’ve been saying all your working life? Especially if you had publicly ridiculed that philosophy in years gone by? If you were Kevin Hall and you spent the last year insisting that low-fat diets were the best way to lose weight and that you’d already refuted the carbohydrate-insulin hypothesis, wouldn’t you be tempted to spin your data to suggest that you’re still right?
Like Donald Trump on TV, Hall and his video appear every time I fire up my browser. The widespread response to the video seems to be an ink blot test separating those who believe there is a metabolic advantage with a low-carb or ketogenic diet and those who don’t. It has stoked non-believers into competing over who can throw up the most gloating post the most quickly. The internet is crackling with writings from all the calories-in vs calories-outers (CICO), who believe only calories matter. And, thanks to Hall’s video, they are absolutely topsy turvy over the moon with glee. Bloggers who have spent the last years insisting that they can read and understand the details in research papers better than the rest of us are now taking victory laps based on a ten minute interview and no paper at all. Talk about cognitive dissonance.
I think it would be nice to have the actual data and the final peer-reviewed paper itself before we make claims, which is why I wonder what Hall’s agenda is here. But I have my confirmation bias to deal with also. Based on a lot of experience with patients and deep digging in the medical literature, I think and have said for years that the human body will waste calories when it’s mobilizing fat. I think it mobilizes fat when we don’t eat a lot of carbohydrates and that results in the metabolic advantage we’ve been talking about.
What is the Metabolic Advantage?
Let me digress a bit here. I use the term “metabolic advantage” because it’s one most people are familiar with and how we’ve been discussing it for a long time. But realize that the metabolic advantage is really an effect, not a cause. If your insulin is high, it’s going to trap fat in your fat cells and prevent other cells in your body from oxidizing it. If you drop your insulin levels, you’re going to release this fat from the fat cells and your lean tissue is going to want to take it up and burn it. When that happens, your energy expenditure is going to increase. Mobilizing fat and burning it is the cause. The extra calories burned, the increase in expenditure – what we all think of as the metabolic advantage – is the consequence.
Back in 2007 I suggested that the metabolic advantage amounted to maybe 300 calories a day, although I’m sure it’s going to be different, maybe dramatically different, from person to person. In 2012, David Ludwig’s group at Harvard did a controlled study showing an average metabolic advantage of about 325 calories from a very low carb diet, so I wasn’t much off the mark. At least based on the Ludwig study.
But, according to Hall’s videotaped interview, he seems to think that there’s some science out there suggesting it’s far bigger.
And previous folks have suggested that that should be 400-600 calories a day of extra energy expenditure.
No one I know says the metabolic advantage is 400-600 calories a day, but that’s the least of the issues I have with Hall and this video.
Now with funding from NuSI and the NIH, Hall steps forward saying it doesn’t matter what we all think the metabolic advantage might be, because his carefully-controlled study shows it doesn’t exist, or at least that it’s smaller than he thinks it should be, and therefore it doesn’t exist, or kind of doesn’t exist. It’s a little confusing because, again, what Hall says in the video is contradicted by what Hall and his colleagues write in their abstract and so presumably their paper.
Unless you’ve already seen it, take a few minutes to watch the video below. As you listen to Hall describe his research and explain his data, see if you can figure out what the take home message is. It shouldn’t be difficult. After you’ve watched, skip on down, and we’ll pick it back up.
Okay. So, what did you think? Did he find a metabolic advantage or not?
Just so we’re all on the same page here, another term for a metabolic advantage is an energy expenditure increase on a fixed number of calories. In other words, burning more energy while taking in the same number of calories. Many people, myself included, believe that changing from a high-carb diet to a low-carb diet of the same number of calories allows people to increase the amount of fat they burn, and that manifests itself as this metabolic advantage. Their energy expenditure increases, even while they’re losing weight.
Also, bear in mind that not too many years ago there was a movement in the CICO community to try to get everyone to simply decrease their caloric intake by a mere 100 calories per day. To do so, said the experts, would ultimately get rid of the obesity epidemic.
So if a study suggested that people increased their energy expenditure by 100 calories per day simply by switching to a different diet wouldn’t that be a good thing? Only 100 calories per day. Remember that number because we’ll be coming back to it.
You’ve heard what Hall says; now let’s take a look at what he and his colleagues write for professional consumption in the abstract of the same study he discussed in the video. This is the official abstract published in the program of the World Obesity Federation, International Congress for Obesity (ICO) meeting in May 2016 in Vancouver. You have to click the link, scroll down to the bottom section and about 3/4 of the way down on the left, you’ll see K.D. Hall. Click the title to the right of his name, and you’ll be presented with the abstract.
If you don’t want to go through all that, I’ve pasted it below. The reason I want you to be able to find it yourself is so you won’t think that maybe I’ve altered it to make my point that it bears so little resemblance to what Hall’s saying in the video. See for yourself.
Pretty astonishing, eh?
First, take a look at the title. Authors summarize their papers in the title. This title is pretty clear: “Energy Expenditure Increases Following An Isocaloric Ketogenic Diet in Overweight And Obese Men.”
Remember, an energy expenditure increase on an isocaloric diet = a metabolic advantage.
So, another way of writing the title would be the following: A Metabolic Advantage Follows an Isocaloric Ketogenic Diet in Overweight And Obese Men.
Right off the bat with the title alone, we’ve got cognitive dissonance. How can Hall say the data from this experiment produced no metabolic advantage when the title of his paper says the opposite? Good question.
Let’s take a look at some of the specific data he presents in his poster. Along the way, I’ll air some gripes about how the data were (or weren’t) presented.
The Poster and the Poseur
Here is how the experiment was done. The researchers confined 17 obese subjects to a metabolic ward for 8 weeks. During the first four weeks, the subjects consumed a baseline high carb diet, then switched to a low-carb/ketogenic diet for the last four weeks.
What was the macronutrient composition of the two diets? We can’t tell from what Kevin says.
We don’t know what the actual macronutrient composition of the lead in diet was because it isn’t revealed in the poster or in the abstract, so all we can do is infer from what Hall says in the interview, which is:
…so here is the study design, basically. Two months onsite metabolic ward. Basically the first period is a baseline diet, high carbohydrate diet, 25% of total calories coming from sugar and every week they spent two days inside a metabolic chamber. And basically what we do is over the first period of time we adjust the number of calories to match what they were burning inside the chamber so that we put them in energy balance inside the chamber and then we clamp the calories for the rest of the period of time and switch them to an 80% fat 5% carbohydrate diet keeping protein clamped all the way along.
I navigated through the NuSI website until I found the link to the info for the lead in diet, which was 50 percent carbs, 15 percent protein and 35 percent fat. Then over the last four weeks the subjects ate a diet made of 80 percent fat, 15 percent protein, and 5 percent carbs.
Each week, each subject spent two days inside a metabolic chamber. A metabolic chamber is a very expensive small room in which precise measurements can be made of energy expenditure. The idea in Hall’s study was that they were going to measure people’s energy expenditure in the metabolic chamber on this run-in diet and that would tell them how many calories they were expending. Then they could feed them that many calories and that would be a weight maintenance diet.
But immediately Hall tells us there was something wrong with using the metabolic chamber. As Hall says, much to their surprise, they found out that the energy expenditure of subjects in the chamber is unnaturally low. So when the researchers fed the subjects the number of calories they thought they needed as revealed by the metabolic chamber they were dramatically underfeeding them. There was quite a discrepancy. Says Hall:
One of the interesting things was that despite the fact that we were trying to put them in energy balance in the chamber and we were actually able to do that. Um, outside the chamber they actually burned more calories. Not too surprising in retrospect but we actually were surprised by the magnitude of how many more calories they were burning outside the chambers. Upwards of 500 calories a day more outside of the chamber than inside the chamber. So what that meant was that we were feeding them more or less in balance in the chamber but we were underfeeding them outside the chamber.
When Hall says, “not too surprising in retrospect,” he might as well be saying “oops!” And what he also means when he says that is his own colleague and last listed author on this study, Eric Ravussin, published this finding 25 years ago. This makes me wonder why they didn’t plan for it in advance. Oops again.
So, each subject was spending two days per week in the chamber, leaving five days per week spent outside the chamber. According to Hall, during these five days outside the chamber, the subjects were being fed about 500 fewer calories than required for their weight maintenance and energy balance. Aside from the fact that this makes it a very sloppy study — the oops! factor — and makes me wonder again why Hall is so smug and confident, there’s another interesting issue it brings up that I’ll address in a bit.
The results for the various parameters measured during the 8 week study are shown in graphs displayed on the poster. It aggravates me that only the last two weeks of the first four week baseline study are displayed. In my view, it would have been just as easy to show the entire thing. As it is, it implies that the first two weeks mirrored the last two weeks of the baseline, therefore didn’t need showing. But we don’t know that from what we can see.
Below is the poster for the study (click to enlarge).
Let’s look at the results in some of the sections.
Fat loss over time…
I don’t know about you, but it’s pretty clear to me that weight loss accelerated once subjects were switched over to the ketogenic diet. The fat mass, as portrayed by the broken line, seems to level out a bit as the subjects start on the ketogenic diet, but then appears to resume its same rate of decline for the last two weeks. (Remember we don’t know what it was the first two weeks of the run-in period, so we have to take Hall’s word for it and clearly that’s a questionable thing to do.)
Okay, so what happened… The blue period of time, this is the last 15 days of the baseline run-in high carbohydrate diet so they’re losing weight slightly. It’s a roughly 300 calorie a day negative energy balance over all. They’re losing fat over this period of time. We switch them to the low carbohydrate diet, they lose a good chunk of weight about 1.6 kilos right away. Interestingly, and I think most importantly for the carbohydrate insulin hypothesis is that the fat mass [sic] slowed.
I’m pretty sure that Hall meant to say the fat loss slowed. It did, but only briefly. Then it picked back up. And it slowed during the period when one diet transitioned to the other. I’m a little confused about the calories now. At first Hall said there was a 500 calorie a day negative balance, and now he says it’s a 300 calorie daily deficit. Did he misspeak the first time or this time or was the 500 calorie deficit during the baseline diet and the 300 calorie deficit during the last four weeks? We don’t know.
One of the things that I find very curious here is that Hall never mentions in the video whether they measured ketone losses in breath and urine and fat loss in feces. And I can’t find any discussion of it on the poster. I sure hope the paper will have these data. If you’re going to claim to to have done any kind of definitive experiment on how energy expenditure and fat loss change between two different diets, then it seems you have to make every effort to track energy and fat.
These can be expended — and that will show up as changes in energy in the calorimeter — or they can be excreted and that won’t show up in the calorimeter, but is still energy loss just the same. Hall is talking about measuring the expenditure, but not the excretion. Calories are lost as ketones in breath and urine; fat is lost in feces. Did they measure those? And if they didn’t measure it, how can Hall make any claims about what’s happening here, even these contradictory ones? I hate to sound like I’m nitpicking, but isn’t that one of the things that science is all about — getting the details right. In the ketogenic diet, they’re feeding these people a couple thousand calories of fat every day and they should be mobilizing the fat they’ve stored. We can expect some of this to show up in feces. It would be nice to know. And if it doesn’t, that would certainly help make Hall’s point. I wonder why he doesn’t show it.
And the changes in Insulin…
C-peptide levels are a measure of insulin production. The beta cells in the pancreas produce proinsulin, which splits into a molecule of C-peptide and a molecule of insulin. The C-peptide remains in the circulation and is a proxy for insulin production because one molecule of C-peptide is made for each molecule of insulin.
The chart above demonstrates that switching from a high carb diet to a ketogenic diet truly does drop insulin levels quickly.
If the carbohydrate-insulin hypothesis is correct, Hall claims, this drop in insulin should result in a big increase in energy expenditure. Why? Because, well, let him tell it:
…basically we were really interested in the idea of trying to test this carbohydrate insulin hypothesis for obesity which posits that basically obesity is caused primarily because of high insulin levels which are driving fat into fat cells starving the rest of the body and thereby basically causing increased hunger and decreased energy expenditure.
So one of the logical consequences of that model of obesity, if it’s correct, is that if I take people who are overweight and have Class 1 obesity, if I then swap out, keep the calories the same, but swap out carbohydrates for fat. In particular if you crank down the carbohydrates really low to 5% of total calories and increase fat to 80% of total calories, that you would reverse some of these things.
And if I clamp, basically, the number of calories then I’m not going to be able to test the effects of hunger but I will be able to test the effects on energy expenditure.
So what happened to energy expenditure?
Okay, so we’re looking for an increase in energy expenditure if the carbohydrate-insulin hypothesis is valid because we’re not really testing that hypothesis directly, despite what Hall says (and I’ll get to that in a moment, too). So let’s look at the next two graphs.
The top one shows the overall energy expenditure as measured in the metabolic chamber and the next one down shows energy expenditure measured while the subjects were sleeping. The sleeping energy expenditure is a measurement of fat burning during fasting. If insulin levels are high, holding fat in the fat cells, then you would expect sleeping energy expenditure to be low, as it was in the blue part of the chart, the last two weeks of the high-carb diet. On the ketogenic diet, insulin falls like a rock, allowing fat to flow easily from the fat cells to meet the body’s needs. As this fat is burned during sleeping, you would expect to see an increase in energy expenditure during the ketogenic diet, which is exactly what you see. Again, that’s a nice metabolic advantage.
Now what are you going to believe? What Kevin Hall tells you the study showed, or your own lyin’ eyes?
What does it all mean?
Nullifying the Facts?
An important missing point to keep in mind in Hall’s discussion is the idea of the null hypothesis.
Every hypothesis test requires the analyst to state a null hypothesis and an alternative hypothesis. The hypotheses are stated in such a way that they are mutually exclusive. That is, if one is true, the other must be false; and vice versa.
If you google “hypothesis testing”, you’ll find dozens of sites like the one linked above and this one — Wikipedia’s — that explain what these experiments are doing, which is they test the null hypothesis. You can also ask any biostatistician, if you happen to have one in your neighborhood. Hall says from the get-go — both in the video and in the poster — that he and his colleagues are testing the carbohydrate/insulin hypothesis. But that’s not the null hypothesis. (If obesity researchers had been doing their job right for the past 50 years, it would be. But that’s a topic for another post.) The carbohydrate/insulin hypothesis is the alternative. In Hall’s study the null hypothesis is that obesity is an energy balance problem. It predicts that as far as fat storage and energy expenditure are concerned, a calorie is a calorie. It doesn’t matter what the diet composition is — all that matters is calories. Change the diet composition, according to this null hypothesis, and it makes no difference in fat storage or energy expenditure.
The alternative hypothesis is that obesity is a hormonal disorder (too much insulin driving fat storage). If it’s correct, then changes in macronutrient composition designed to reduce insulin levels would result in decreased fat storage and increased energy expenditure even if caloric intake were kept the same.
The main thing Kevin Hall’s study is doing, the primary thing, is testing the null hypothesis, and it’s kind of mystifying to me why he presents it any other way. If he refutes the null hypothesis — which I think would be pretty damn interesting, since that hypothesis is the basis of all our public health and research and medical thinking on obesity — then he can say whether the result goes in the direction he would expect from the alternative hypothesis or not. In this case, that would mean an increase in energy expenditure when his subjects switched to the low-carb diet.
If the null hypothesis were correct, we should find no statistically significant change in energy expenditure and/or fat accumulation. If we find either, then the null hypothesis is rejected and a calorie isn’t simply a calorie.
Since there is no specific data on fat accumulation in the poster, we can’t tell the difference there. All we’ve got to go by are the graphs shown, which are difficult to interpret, and what Hall tells us. The energy expenditure is a different story. Those figures are published in the abstract.
The ketogenic diet coincided with a rapid and persistent decrease in respiratory quotient (mean±SE; 0.111±0.003, p<0.0001) along with increased total energy expenditure (+57±13 kcal/d, p=0.0004) and sleeping energy expenditure (+89±14 kcal/d, p<0.0001) measured by metabolic chambers. Energy expenditure measured by doubly labeled water increased by 151±63 kcal/d (p=0.03).
The increased energy expenditure numbers from the metabolic chamber are highly statistically significant, meaning there is very little probability these findings occurred by chance. The energy expenditure measured by doubly labeled water is more than twice as high and still statistically significant. And the interesting thing about that, remember, is that Hall told us ‘something weird happens in the chambers’ because the subjects expended 500 calories a day less inside the chamber than out. Remember the oops!? Hall tries to make light of that, but that’s the equivalent of running half a dozen miles a day in the ward, while not moving at all in the chamber. Now I don’t know about you, but as a doctor I’ve been in plenty of hospital wards and I’ve never seen patients running laps in the corridors. So something weird is happening in the chamber. And it suggests that the measure of energy expenditure outside the chamber, in the ward itself, is the more meaningful one.
It isn’t really shown in the poster and Hall only briefly alludes to it in his interview, but there it is prominently stated in the abstract. The energy expenditure measured by doubly labeled water includes the five days each week the subjects are on the wards. It’s not just the two unnatural days in the metabolic chamber. And it increased by 150 kcal/day. That’s a hell of a lot of energy considering all the things we don’t know. Enough to roll back the obesity epidemic and bring people’s weights to 19th Century levels if it was maintained day in and day out.
The fact that the energy expenditure is increased at such highly significant rates rejects the idea that a calorie is a calorie — the null hypothesis. Hall can argue this till he’s blue in the face and mug condescendingly into the camera all he wants, but his own abstract tells the story.
Both the chamber and the doubly labeled water measurements show a definite increase in energy expenditure and they both reject the null hypothesis while going in the direction we’d expect from the alternative one. Based on the data presented (and the data that may not have been measured), the study appears to be poorly done. (Oops!) But what they report seems clearly to contradict Hall’s spin.
If you want the final word (so far) on this pilot study, just look at the last sentence in the abstract. Remember how I said I would come back to the 100 kcal/day? That’s the amount, you recall, that the CICOers said would solve the obesity epidemic. We’ll there it is.
Therefore, an isocaloric ketogenic diet was associated with increased energy expenditure of ~100 kcal/d.
There’s one last point I want to make and it’s about the use of the words “associated with” and “coincided” in the abstract: “The Ketogenic diet coincided with….” It doesn’t say the diet caused what they saw. It only says it coincided. I think that’s because the study wasn’t a randomized trial. Maybe the changes they saw would have happened on any diet after the first four weeks. So statisticians would say they can’t “infer causality” and older and perhaps wiser scientists than Hall would know that, which may be why what the abstract says and what Hall says in his video are so different. Hall is saying what he’d like to believe he’d done. The abstract says what he and his colleagues have actually done. As I said, they seem to be two very different things. I can’t wait to read the actual paper when it gets into print to see what it really says, and whether this study is really as bad as Hall suggests in the interview.