Featured topic and speakers
Kirsten Bibbins-Domingo, PhD, MD, MAS, editor-in-chief of JAMA® and JAMA Network™, and Stephen Parodi, MD, executive vice president of The Permanente Federation, discuss how medical journals and research are evolving to build trust, meet the needs of physicians and leverage AI. Learn more in the second part of a two-part episode of the Moving Medicine podcast.
Speakers
- Kirsten Bibbins-Domingo, PhD, MD, MAS, editor-in-chief, JAMA® and JAMA Network™
- Stephen Parodi, MD, executive vice president, The Permanente Federation
Host
- Todd Unger, chief experience officer, AMA
Listen to the episode on the go on Apple Podcasts, Spotify or anywhere podcasts are available.
Transcript
Unger: Welcome to Moving Medicine, a podcast by the American Medical Association. Today’s episode is part two in a collaboration with The Kaiser Permanente Federation’s virtual event, The Reinvention of Research. Picking up on their conversation are editor-in-chief of JAMA® and JAMA Network™, Dr. Kirsten Bibbins-Domingo, and Dr. Stephen Parodi, associate executive director of The Permanente Medical Group. They’ll talk about a critical piece of this puzzle—the impact AI could have on medical journals and research. If you missed part one, go back and give it a listen. Here’s Dr. Parodi.
Dr. Parodi: The question I have for you around AI is how does it fit in from a either publishing perspective, writing the actual ... people are using ChatGPT to compose not just email and responses to patients, but maybe even publishing medical science. So how are you sorting that out? I mean, is ChatGPT now an author on a number of these papers, or what's that look like?
Dr. Bibbins-Domingo: Oh, it's such a good question. We think about this issue a lot. Yeah, in publishing, we are quite bullish on what generative AI is going to do for the field of medical publishing. One of the things that's still hard for us as a journal is figuring out how to communicate across multiple different levels of literacy. How to allow authors who might not be English speaking as their primary language to navigate our process seamlessly. How to make sure that we publish across many different types of ways of explaining a particular scientific content. And I think the tools with generative AI are really going to allow us many more options to be able to do the thing we want to do. I think with many things, as I'm sure you are thinking about as well in clinical practice, it's not without some risks or understanding how to use a new technology most effectively.
So soon after ChatGPT first came on the scene, we realized that we had to come out strongly and help guide authors. So ChatGPT was November 2022, and in January 2023, we published an editorial that said only humans can actually be authors on articles. And you would think that that would be self-evident, but at that time, ChatGPT had already been indexed as an author in PubMed on multiple articles. And so we basically said to authors, "Look, you can use this tool. You need to tell us how to use this. And depending on how you use this, you need to give us more information so that we can be upfront with our readers about how you've used it. We can take that into account when we're vetting the science. Only a human can testify to the integrity of the work."
And so humans still have to do that, but AI is a powerful tool. And that's our commitment to our readers as well. Whenever we use AI as a tool in anything that we publish or in anything in our publication process, we will also be upfront as well. But we are bullish on it. But we also think part of our job is to help guide authors and readers in how to use it responsibly.
Dr. Parodi: Now talk about challenging the rubric of what we think research looks like and how you publish, so let's dig into this a little bit more. So I want to talk to you about peer review. And of course, there've been critiques around traditional approaches to peer review. Particularly during the pandemic, I'm reflecting on just the speed at which publications were coming, and questions around timeliness, relevance, are the publications actionable? What do you think about all of that in that debate, in terms of the nature of peer review as it has been done? Are there changes that need to happen? Is it all fine? What's your sense of that?
Dr. Bibbins-Domingo: Yeah. It is definitely not a perfect system. I think though that many of the criticisms that I view as failures of the entire biomedical process oftentimes fall on the lap of peer review. They say, "Well, peer review should have caught this." If somebody is out to commit fraud, if somebody is sloppy in their science, peer review is critical, but it should not be the only fail-safe. And I think the challenge of the pandemic was the rush, the need, the desire to get information out quickly. The fact that we had multiple journals, that oftentimes poses an additional challenge just to the body of literature.
So I would say during this time, we take very seriously finding high quality peer reviewers. We're going to launch a new initiative in the upcoming year to actually make sure that we provide resources to educate people on how to do peer review effectively so they know what we are looking for. We take the conflicts of interest of peer reviewers very seriously. That's very much part of our process.
The one thing I want to make sure that every reader of JAMA knows is that peer review is only one input into the editorial decision. And we're constantly looking at peer reviewers who are volunteers who provide us essential input. But it's one piece of input and we are always trying to get as much as we can. And then to publish in a way that recognizes work is not perfect. Work always has its limitations, and we have a responsibility to also give voice to that. And I think as a clinical journal, we have a particular responsibility to provide the clinician an understanding of, "Is this ready for prime time? Should we be able to act on this? Is this a preliminary finding?"
And so you'll see all of our work really is published with editorials around them to really give readers the sense of, "Well, what does this actually mean?" And unfortunately, the world is nuanced and complex. And we have embrace that―to do our job as good as we can possibly do it, but we also have to embrace a little bit of the complexity to communicate with our readers.
Dr. Parodi: Yeah. I appreciate the amount of work that takes and the complexity. We actually have our own Permanente journal that's peer reviewed and open access. And the amount of work to have strict integrity to the system, building that whole peer review network and maintaining it, an incredible commitment. So thank you.
So just following up on that question here, which is peer review and shoring up that whole process, amazingly, in 2023, there were more than 10,000 academic papers that were retracted, which was an all-time high, both in health and health care. And so what's your take on why retractions appear to be increasing, and what does that mean from the standpoint of credibility? Does it actually enhance credibility because actually we're just being better stewards, or does it actually decrease credibility?
Dr. Bibbins-Domingo: Right. So let me just say, I think part of the process that I think journals serve and JAMA takes very seriously is the post publication process, which is about somebody flagging something that needs to be corrected in an article, somebody raising concerns about the legitimacy of an article. And ultimately, if it's found that the article does need to be retracted, is retracting it for the sake of the scientific record. But what you're talking about is critically important to highlight and to probe a little bit deeper about what this means.
First of all, I would say some of this reflects just the larger volume of publications. The number of journals are increasing. I alluded to this at the beginning, the good side of open access is in fact enhancing access. The downside is that when you align the paying per article, which is the model that open access currently exists, it's meant more journals have flooded into the market, including journals that are clearly predatory journals. These are predatory journals, they're rogue journals. There are other clearly substandard journals. And I say this because they oftentimes are delisted as certified publications. And there are also paper mills, mills that are basically submitting large volumes because of other types of pressures for publication.
And so if you look at the mountain of retractions and the increasing volume, some of it is just more literature out there, some of it is the good thing that we do want to retract bad science, and some of it is this ... we have a dark underbelly of medical publishing, which is predatory journals, the paper mills, those types of things, which really, when they're detected, they feed a lot of these retractions.
I have made the case, and I say this every time I'm asked to speak, is that it's fine for me to say, "This wasn't an article we retracted at JAMA. We don't have the problem of massive retractions." It's fine to say, "Oh, you esteemed scientists, your paper is not being retracted." But when science is being retracted at such a high rate, it affects the credibility, it affects the trustworthiness of the entire enterprise.
When it makes it on the front page of major news outlets, it means that people start not to trust what's published in a scientific journal. The lay audience starts not to trust. "Well, how do we know what science is actually doing? You're retracting articles all the time. How do I trust that?"
And I think we have to reckon with that aspect of it, all of us, even if we are not directly contributing to that dark underbelly and figure out, "Well, what can we do?" Because the system is ultimately built on trust.
Dr. Parodi: That's absolutely right. And I'm glad that I got the JAMA editor-in-chief to talk about the dark underbelly of medicine. So I do want to touch on this question of trustworthiness because it's very much of course a conversation within medicine and researchers, but you've also referenced that part of the audience here are people that are making decisions around policy and health care policy.
And so from your standpoint, what can medical journal editors do to rebuild public trust? Because I think that has been very much a point of discussion in the current environment that we're in. And let me just follow up with a second piece to that, which is medical misinformation and that topic of what is medical misinformation and what do we do to help physicians like myself who are practicing in an exam room to have the trust of our patients and regain some of that? I think people think we've lost a little bit of that over the last couple of years.
Dr. Bibbins-Domingo: Well, let me start with the clinician side of it. You and I are both practicing physicians. I'm sure you have the experiences I do, that we oftentimes have to convey information to patients who might have different understanding, a different level of understanding, who have different backgrounds, who might have a very different worldview than we have. And I think good clinicians know how to have those conversations, sometimes over time, that sometimes framing things a little bit differently, sometimes using whatever it is that we have at our disposal to explain, engage, whatever.
I think the challenge for us who are thinking about how to scale it is, how does that scale? So I have that same goal that I have when communicating with a patient is the same goal I have when I'm publishing a journal and trying to help authors to communicate to multiple audiences. But that means I have to try to figure out how to scale all those things that we're doing in practice, right?
So all of those things about how do you frame differently for different audiences? How do you paint the nuance in a way that doesn't get overwhelming, but allows somebody who wants to delve a little bit deeper to see the complexity and then arrive at the decision that they want to arrive at? And I think that takes a little more time. It's harder to do in the moment, it's more complex and it's a little bit messier in that way, but it is sort of I think what we are required to do.
I will tell you one of the things that always ... it's amazing to me how consistent this finding is, is that while the trust in scientific institutions is waning, the trust in institutions is waning, the trust in the doctor is as high as it has ever been. It's high as it's ever been, regardless of political affiliation, regardless of where you are in the country, it's as high as it's ever been. And so as a journal that prides itself on communicating science, publishing science for a clinical audience, part of what we're trying to do is really give clinicians more tools so that they have the tools to have these conversations. That is one of the ways I think we can do that.
We started a series called Communicating Medicine, which is just about how do you communicate ideas, how do you communicate in the doctor-patient encounter? And I think there's been a real interest and hunger for that. Just some of these basic tools. Some of the ways we try with our multimedia teams and our videos and our podcasts and our explainers and our graphics is designed to give people more tools to try to think about that. So I don't have the answer. And clearly, everything in the environment is telling us we have to be able to do this. So we're trying many approaches, but one of them is definitely to try to make sure that we're focused on the clinician because I think really the frontline clinicians, they are the trusted source of information. And if we can reach them in a way that also gives voice to the complexity, I think we're at least part of the way on the right track.
Dr. Parodi: It's so interesting what you're talking about here, because I'm going to go back to medical school, one of the first courses in terms of working with patients was how to communicate with patients. And then actually, that was the end of it. We never had another course about it. So I'm really interested in what you're touching on here, which is the idea of ... and I'm thinking about some of the real world conversations that I was having around vaccines and learning, by the way, by the school of hard knocks in terms of how to have these conversations, particularly if somebody has a different point of view, may or may not be informed by science. So I'm interested in, how did JAMA think about that in terms of needing to focus on communications? And I guess there's a second piece here, which is you were referencing modalities of how people learn, right? So the traditional, you're going to get something in the mail and read a paper journal, that's no longer the model. So what does that evolution look like from a JAMA perspective?
Subscribe to free AMA email newsletters
Get the latest news from the AMA on the topics that matter to you—delivered to your inbox.
Dr. Bibbins-Domingo: Yeah, you're so right. Let me start on the second part first. We all know―we consume, we're overloaded, each of us personally with information, and we are consuming it in many ways. When I started at JAMA, some of the editors would laugh at me. The real in-house editors and the team would say, "Oh, Kirsten always reads on her phone." And I'm like, "Well, everybody I know reads on their phone." And there's a tendency to think sometimes, "Well, this is a generational thing." And I'm like, it's not even quite a generational thing. One of my most esteemed board members said in a board meeting, who's a very senior, now emeritus professor, he said, "Your abstracts are too long. You need to get to the point quicker. Nobody has time to read all this. We have so many things we're reading." And so that is one of the things we have to think about.
My view is that journals have to do the job they've traditionally done, vetting the science―the high integrity of the science, understanding the limitations, doing that when you accept … they have to do that the same way they've always done it, and the best that they've ever done it. And then we have to do something we've never done, which is to then take that piece and then say, "Okay, let's now put it in a shorter form. Let's put it in a form that a busy clinician can really come to the key points very quickly. Let's have the video explainers. Let's have the podcast. If you're running on the treadmill on Saturday morning and you want to hear just the highlights, let's make sure there's a patient page to accompany that, so that if you …" get the idea?
And we're trying with many different types of ways to do this. We're also going to play a little bit with how do we frame? Sometimes even how things are framed, we come at it in the, "Well, here's the hazard ratio, and so therefore it means this." And there are many ways to frame what is the scientific finding. It doesn't mean we change what we do with the vetting or change how we publish the methods or the rigor of all that. It changes in, "Well, what's the bottom line we're trying to communicate and how can we communicate that as clear as possible for multiple audiences?" Especially audiences that may not come at it from the, "Oh, well, because it was published in JAMA, I'm inclined to necessarily believe it."
Many of the things that we publish through multimedia are discoverable by the lay public who doesn't read JAMA, who might not know what JAMA is. But those types of things have to be compelling and interesting for that audience too, if we're to do our job, because we know a lot of the other things that are not good scientific information are dispersed mainly on these types of platforms.
Dr. Parodi: This is great, Kirsten. How do you foresee the role of medical journals evolving over the next decade?
Dr. Bibbins-Domingo: I do think the issue of trust, the issue of access, the models of publishing that right now have led to an explosion of journals and a little bit of the, as I say, the dark underbelly, I think that's a little bit of ... we have to figure out how we get the best from our publishing models and minimize the worst. And I don't think journals can think about this by themselves. They have to think about this with academic communities, with the main readers of journals. And I don't know where it's going, I just don't think that we are going to continue in exactly the same path that we're on right now. And so it's a little bit of an unsettling time.
I will say that journals, you're probably aware―well JAMA, we are independently published. We're very proud to be independently published. So our publishers, the business side of the house works closely with us, the editorial side of the house. We are fire-walled where we need to be, but we're on the same page on our goals of publishing the best science.
Most journals right now are really part of these big commercial publishers who―it's a very different economic incentive I think there. And I think there's a risk for some of smaller journals, and you'll see that over time as well. So I don't know. I know that right now we are focused a lot at JAMA on how we can make sure that we're disseminating science across these platforms to reach a broad audience and then vetting the science in the best way we've ever done it. But as a field as a whole, I think there's a lot to suggest that we're in a little bit of flux.
Dr. Parodi: I'm going to pivot us here with a different topic. And this is really interesting, because at Kaiser Permanente and amongst the medical groups, we've got a number of clinicians that also practice as researchers and vice versa. And so this question's because of your background and expertise.
What practical tips do you have to foster more research amongst physicians who have limited time? They're practicing clinicians, and balancing a clinical load, but then also pursuing the research interests.
Dr. Bibbins-Domingo: Yeah, this is a major one. And I'm on leave from UCSF, but that's one of the issues that I've thought a lot about at UCSF as well. I think I am very excited about the many advances in trying to integrate clinical trials or clinical studies into the practice of medicine. And my hope would be that we continue to find models for doing that, that allows physicians who are in practice, who do see patients, who practice medicine to also take part in this as a part of what they do.
I think that the grand vision of having us learn all the time from what we're doing in clinical practice, whether in structured ways or in more decentralized unstructured ways, is actually the goal that serves us well, both scientifically as well as in a clinical side. And my hope would be that we would find systems of care that recognize the value of having practicing physicians participate in research as well.
We're not there, but I think if we believe, and I believe that science is essential to the best practice of medicine, having that more closely integrated, as it is in many places, is a goal that we should aspire to and then therefore protect some of the time for physicians to be able to do that.
Dr. Parodi: That's fantastic. And it's always a balancing act in terms of trying to figure that out. I have a question that we touched on earlier, but I'm going to delve into a little bit more. Because I've noticed JAMA does have the editorials, there are viewpoint articles, and then there are of course the traditional, as you were referencing, an abstract within an actual publication in terms of manuscripts. So talk to me a little bit about how you think about the publishing of quote, unquote hard science versus opinion in viewpoint. Is that a good thing to have those juxtaposed? Does it conflict? How do you think about that as an editor?
Dr. Bibbins-Domingo: Yeah. It's one of the areas I think of challenge, but I think the challenge is easier once you're clear on what the mission is and why you have a section called viewpoints or a section called editorials. So for us, this is about the art and science of medicine. We know medicine, a lot of it is art. A lot of it requires nuance. And if we only stuck to the thing that was going to survive peer review of the science or the well-researched clinical review, we would leave out a lot of the important texture to the practice of medicine. And so that is what the editorials allow us to do. It's what the viewpoints allow us to do.
For example, we published the phase III trial of Donanemab for Alzheimer's disease. We published that with four editorials. So a really important phase III clinical trial, clear implications for practice. And then the four editorials, one touched on the issues of the under-representation of certain patient populations in those trials and what that means for the practice. We published an editorial on the cost of these medications. We published one on the mechanisms, and we published one on the implications from the geriatricians on what does this mean for clinical practice? That is the reality of what you grapple with when you have a new drug that is powerful.
And so for us just to publish that I don't think is in keeping with what our responsibility is. The viewpoints―I think we want to keep on viewpoints that are really related to clinical practice. And so we are not interested in things that are tangential to that. We try to stay in that lane. We're not going to always publish a thing that is everyone's cup of tea. That's the nature of viewpoints. But I'll tell you, the top viewed viewpoints in the last few months at JAMA are related to "Well, how do you actually use the GLP-1s in clinical practice? How do you integrate that with your recommendations for healthy diet? What does it mean that a lot of our patients are discontinuing GLP-1s?"
And those are really important topics for us to grapple with, because we all know as clinicians, that is what you see in practice. And if we waited until we had a great review or a great clinical study, we're not going to get there.
We published a viewpoint that's highly viewed on why do we have so few pediatricians right now? And that is one that everybody who's in the system, we know that to be the case. These are peer reviewed also; we use our editorial judgment. But I do think they're important to the conversations that we have in medicine that we don't yet have the science, and they are about that broader texture. But it's one where there's a lot of like, "Should we be doing this? Is this really what we want to be doing?" And we ask that questions as editors all the time.
Dr. Parodi: That's great. And what I really enjoy about hearing your thoughts on this is that you're talking about real world issues that we're grappling with day-to-day in our clinical practices, whether that's in the exam room, whether it's as a department or even as a system. And so let me touch on this. This question's a good one because it's talking about the speed of information that's getting out there. And now it's at the speed of light essentially when you think about social media and what can get posted on there and the spread of scientific misinformation. So the question here is, should our profession, should the research committee, should medical literature put more efforts into combating misinformation? And what would that look like or should that look like?
Dr. Bibbins-Domingo: Yeah, I don't know what it looks like. I mean, there are people who have suggested that what our job should be is to be more vocal about disinformation. And the challenge of doing that is that that looks self-serving. Medical journals are oftentimes criticized if we're publishing studies that were done, that we're in bed with pharma companies or with other types of institutional practices of medicine.
So I think the people who need to hear the messages about disinformation are not necessarily those who are going to take kindly to the finger-wagging of major journals telling you that. So what we've chosen to do instead is to say a lot of people just don't have high quality information, and my job is to put more high quality information out there for a lot of different types of readers, a lot of different types of consumers. And that's my job. That’s my job is also to educate clinicians because they're on the front lines of talking to people one-on-one, and to give people more tools to do that.
I think we have to use the same tactics as others who are using information for other goals to use the information for the goals we have of providing the best evidence to inform the practice of medicine. And so we focus more on those things, the high quality, and doing that across multiple platforms. But it's not an easy one to do.
Dr. Parodi: Yeah. And I guess this touches on another question. Again, this goes back to the original discussion we were having around democratization. And so the question the audience is asking here, which is an interesting one, is what are your thoughts on high fees involved in article publication, the culture of pay-to-play in journal publishing? Do you have concerns about the implications of this model, both for researchers in general and then for the ones that are less well-funded?
Dr. Bibbins-Domingo: Yes. I mean, this is what I've alluded to before. So full disclosure, JAMA Network has two open access journals. We are proud to have open access options for all of our other journals as well. But the model for open access is, the author pays for the publication to publish. And for many, those fees are extraordinarily high. You'll see some journals, oftentimes actually ironically from the major commercial publishers, with exorbitantly high fees.
But JAMA took the position on and the JAMA Network took the position on a few months after I joined, about six months after I joined, we decided that all of our science will be publicly accessible. So authors can deposit their articles in public repositories without any fees at the time of publication. And this is because we believe in the accessibility of science, but I think that the pay-to-play is both ... unfortunately, we focused our attention on the reader, but it crowds out many authors―authors who might be earlier in their careers, authors who are less well-funded disciplines, authors from less well-funded institutions. That's not good for science at all.
And so I think we have to collectively think about better models for publishing. We have to recognize―for me, I think that there needs to be a process, particularly with clinical journals that involves peer review, the editorial review. Some of this that we are trying to do at JAMA with really thinking about how do we disseminate across multiple audiences, that doesn't come for free. And then thinking about how you keep access in mind, but also get the value that you get from publishing is important. And I don't think, I really don't think that the pay-for-play model is in fact the open access model. When you say open access, no one is against it. We all want more access. But the pay-to-play model, that is one where some of the downsides are also part of it, including what's led to all the retractions. And so I don't think that's sustainable.
Dr. Parodi: So Kirsten, final question for you, and to end on an optimistic note, what are your upcoming initiatives for JAMA? What are you most excited about?
Dr. Bibbins-Domingo: Sure. In October, we launched JAMA+ AI. And JAMA+ AI sounds like it's a new journal. And if you go to the website, you can put in JAMA+ AI. You'll see the website. It looks like a journal. What it is actually is a window into AI across the JAMA Network. It's updated weekly. It has all of the newest content across the JAMA Network. And because we're focused on clinical practice in the way we think about the science of AI, it allows, I think, a reader who wants to come in to learn a little bit more, to get access to that content. And it allows those computer scientists who―they have never published in JAMA, they don't read JAMA, but they really want to make sure that their innovation has clinical applicability. They've designed their studies in that way that we can help them to actually have that science reach a clinical audience.
So that's what we're excited about. JAMA+ AI has a new editor-in-chief who's a psychiatrist at Mass General who uses AI. And what he's very passionate about is actually really bringing more people into the discussion. He has a wonderful podcast series. It's really focused on clinical practice. But he himself is very skilled in this as well and is really helping all of our journals to do better.
I don't think what we need right now is more journals, at least for us at the JAMA Network. I think what we need is to basically help people to find the content that they want to find, to find it at the level they want to read and access that content. JAMA+ AI is a channel that's like where we want to be going, and I think you'll see us launching more and more of these things so that you can come in to see the work we are publishing across the JAMA Network. But you're going to find the thing that you want to read, hopefully use for your patient care, maybe explore something new that you haven't thought about before, and that we're going to be able to help that experience be a good one for all of our readers and our authors.
Dr. Parodi: Yeah. That's a really exciting way to end this conversation because it really gives me hope, compared to when we first introduced the electronic health record, and it was this disembodied thing that wasn't part of medicine. We've got an opportunity to reset, is what I'm hearing from you, to actually have the computer scientist integrated with medical science and medical publishing. What an exciting time.
So Kirsten, thank you so much for this conversation, and I want to thank everyone who joined us today. Check out Permanente.org for our library of past videos and podcasts. As we continue to drive the future of value-based care at Kaiser Permanente, collaboration between health care clinicians, researchers and academic journals will be crucial in shaping a more efficient, effective and patient-centered system. By fostering an open dialogue and emphasizing the importance of research and care delivery, we can collectively create a better health and better outcomes for our patients.
Unger: This has been Moving Medicine, a podcast by the American Medical Association. Subscribe today to never miss an episode. Thanks for listening.
Disclaimer: The viewpoints expressed in this podcast are those of the participants and/or do not necessarily reflect the views and policies of the AMA.
