Regions
MAIN/REGIONS
regions/
Regions
  1. Americas
  2. Asia & Oceania
  3. Europe
  4. Middle East & Africa
  1. Solutions
  2. Products
  3. Insights
  4. About
  5. Careers
  6. Contact Us
MAIN/search
MAIN/Solutions
Solutions/
SOLUTIONS
  • Research & Development
  • Real World Evidence
  • Commercialization
  • Safety & Regulatory Compliance
  • Technologies
LIFE SCIENCE SEGMENTS
HEALTHCARE SEGMENTS
THERAPEUTIC AREAS

Impacting People's Lives

"We strive to help improve outcomes and create a healthier, more sustainable world for people everywhere.

LEARN MORE
Harness the power to transform clinical development

Reimagine clinical development by intelligently connecting data, technology, and analytics to optimize your trials. The result? Faster decision making and reduced risk so you can deliver life-changing therapies faster.

Research & Development Overview
Research & Development Quick Links
Real World Evidence. Real Confidence. Real Results.

Generate and disseminate evidence that answers crucial clinical, regulatory and commercial questions, enabling you to drive smarter decisions and meet your stakeholder needs with confidence.

REAL WORLD EVIDENCE OVERVIEW
Real World Evidence Quick Links
See markets more clearly. Opportunities more often.

Elevate commercial models with precision and speed using AI-driven analytics and technology that illuminate hidden insights in data.

COMMERCIALIZATION OVERVIEW
Commercialization Quick Links
Service driven. Tech-enabled. Integrated compliance.

Orchestrate your success across the complete compliance lifecycle with best-in-class services and solutions for safety, regulatory, quality and medical information.

COMPLIANCE OVERVIEW
Safety & Regulatory Compliance Quick Links
Intelligence that transforms life sciences end-to-end.

When your destination is a healthier world, making intelligent connections between data, technology, and services is your roadmap.

TECHNOLOGIES OVERVIEW
Technology Quick Links
MAIN/Products
Products/
CLINICAL PRODUCTS
COMMERCIAL PRODUCTS
COMPLIANCE, SAFETY, REG PRODUCTS
REAL WORLD PRODUCTS

CASE STUDY

"Data in a Day

LEARN MORE
MAIN/Insights
Insights/
BLOGS, WHITE PAPERS & CASE STUDIES

Explore our library of insights, thought leadership, and the latest topics & trends in healthcare.

DISCOVER INSIGHTS
THE IQVIA INSTITUTE

An in-depth exploration of the global healthcare ecosystem with timely research, insightful analysis, and scientific expertise.

SEE LATEST REPORTS

INSTITUTE REPORT

"Digital Health Trends 2024 - Implications for Research and Patient Care

READ MORE

EMEA Insights

"IQVIA thought leadership with a focus on EMEA.

Learn more

WHITE PAPER

"DCTs Deliver Big ROI

LEARN MORE

WHITE PAPER

"Capturing value at scale: The $4 billion RWE imperative

LEARN MORE
MAIN/About
About/
FEATURED INNOVATIONS
  • IQVIA Connected Intelligence™
  • IQVIA Healthcare-grade AI™
  • Human Data Science Cloud
  • IQVIA Innovation Hub
  • Decentralized Trials
  • Patient Experience Solutions with Apple devices
WHO WE ARE
NEWS & RESOURCES

INVESTOR RELATIONS

"Visit our investor relations site for more information.

LEARN MORE
Unlock your potential to drive healthcare forward

By making intelligent connections between your needs, our capabilities, and the healthcare ecosystem, we can help you be more agile, accelerate results, and improve patient outcomes.

LEARN MORE
IQVIA AI is Healthcare-grade AI

Building on a rich history of developing AI for healthcare, IQVIA AI connects the right data, technology, and expertise to address the unique needs of healthcare. It's what we call Healthcare-grade AI.

LEARN MORE

Your healthcare data deserves more than just a cloud.

The IQVIA Human Data Science Cloud is our unique capability designed to enable healthcare-grade analytics, tools, and data management solutions to deliver fit-for-purpose global data at scale.

LEARN MORE
Innovations make an impact when bold ideas meet powerful partnerships

The IQVIA Innovation Hub connects start-ups with the extensive IQVIA network of assets, resources, clients, and partners. Together, we can help lead the future of healthcare with the extensive IQVIA network of assets, resources, clients, and partners.

LEARN MORE
Proven, faster DCT solutions

IQVIA Decentralized Trials deliver purpose-built clinical services and technologies that engage the right patients wherever they are. Our hybrid and fully virtual solutions have been used more than any others.

LEARN MORE
IQVIA Patient Experience Solutions with Apple devices

Empowering patients to personalize their healthcare and connecting them to caregivers has the potential to change the care delivery paradigm.

LEARN MORE
MAIN/Careers
Careers/
WORKING AT IQVIA

Our mission is to accelerate innovation for a healthier world. Together, we can solve customer challenges and improve patient lives.

LEARN MORE
LIFE AT IQVIA

Careers, culture and everything in between. Find out what’s going on right here, right now.

LEARN MORE

WE’RE HIRING

"Improving human health requires brave thinkers who are willing to explore new ideas and build on successes. Unleash your potential with us.

SEARCH JOBS
MAIN/Contact Us
Contact Us/
Blog
Generative AI in Life Sciences: Evaluating LLMs/GPT for summarization of Primary Market Research
Arpit Rajauria, Sr. AI Engineer, IQVIA
Lauren Poole, Associate Principal, IQVIA X
Sandip Ghosh, Manager, AI Engineering, IQVIA
Sunil Kumar Singh, Sr. Director AI Engineering, IQVIA
Raja Shankar, Managing Principal and Head of AI and Technology Consulting, IQVIA
Mar 14, 2023
  • Blogs
  • Generative AI in Life Sciences: Evaluating LLMs/GPT for summarization of Primary Market Research

Since its launch in November 2022, chatGPT has taken the world by storm. By January 2023, the AI-powered Q&A engine had already amassed over 100 million users, leading many to view it as the next revolution in search engines considering the integration with Microsoft Bing – it has even (at least temporarily) led to a restructure of essay requirements in secondary schools and universities.

We are excited to announce that here at IQVIA, we have been working on multiple use cases using Large Language Models like GPT, in a bid to provide our clients with the best possible services.

A significant portion of the strategic work for which IQVIA is commissioned relies on the application of domain expertise, complemented by hypothesis-driven primary research with industry experts, decision makers, and influencers (not limited to healthcare system payers and former payers, key opinion leaders, pharmacists). The hypothesis validation is essential, but the process of synthesizing and analyzing the findings, especially when the sample size is extensive, can be cumbersome.

Using a series of prompt engineering techniques that likely involved designing and refining the input, we have created a new GPT-based tool to help our teams synthesize the transcription data we collect during Primary Market Research. Summarization at the click of a button.

Deploying GPT on interview transcripts – assessing the quality

To gain more internal buy-in for the tool, we wanted the future users to assess the quality for themselves, in a blinded manner. Posting an internal survey (n=30), we had the respondents assess four summaries. Two were generated by the project team, two were generated by GPT3.5 (text-davinci-003). Each summary differs, in the sense that as human recollection and response changes, GPT also does not issue verbatim responses each time it is queried.

They were assessed based on the following criteria:

  • Relevance: How consistent the summary is
  • Factuality: Whether the generated summary contains only statements entailed by the source text
  • Coherence: Whether the text builds from sentence to sentence to a coherent body of information about a topic
  • Semantic Coverage: How much of the important information from the original text is included in the generated summary

The results were impressive.

Whilst at the point of survey construction we had been strictly utilizing prompt-engineering techniques and had not yet employed indexing methods let alone fine-tuning (therefore GPT lacked the general context of the overall project content), it was able to generate a factual summary from the raw input data. If anything, the respondents noted that one of the GPT summaries was more coherent than the human generated versions, and overall rated the quality very similar to that of the human.

Because the study was blinded, respondents were then asked to mark whether the summary was human generated, or GPT generated. Perhaps not surprisingly, they could identify that it was generated by the model simply based off the fact that it was so articulate, something that lacked in the more informal human generated versions.

What else have we learned?

We are ‘cautiously’ optimistic. Whilst deploying GPT on transcripts has proven beneficial, generating more time for our strategy consultants to apply critical thinking, we are aware that the model outputs can be temperamental, with loss of nuance and even occasional ‘hallucinations’, i.e. when the model outputs a seemingly coherent piece of text, that is factually incorrect. In other words, GPT can’t be used straight ‘out-of-the-box’ in Life Sciences. To explain in more detail, we asked chatGPT:

While GPT has shown impressive performance in various language tasks, it still requires indexing and prompt engineering for several reasons.

First, GPT generates text based only on the context it has been trained on. Therefore, it is crucial to provide GPT with a prompt that adequately sets the context for the generated text.

Second, GPT processes text in chunks, and indexing is necessary to access the chunks efficiently. GPTs architecture relies on a sequence of tokens to represent text, and indexing involves creating an efficient mapping between the tokens and their positions in the input text.

Finally, indexing and prompt engineering can improve the efficiency and effectiveness of GPT for specific tasks. For example, in question-answering tasks, a well-crafted prompt can help GPT generate a concise and accurate answer more quickly. Similarly, indexing can speed up the text generation process making it possible to generate longer and more complex texts without sacrificing performance.

Where are we headed?

Here at IQVIA, we know not to trick ourselves when we’re making a lot of progress.

We initially rolled out the capability in a less sensitive area to test, and we have already moved on to successful theme generation and are building a Q&A capability on top of transcripts. Progress has been significant, but we continue to keep the human in the loop to reinforce the model and to make final selections on the preferred outputs.

Large Language Models like GPT have the ability to make an incredible impact. However, it is important that we assess reliability, safety, trust and bias of these models if we in turn are to be trustworthy in what we’re talking about. Therefore, we are investing heavily to ensure that when we apply GPT to other, more sensitive use cases, the margin for error is minimal.

As we continue to accelerate innovation for a healthier world, rest assured there is more to come from us here at IQVIA. Watch this space…

For more information, please reach out to Lauren Poole

Contact Us