How AI is Changing Product Development with Gong AI
In this episode, Raz Nussbaum from Gong AI covers topics from the impact of large language models on Gong's product offerings, the challenges of operating AI at scale, to the evolving role of product managers in the AI era. Raz also offers practical advice for teams venturing into Generative AI development for the first time and shares his perspective on the future of AI in sales.
Subscribe to Humanloop’s new podcast, High Agency, on YouTube, Spotify, or Apple Podcasts
Key Takeaways:
1. Stop talking, start prompting
Raz emphasizes the need for PMs to directly interact with the models. Forget the theoretical—get hands-on with prompt engineering, QA, and iterate based on real-world feedback as quickly as possible.
2. The competitive landscape is tougher than ever
With barriers to entry lower than ever, Raz shares his strategy to move fast, adapt quickly, and stay ahead in an industry where everyone is racing to innovate.
3. Real customer data is your best friend
Before you even have a UI, test with real customer data. Raz gives you the blueprint for involving customers from the earliest stages and iterating in real-time to build features they’ll actually love.
Chapters
00:00 - Introduction
01:16 - How LLMs Changed Product Development at Gong AI
08:32 - Including Product Managers in Development Process
13:05 - Testing and Monitoring Pre vs Post-deployment
17:53 - New Challenges in the Face of Generative AI
19:39 - Shipping Fast and Interacting with the Market
23:25 - What's Next For Gong AI
25:13 - The Psychology of Trusting AI
28:19 - Is AI Overhyped or Underhyped?
Podcast:
[00:00:30] Raza Habib:
This is High Agency, the podcast for AI builders. I'm Raza Habib, and I'm
delighted today to be joined by Raz Nussbaum, who's a Senior Product Manager in
AI at Gong, which is the AI platform for revenue teams. Over there, he's led
conversational AI and spearheaded the deployment of many generative AI products.
So, I'm excited to speak to him and hopefully share learnings about what it
takes to build generative AI products at scale. Raz, it's nice to have you on
the show.
[00:00:54] Raz Nussbaum:
Nice to be invited to the show, and I'm looking forward to discussing AI and
Product Management with you and the listeners.
[00:01:01] Raza Habib:
Raz, just to set the scene for people, can you tell me a little bit about the AI
features that Gong has added with large language models (LLMs) and post-gen AI?
I know Gong has been building conversational intelligence with AI for a long
time. What changed for you after large language models came out?
[00:01:16] Raz Nussbaum:
Basically, when I joined Gong as an AI Product Manager, Gong was already an AI
company, even before the GPT boom, right? Before the LLMs. Now, it's
everywhere—everyone talks about it, even during family dinners with my grandma.
But before that, Gong was an AI company. One of the features I led before the
LLM revolution was allowing business users to create their own classifiers—NLP
classifiers—by themselves, doing all the training like the machine learning
process. It took a lot of time and engineering resources since it was a
pioneering effort. It wasn't widespread yet, and there weren't many references
to how to build such AI features. It was tougher, both from a product and
research perspective. For example, let's talk about summarization, which is one
of the features I led at Gong. Before the LLM boom, summarizing text in a
meaningful way was a technical challenge that was not solved in a good way. Gong
had wanted to create a feature like "Call Spotlight" for years, but the LLM
revolution enabled us to do it. To sum it up, the LLMs helped me as a Product
Manager in AI by making the process easier. Now, the tools are much more
accessible, and it's much faster to create products based on those long-time
dreams. That's the main takeaway—it's now easier.
[00:02:53] Raza Habib:
Were there any features that you couldn’t build before, not just easier but
actually unlocked?
[00:02:58] Raz Nussbaum:
I think the summarization feature is a prime example. It wasn't solved before.
We tried internally and realized the amount of effort it would take for our team
to do it. It was a significant challenge. But now with LLMs, we can provide them
the transcription and metadata, use prompts, and get results so fast. It's a
huge enabler. Without that, we would have spent a lot of time developing a
solution that might have only been okay. The evolution of LLMs solved that.
[00:03:37] Raza Habib:
Can you give me a sense of the scale at Gong? How many people are you roughly
serving, and what do these models need to accomplish? Summarizing calls with a
large LLM is slow and expensive. How do you operate at that scale?
[00:03:52] Raz Nussbaum:
That’s a great question. The industry is moving so fast. When selecting an LLM
vendor for your product, you must ask if they can support your scale. Gong is
transcribing millions of calls in a short period. So, when choosing a
vendor—whether it's GPT on Azure, AWS, Anthropic, or others—you have to ask,
"Can it support our scale?" That's the first consideration. Then you have to
think about the cost and quality. These are the main factors we evaluate.
Thankfully, the competition is high, which is driving down costs, increasing
quality, and making scale less of a problem. Even processing time is improving.
So, when we launched Call Spotlight, a feature summarizing our calls, we
initially used one LLM vendor. As technology advanced and competition increased,
we quickly switched to a different vendor that was cheaper, supported scale
better, and offered higher quality.
[00:06:06] Raza Habib:
Can you talk me through the process of building an AI feature at Gong? How do
you go from idea to deployed feature? Who’s involved, and what's the process for
you?
[00:06:14] Raz Nussbaum:
The process hasn’t changed that much. People think AI has changed Product
Management entirely, but it hasn't. You still need to work closely with
customers. At Gong, we serve B2B, so I need to talk to my personas and
understand their problems. The problems haven’t changed because of AI; they are
the same core problems. For example, a call is recorded, and it takes a lot of
time to analyze. Summarization could be a solution, whether it’s AI or a manual
service. As a Product Manager, the problem is the focus. That’s 80% of what a
Product Manager should do—understanding and defining the problem. The solution,
including AI, is just another layer. What has changed is that it's now easier to
create models. You can iterate quickly and release an MVP to get feedback fast.
But the core process remains the same.
[00:08:32] Raza Habib:
What about the literal process or the team involved? Are you doing prompt
engineering, fine-tuning models, or using machine learning engineers? How does
it work in practice?
[00:08:45] Raz Nussbaum:
We do all of that. Once the problem is well-defined, we discuss solutions with
the research team, which advises on the best approach—whether it’s a general LLM
or a custom fine-tuned model. We involve engineers in the conversation as well.
In some cases, we use general LLMs, while in others, we need to fine-tune a
model to meet customer expectations. Our research team leads the ideation, and
we also have prompt engineers who specialize in writing prompts well. I also
write prompts myself and work closely with the team on optimizing them.
[00:10:57] Raza Habib:
That’s an interesting point. You mentioned earlier that the job of a Product
Manager hasn’t changed much, but it seems like you’re now involved in
implementation with prompt engineering. Would you say that’s a big change?
[00:11:46] Raz Nussbaum:
Definitely, but it’s similar to what I did before with NLP models. I’ve always
been involved in QA for AI models, understanding what outputs work and what
doesn’t. Now, it’s easier because I can suggest prompt changes directly. In the
past, making changes required drastic steps—relabeling data, different datasets,
etc. Now, it’s as simple as tweaking the prompt. We iterate quickly and compare
versions to decide which works best.
[00:13:05] Raza Habib:
How do you handle quality assurance? What’s the process for testing, and how do
you measure success?
[00:13:12] Raz Nussbaum:
It depends on the feature. We like to have templates, but we also adapt based on
the task. For example, when migrating from one LLM vendor to another, we took a
test set of calls, ensured diversity across industries, and had human reviewers
rate the outputs. We compared outputs from the old vendor and the new vendor,
rated them on a scale from one to five, and selected the better one. We do this
for critical features used by thousands of users daily.
[00:14:39] Raza Habib:
Do you do this every time you change model providers?
[00:14:43] Raz Nussbaum:
For critical features that impact many users, yes, we run thorough tests. We
need to feel comfortable before making changes. It’s not enough for a researcher
to say it works; we need to see results.
[00:15:23] Raza Habib:
What about post-deployment? Once the feature is live, how do you monitor
quality?
[00:15:33] Raz Nussbaum:
We have feedback mechanisms like thumbs up/thumbs down after the feature is
live. It helps us collect more data at scale. However, if I did my job well
during beta testing and spoke with customers, I capture about 95% of the
feedback. Once we launch to GA, the feedback we receive shouldn’t surprise us.
If it does, it means I didn’t do a good enough job.
[00:17:30] Raza Habib:
One of the themes in this conversation is that building with generative AI isn’t
fundamentally different from building AI products before. Is there anything
harder with generative AI, or are there new challenges?
[00:17:53] Raz Nussbaum:
The challenge is competition. Everyone can do everything now; there aren't many
entry barriers for certain features. It seems that way, at least. That's one
challenge for AI companies—it's easier for others to copy what you’re doing. The
second challenge is the pace of advancement. It’s hard to predict what will
happen next week in the AI space. So at Gong, our strategy is to release things
fast, interact with the market, and adapt to new solutions. Core problems don’t
change, but the solutions do. So, we have to implement quickly and then improve
as we go. That way, when the next solution comes out, we can implement it faster
with all the feedback we’ve collected. You need to interact with the market
constantly and embrace change.
[00:19:39] Raza Habib:
That makes a lot of sense. Just to make sure I’ve understood, your advice is to
move quickly and learn from real feedback. The space is changing so fast that
the need to strategize has shifted to the need to ship fast and learn in
production.
[00:20:07] Raz Nussbaum:
Yes, more than ever. You need to learn from the market. AI is just one part of
the solution. The full application, including user experience, is what makes the
difference. If you can build an app customers love, that’s the key. AI might be
the core of the solution, but the entire user flow and experience are what
matter in the end.
[00:20:56] Raza Habib:
What advice would you give to a product manager or team lead who is building
with generative AI for the first time? What should they consider? How did you
get up to speed?
[00:21:08] Raz Nussbaum:
First, stop talking and start prompting. If you're not doing prompt engineering
and QA for outputs, you don't know what you're talking about. Go open accounts
on platforms like Anthropic or GPT, and start experimenting. Understand the
process, learn how prompts work, and get hands-on experience. The second piece
of advice is to always test with real customer data, not mock-ups. Before we
even have a UI, we let customers interact with the outputs in a developer
notebook. The designer and engineers are in the room to observe how customers
react to the outputs, and we gather feedback. The key is real data and real
feedback, even before any UI is built.
[00:23:17] Raza Habib:
That resonates with our experience. You've been in the AI space for a while.
Gong was an AI company before LLMs, and it’s adopted generative AI quickly. How
do you see your product evolving in the future? How will people sell differently
with Gong three to five years from now?
[00:23:40] Raz Nussbaum:
I think AI's biggest contribution is efficiency. Gong serves go-to-market teams,
and most of their challenges involve spending time on non-selling activities. A
lot of the tasks they need to do can now be handled by AI. For example, feeding
the CRM, writing follow-up emails, summarizing calls, preparing handover
documents—AI can take care of 90% of that. I see a future where AI handles the
heavy lifting, leaving human teams to focus on the important, value-added work
like building connections with prospects and closing deals. It’s a huge
opportunity for humanity to let AI take over the repetitive tasks, while we
focus on the more meaningful work.
[00:25:13] Raza Habib:
The models have improved a lot recently. Do you think humans will still be
needed in the loop, or will AI eventually take over entirely?
[00:25:32] Raz Nussbaum:
It’s a tricky question because it’s also a matter of psychology. Humans may not
feel comfortable leaving certain tasks entirely to AI. For example, would you
trust AI with a critical medical diagnosis? Some people might say yes, but
others might want a human doctor’s opinion. It comes down to risk tolerance and
mistake tolerance. People need to feel confident in the technology before
they’re willing to trust it with major decisions.
[00:26:23] Raza Habib:
But as AI improves, isn't there a point where it might be riskier to rely on
humans instead of AI?
[00:26:33] Raz Nussbaum:
I had a personal experience with this. I was on vacation with my family, and one
of my kids had something on their skin. I could either use our travel insurance
and talk to a doctor or upload an image of the wound to GPT and get an opinion
from the AI. I ended up doing both, and the AI gave me almost the same diagnosis
as the doctor. That was pretty shocking, even to me. The difference was the AI
was available immediately, while the doctor wasn’t. It’s surprising how much AI
can already handle, but we’re still in the early stages. I think over time,
we’ll become more comfortable with AI doing more of the work.
[00:28:04] Raza Habib:
Absolutely. Well, Raz, we’re coming up on time. One question I always like to
ask: Do you consider AI today to be overhyped or underhyped?
[00:28:19] Raz Nussbaum:
Underhyped. The opportunities are endless with today’s AI models. We’re still
just scratching the surface of what AI can do. I think people don’t yet fully
understand the potential. New capabilities are emerging so quickly, and there
are so many use cases that will be unlocked. It’s hard to predict what’s coming
next. Even as an expert, I don’t know what new opportunities tomorrow’s models
will bring. So, yes, I believe AI is underhyped, not overhyped. The sky's the
limit.
[00:29:17] Raza Habib:
I tend to agree. It’s been a great conversation learning about how you're
building AI at Gong. Thanks so much, Raz.
[00:29:24] Raz Nussbaum:
Appreciate the time. Thank you.
[00:29:27] Raza Habib:
That’s it for today’s conversation on High Agency. I’m Raza Habib, and I hope
you enjoyed the episode. If you did, please take a moment to rate and review us
on your favorite podcast platform—Spotify, Apple Podcasts, or wherever you
listen—and subscribe. It really helps us reach more AI builders like you. For
show notes and more episodes, visit humanloop.com/podcast. If today’s
conversation sparked any ideas or insights, I’d love to hear from you. Email me
at raza@humanloop.com or find me on X at @RasRascal.
About the author
- 𝕏@RazRazcle