How AI is Changing Product Development with Gong AI

In this episode, Raz Nussbaum from Gong AI covers topics from the impact of large language models on Gong's product offerings, the challenges of operating AI at scale, to the evolving role of product managers in the AI era. Raz also offers practical advice for teams venturing into Generative AI development for the first time and shares his perspective on the future of AI in sales.

Subscribe to Humanloop’s new podcast, High Agency, on YouTube, Spotify, or Apple Podcasts

Key Takeaways:

1. Stop talking, start prompting

Raz emphasizes the need for PMs to directly interact with the models. Forget the theoretical—get hands-on with prompt engineering, QA, and iterate based on real-world feedback as quickly as possible.

2. The competitive landscape is tougher than ever

With barriers to entry lower than ever, Raz shares his strategy to move fast, adapt quickly, and stay ahead in an industry where everyone is racing to innovate.

3. Real customer data is your best friend

Before you even have a UI, test with real customer data. Raz gives you the blueprint for involving customers from the earliest stages and iterating in real-time to build features they’ll actually love.

Chapters

00:00 - Introduction
01:16 - How LLMs Changed Product Development at Gong AI
08:32 - Including Product Managers in Development Process
13:05 - Testing and Monitoring Pre vs Post-deployment
17:53 - New Challenges in the Face of Generative AI
19:39 - Shipping Fast and Interacting with the Market
23:25 - What's Next For Gong AI
25:13 - The Psychology of Trusting AI
28:19 - Is AI Overhyped or Underhyped?

Podcast:

[00:00:30] Raza Habib:
This is High Agency, the podcast for AI builders. I'm Raza Habib, and I'm delighted today to be joined by Raz Nussbaum, who's a Senior Product Manager in AI at Gong, which is the AI platform for revenue teams. Over there, he's led conversational AI and spearheaded the deployment of many generative AI products. So, I'm excited to speak to him and hopefully share learnings about what it takes to build generative AI products at scale. Raz, it's nice to have you on the show.

[00:00:54] Raz Nussbaum:
Nice to be invited to the show, and I'm looking forward to discussing AI and Product Management with you and the listeners.

[00:01:01] Raza Habib:
Raz, just to set the scene for people, can you tell me a little bit about the AI features that Gong has added with large language models (LLMs) and post-gen AI? I know Gong has been building conversational intelligence with AI for a long time. What changed for you after large language models came out?

[00:01:16] Raz Nussbaum:
Basically, when I joined Gong as an AI Product Manager, Gong was already an AI company, even before the GPT boom, right? Before the LLMs. Now, it's everywhere—everyone talks about it, even during family dinners with my grandma. But before that, Gong was an AI company. One of the features I led before the LLM revolution was allowing business users to create their own classifiers—NLP classifiers—by themselves, doing all the training like the machine learning process. It took a lot of time and engineering resources since it was a pioneering effort. It wasn't widespread yet, and there weren't many references to how to build such AI features. It was tougher, both from a product and research perspective. For example, let's talk about summarization, which is one of the features I led at Gong. Before the LLM boom, summarizing text in a meaningful way was a technical challenge that was not solved in a good way. Gong had wanted to create a feature like "Call Spotlight" for years, but the LLM revolution enabled us to do it. To sum it up, the LLMs helped me as a Product Manager in AI by making the process easier. Now, the tools are much more accessible, and it's much faster to create products based on those long-time dreams. That's the main takeaway—it's now easier.

[00:02:53] Raza Habib:
Were there any features that you couldn’t build before, not just easier but actually unlocked?

[00:02:58] Raz Nussbaum:
I think the summarization feature is a prime example. It wasn't solved before. We tried internally and realized the amount of effort it would take for our team to do it. It was a significant challenge. But now with LLMs, we can provide them the transcription and metadata, use prompts, and get results so fast. It's a huge enabler. Without that, we would have spent a lot of time developing a solution that might have only been okay. The evolution of LLMs solved that.

[00:03:37] Raza Habib:
Can you give me a sense of the scale at Gong? How many people are you roughly serving, and what do these models need to accomplish? Summarizing calls with a large LLM is slow and expensive. How do you operate at that scale?

[00:03:52] Raz Nussbaum:
That’s a great question. The industry is moving so fast. When selecting an LLM vendor for your product, you must ask if they can support your scale. Gong is transcribing millions of calls in a short period. So, when choosing a vendor—whether it's GPT on Azure, AWS, Anthropic, or others—you have to ask, "Can it support our scale?" That's the first consideration. Then you have to think about the cost and quality. These are the main factors we evaluate. Thankfully, the competition is high, which is driving down costs, increasing quality, and making scale less of a problem. Even processing time is improving. So, when we launched Call Spotlight, a feature summarizing our calls, we initially used one LLM vendor. As technology advanced and competition increased, we quickly switched to a different vendor that was cheaper, supported scale better, and offered higher quality.

[00:06:06] Raza Habib:
Can you talk me through the process of building an AI feature at Gong? How do you go from idea to deployed feature? Who’s involved, and what's the process for you?

[00:06:14] Raz Nussbaum:
The process hasn’t changed that much. People think AI has changed Product Management entirely, but it hasn't. You still need to work closely with customers. At Gong, we serve B2B, so I need to talk to my personas and understand their problems. The problems haven’t changed because of AI; they are the same core problems. For example, a call is recorded, and it takes a lot of time to analyze. Summarization could be a solution, whether it’s AI or a manual service. As a Product Manager, the problem is the focus. That’s 80% of what a Product Manager should do—understanding and defining the problem. The solution, including AI, is just another layer. What has changed is that it's now easier to create models. You can iterate quickly and release an MVP to get feedback fast. But the core process remains the same.

[00:08:32] Raza Habib:
What about the literal process or the team involved? Are you doing prompt engineering, fine-tuning models, or using machine learning engineers? How does it work in practice?

[00:08:45] Raz Nussbaum:
We do all of that. Once the problem is well-defined, we discuss solutions with the research team, which advises on the best approach—whether it’s a general LLM or a custom fine-tuned model. We involve engineers in the conversation as well. In some cases, we use general LLMs, while in others, we need to fine-tune a model to meet customer expectations. Our research team leads the ideation, and we also have prompt engineers who specialize in writing prompts well. I also write prompts myself and work closely with the team on optimizing them.

[00:10:57] Raza Habib:
That’s an interesting point. You mentioned earlier that the job of a Product Manager hasn’t changed much, but it seems like you’re now involved in implementation with prompt engineering. Would you say that’s a big change?

[00:11:46] Raz Nussbaum:
Definitely, but it’s similar to what I did before with NLP models. I’ve always been involved in QA for AI models, understanding what outputs work and what doesn’t. Now, it’s easier because I can suggest prompt changes directly. In the past, making changes required drastic steps—relabeling data, different datasets, etc. Now, it’s as simple as tweaking the prompt. We iterate quickly and compare versions to decide which works best.

[00:13:05] Raza Habib:
How do you handle quality assurance? What’s the process for testing, and how do you measure success?

[00:13:12] Raz Nussbaum:
It depends on the feature. We like to have templates, but we also adapt based on the task. For example, when migrating from one LLM vendor to another, we took a test set of calls, ensured diversity across industries, and had human reviewers rate the outputs. We compared outputs from the old vendor and the new vendor, rated them on a scale from one to five, and selected the better one. We do this for critical features used by thousands of users daily.

[00:14:39] Raza Habib:
Do you do this every time you change model providers?

[00:14:43] Raz Nussbaum:
For critical features that impact many users, yes, we run thorough tests. We need to feel comfortable before making changes. It’s not enough for a researcher to say it works; we need to see results.

[00:15:23] Raza Habib:
What about post-deployment? Once the feature is live, how do you monitor quality?

[00:15:33] Raz Nussbaum:
We have feedback mechanisms like thumbs up/thumbs down after the feature is live. It helps us collect more data at scale. However, if I did my job well during beta testing and spoke with customers, I capture about 95% of the feedback. Once we launch to GA, the feedback we receive shouldn’t surprise us. If it does, it means I didn’t do a good enough job.

[00:17:30] Raza Habib:
One of the themes in this conversation is that building with generative AI isn’t fundamentally different from building AI products before. Is there anything harder with generative AI, or are there new challenges?

[00:17:53] Raz Nussbaum:
The challenge is competition. Everyone can do everything now; there aren't many entry barriers for certain features. It seems that way, at least. That's one challenge for AI companies—it's easier for others to copy what you’re doing. The second challenge is the pace of advancement. It’s hard to predict what will happen next week in the AI space. So at Gong, our strategy is to release things fast, interact with the market, and adapt to new solutions. Core problems don’t change, but the solutions do. So, we have to implement quickly and then improve as we go. That way, when the next solution comes out, we can implement it faster with all the feedback we’ve collected. You need to interact with the market constantly and embrace change.

[00:19:39] Raza Habib:
That makes a lot of sense. Just to make sure I’ve understood, your advice is to move quickly and learn from real feedback. The space is changing so fast that the need to strategize has shifted to the need to ship fast and learn in production.

[00:20:07] Raz Nussbaum:
Yes, more than ever. You need to learn from the market. AI is just one part of the solution. The full application, including user experience, is what makes the difference. If you can build an app customers love, that’s the key. AI might be the core of the solution, but the entire user flow and experience are what matter in the end.

[00:20:56] Raza Habib:
What advice would you give to a product manager or team lead who is building with generative AI for the first time? What should they consider? How did you get up to speed?

[00:21:08] Raz Nussbaum:
First, stop talking and start prompting. If you're not doing prompt engineering and QA for outputs, you don't know what you're talking about. Go open accounts on platforms like Anthropic or GPT, and start experimenting. Understand the process, learn how prompts work, and get hands-on experience. The second piece of advice is to always test with real customer data, not mock-ups. Before we even have a UI, we let customers interact with the outputs in a developer notebook. The designer and engineers are in the room to observe how customers react to the outputs, and we gather feedback. The key is real data and real feedback, even before any UI is built.

[00:23:17] Raza Habib:
That resonates with our experience. You've been in the AI space for a while. Gong was an AI company before LLMs, and it’s adopted generative AI quickly. How do you see your product evolving in the future? How will people sell differently with Gong three to five years from now?

[00:23:40] Raz Nussbaum:
I think AI's biggest contribution is efficiency. Gong serves go-to-market teams, and most of their challenges involve spending time on non-selling activities. A lot of the tasks they need to do can now be handled by AI. For example, feeding the CRM, writing follow-up emails, summarizing calls, preparing handover documents—AI can take care of 90% of that. I see a future where AI handles the heavy lifting, leaving human teams to focus on the important, value-added work like building connections with prospects and closing deals. It’s a huge opportunity for humanity to let AI take over the repetitive tasks, while we focus on the more meaningful work.

[00:25:13] Raza Habib:
The models have improved a lot recently. Do you think humans will still be needed in the loop, or will AI eventually take over entirely?

[00:25:32] Raz Nussbaum:
It’s a tricky question because it’s also a matter of psychology. Humans may not feel comfortable leaving certain tasks entirely to AI. For example, would you trust AI with a critical medical diagnosis? Some people might say yes, but others might want a human doctor’s opinion. It comes down to risk tolerance and mistake tolerance. People need to feel confident in the technology before they’re willing to trust it with major decisions.

[00:26:23] Raza Habib:
But as AI improves, isn't there a point where it might be riskier to rely on humans instead of AI?

[00:26:33] Raz Nussbaum:
I had a personal experience with this. I was on vacation with my family, and one of my kids had something on their skin. I could either use our travel insurance and talk to a doctor or upload an image of the wound to GPT and get an opinion from the AI. I ended up doing both, and the AI gave me almost the same diagnosis as the doctor. That was pretty shocking, even to me. The difference was the AI was available immediately, while the doctor wasn’t. It’s surprising how much AI can already handle, but we’re still in the early stages. I think over time, we’ll become more comfortable with AI doing more of the work.

[00:28:04] Raza Habib:
Absolutely. Well, Raz, we’re coming up on time. One question I always like to ask: Do you consider AI today to be overhyped or underhyped?

[00:28:19] Raz Nussbaum:
Underhyped. The opportunities are endless with today’s AI models. We’re still just scratching the surface of what AI can do. I think people don’t yet fully understand the potential. New capabilities are emerging so quickly, and there are so many use cases that will be unlocked. It’s hard to predict what’s coming next. Even as an expert, I don’t know what new opportunities tomorrow’s models will bring. So, yes, I believe AI is underhyped, not overhyped. The sky's the limit.

[00:29:17] Raza Habib:
I tend to agree. It’s been a great conversation learning about how you're building AI at Gong. Thanks so much, Raz.

[00:29:24] Raz Nussbaum:
Appreciate the time. Thank you.

[00:29:27] Raza Habib:
That’s it for today’s conversation on High Agency. I’m Raza Habib, and I hope you enjoyed the episode. If you did, please take a moment to rate and review us on your favorite podcast platform—Spotify, Apple Podcasts, or wherever you listen—and subscribe. It really helps us reach more AI builders like you. For show notes and more episodes, visit humanloop.com/podcast. If today’s conversation sparked any ideas or insights, I’d love to hear from you. Email me at raza@humanloop.com or find me on X at @RasRascal.