Learn how LLM monitoring works, why it's important, as well as best practices and important metrics you should know.

LLM Monitoring: A Guide to Evaluating AI Applications In Production 

LLM Monitoring

Peter Gostev shares his experiences implementing LLMs at NatWest and Moonpig. He discusses creating an AI strategy, talks about the challenges of deploying LLMs in large organizations, and shares thoughts on underappreciated AI developments.

How to Create AI Strategy in Enterprises

Learn what prompt caching is, how prompt caching works, the difference between prompt caching on OpenAI and Anthropic, its benefits and challenges, and how it can be applied in AI applications

What is prompt caching? Prompt caching explained for large language models (LLMs)

Learn what prompt caching is, how it works with OpenAI and Anthropic, its benefits and challenges, and how it can be applied in AI applications.

Prompt Caching

In this episode of High Agency, we're joined by Surojit Chatterjee, former CPO of Coinbase and now CEO of Ema. Surojit unveils his audacious plan to create universal AI employees and revolutionize Fortune 1000 workforce.

Building AI Employees with Coinbase's Ex-CPO

In this episode, Hamel Husain and Shawn Wang (aka Swyx) share insights on AI engineering, literate programming, and why your AI product desperately needs evals.

Why Your AI Product Needs Evals with Hamel Husain

Learn how Tree-of-Thought (ToT) prompting enhances the reasoning of large language models by exploring multiple solution pathways. Discover how this advanced prompting technique boosts decision-making and problem-solving capabilities for complex tasks.

Tree of Thoughts Prompting (ToT) Explained

Explore how Tree-of-Thought (ToT) prompting works, why it can improve decision-making and reasoning for large language model (LLM) applications, its benefits, and its limitations.

Tree of Thoughts Prompting (ToT)

Discover the top 8 Retrieval Augmented Generation (RAG) architectures for building generative AI applications. Learn how RAG combines real-time data retrieval with large language models to create more accurate, contextually relevant responses.

8 Retrieval Augmented Generation (RAG) Architectures You Should Know in 2024

Explore the 8 most popular Retrieval Augmented Generation (RAG) architectures, understand their workflows and use cases for building generative AI applications.

Raz Nussbaum is a Senior Product Manager in AI at Gong — the leading AI platform for revenue teams. He is an absolute legend when it comes to building and scaling AI products that genuinely deliver value. In this episode, he opens up about what it takes to build successful AI products in an era where things change at lightning speed.

How AI is Changing Product Development with Gong AI

Learn what chain-of-thought prompting is, how chain-of-thought prompting works, the benefits of chain-of-thought prompting, how to use chain-of-thought prompting with large language models (LLMs)

Chain of thought prompting explained for large language models (LLMs)

Learn what chain-of-thought prompting is, how it improves large language model (LLM) reasoning, the different types of chain-of-thought prompting and other benefits and limitations.

Chain of Thought Prompting (CoT)

Sudowrite's founder James Yu discusses how AI is transforming creative writing, the challenges of building AI tools for fiction, and the future of human-AI collaboration in storytelling.

From Fiction to Reality: AI-assisted Creative Writing

LiveKit CEO Russ d'Sa discusses the challenges and opportunities in building real-time communication infrastructure for AI applications, sharing insights from working with OpenAI

Building the Nervous System for AI with Russ d'Sa from LiveKit

 Learn what fine-tuning is, how fine-tuning works, the benefits of fine-tuning and how to fine-tune a large language model (LLM)

What is fine-tuning? Fine-tuning explained for large language models (LLMs)

Learn how fine-tuning works, its benefit and how to fine-tune a large language model (LLM)

What is fine-tuning?

Lin Qiao, former PyTorch lead at Meta and current CEO of Fireworks AI, details insights on the evolution of AI frameworks, the challenges of optimizing inference for generative AI, the future of AI hardware, and open-source models.

From PyTorch to Fireworks AI: Building AI Infrastructure

In this episode, we are speaking to Paras Jain, who is building the future of AI video generation at Genmo. He details why he chose academia over an offer from Tesla, and the research-minded approach that has lead to Genmo's rapid success.

How Paras Jain is building the future of AI video creation

Eddie Kim details Gusto's GenAI integration in payroll/HR, exploring technical challenges, ethical considerations, and balancing automation with human oversight.

Lessons from Gusto's $9.5 billion journey with Eddie Kim & Ali Rowghani

In this episode, we sit down with Michael Royzen, CEO and co-founder of Phind. Michael shares insights from his journey in building the first LLM-based search engine for developers, the challenges of creating reliable AI models, and his vision for how AI will transform the work of developers in the near future.

Building the First LLM Search Engine

Jason Liu is a true Renaissance man in the world of AI. In this episode, we cut through the AI hype explore effective strategies for building valuable AI products.

The Contrarian's Guide to AI

Logan led developer relations at OpenAI before leading product on the Google AI Studio. He's been closer than anyone to developers building with LLMs and has seen behind the curtain at two frontier labs. In this week’s episode of the High Agency Podcast, Logan and I discuss the future trajectory of AI and share our insights on what AI builders can anticipate in the future.

What is beyond OpenAI?

In this week’s episode of the High Agency Podcast, I  had the pleasure of interviewing Shawn Wang, better known as Swyx. As the author of the influential essay "Rise of the AI Engineer," Swyx is at the forefront of defining the evolving field of AI engineering. We discuss the emergence of the AI engineer, the skills required and the optimal make-up of an AI team.

What is an AI Engineer?

I recently sat down with Bryan Bischof, Head of AI at Hex, to understand how Hex approaches evals and what lessons other teams can take from their experience.

LLM Evals Done Right - Lessons from Bryan Bischof of Hex AI

Over 50% of contracts at Ironclad are now negotiated by AI. I  sat with Cai Gogwilt, Co-founder and Chief Architect to understand what lessons product leaders can take from their success of building AI Agents.

Building Reliable Agents with Ironclad

Learn about key LLM benchmarks, why they should be prioritised for specific tasks, and what metrics should be used to compare LLM performance.

What are Foundation Models and How Do They Work?

Explore what foundation models are, understand how they work and learn why they’re different to traditional AI models.

Foundation Models: Explained

If you’re building an AI product, this post will help you understand if the EU Act will affect you, what you need to do to comply, and what the regulation likely means for the wider tech ecosystem.

The EU AI Act: Guide for Developers

LLM Benchmarks: Understanding Language Model Performance

Retrieval augmented generation is the leading AI framework for customizing and improving LLM-generated responses. Learn how it works and why it's important in our guide.

Retrieval Augmented Generation (RAG): Explained

An overview of evaluating LLM applications. The emerging evaluation framework, parallels to traditional software testing and some guidance on best practices.

Evaluating LLM Applications

Announcing that Humanloop is now certified as SOC 2 Type 2 compliant.

Humanloop is   SOC 2 Type II certified


Generative AI and Large Language Models (LLMs) are new to most companies. If you’re an engineering leader, it can be hard to know what skills and types of people are needed for AI projects. In this post I’d like to share what we’ve learned about the skills needed to build a great AI team.

How to Build the Right Team for Generative AI

An overview of the techniques that OpenAI recommends to get the best performance of your LLM applications. Covering best-practices in prompt engineering, retrieval-augmented generation (RAG) and fine-tuning.

How to Maximize LLM Performance

Brief overview of fine-tuning, why it’s significant, how it works on OpenAI and how Humanloop can help you to finetune your own custom models.

OpenAI Fine-tuning: GPT-3.5-Turbo

Today we’re announcing Tools as a part of Humanloop. Tools allow you to connect an LLM to any API to give it extra capabilities and access to private data.

Humanloop Tools: Connecting LLMs to resources

Last week I had the privilege to sit down with Sam Altman and 20 other developers to discuss OpenAI’s product plans. Sam was remarkably open. The discussion touched on practical developer issues as well as bigger-picture questions related to OpenAI’s mission and the societal impact of AI.  Here are the key takeaways.

OpenAI's plans according to Sam Altman

In this post, we'll explore the fundamentals of prompt engineering. We'll explain how Large Language Models (LLMs) interpret prompts to generate outputs, and provide tips and tricks to get you started prototyping and implementing LLMs quickly.

Prompt Engineering 101

We’re excited to be partnering with Stability AI, to build and release the first open-source large language model that can follow human instructions.

Humanloop partners with Stability AI to build the first open-source InstructGPT

Today we're excited to open up access to Humanloop for all developers who want to build next-generation applications with large language models like GPT-3.

Announcing: Humanloop for Large Language Models

Today, we are announcing the preview of Humanloop for Large Language Models (LLMs). Sign up to be part of the closed beta.

Coming soon: Humanloop for LLMs


Fine-tuning GPT‑3 can make it faster, cheaper and more accurate. Humanloop makes it dramatically easier to do this, while aligning the model with the preferences of your users.

How to Improve GPT‑3 with Human Feedback

Machine learning test metrics should always be calculated with credible intervals. Credible intervals give you upper and lower bounds on test performance so you know how big your test needs to be and when to trust your models. Humanloop Active Testing can give you uncertainty bounds on your test metrics and makes this easy.

Why you need to calculate error bounds on your test metrics

Human-in-the-loop AI is an automation approach which means you can quickly deploy a working model, with less data and with guaranteed quality predictions. In this post we'll explain what HITL is and how you can make use of this approach in your own AI projects.

What is human-in-the-loop AI?

Machine learning systems are born through the marriage of both code and data. Academia mostly focuses on ways to improve the learning algorithms, the _how_ of machine learning. When you come to build practical AI systems though the dataset you're training on has at least as much impact on performance as the choice of algorithm.

Why you should take a data first approach to AI

We discuss the real world performance of Open AI's GPT-3 language model and speculate on what the future of large pre-trained language models may hold.

How good is GPT-3 in practice?

The most surprising lessons about how to build tooling for machine learning, what is needed going forwards and why domain experts will play a much larger role in the future of AI.

Four lessons from a year building tools for machine learning

You can now train any Hugging Face model on your own data just by labeling and curating a small portion of that data, with early access to our Humanloop - Hugging Face integration. Label your data and we'll give you an API to your model, trained on your own data

LLM Monitoring

How to Create AI Strategy in Enterprises

Prompt Caching

Building AI Employees with Coinbase's Ex-CPO

Why Your AI Product Needs Evals with Hamel Husain

The fastest way to get AI from Playground to Production