The New Stack Podcast28 Mars 2024

LLM Observability: The Breakdown

LLM observability focuses on maximizing the utility of larger language models (LLMs) by monitoring key metrics and signals. Alex Williams, Founder and Publisher for The New Stack, and Janikiram MSV, Principal of Janikiram & Associates and an analyst and writer for The New Stack, discusses the emergence of the LLM stack, which encompasses various components like LLMs, vector databases, embedding models, retrieval systems, read anchor models, and more. The objective of LLM observability is to ensure that users can extract desired outcomes effectively from this complex ecosystem.

Similar to infrastructure observability in DevOps and SRE practices, LLM observability aims to provide insights into the LLM stack's performance. This includes monitoring metrics specific to LLMs, such as GPU/CPU usage, storage, model serving, change agents in applications, hallucinations, span traces, relevance, retrieval models, latency, monitoring, and user feedback. MSV emphasizes the importance of monitoring resource usage, model catalog synchronization with external providers like Hugging Face, vector database availability, and the inference engine's functionality.

He also mentions peer companies in the LLM observability space like Datadog, New Relic, Signoz, Dynatrace, LangChain (LangSmith), Arize.ai (Phoenix), and Truera, hinting at a deeper exploration in a future episode of The New Stack Makers.

Learn more from The New Stack about LLM and observability

Observability in 2024: More OpenTelemetry, Less Confusion

How AI Can Supercharge Observability

Next-Gen Observability: Monitoring and Analytics in Platform Engineering

Join our community of newsletter subscribers to stay on top of the news and at the top of your game.

Det här avsnittet är hämtat från ett öppet RSS-flöde och publiceras inte av Podme. Det kan innehålla reklam.

Avsnitt(300)

Dynatrace wants its AI agents to fix problems instead of adding alerts

Dynatrace's view is that AI apps should be monitored like any other app. They still run alongside older systems, including mainframes, and when something breaks, teams need to know what it cost and wh...

5 Okt 20min

Bit Cloud’s next chapter starts after the AI builds your app

For many developers, turning an AI-generated prototype into maintainable software requires more than generating code—it requires infrastructure, collaboration, testing and review. In this episode of T...

1 Okt 31min

CloudBees just committed to an AI-first pivot. Here's why it matters for enterprise DevOps teams

CloudBees CEO Mo Plassnig is leading the CI/CD company through a major transformation as generative AI reshapes software development. Returning to CloudBees eight years after joining through its acqui...

30 Sep 23min

A third option is emerging in the fight over AI and your data

The AI industry has faced a growing enterprise dilemma: companies want access to powerful proprietary AI models without risking sensitive data or intellectual property, while AI labs want to protect t...

23 Sep 30min

Drowning in AI pull requests: Harness's field CTO on code review and a Git repo built for agents

Harness Field CTO Martin Reynolds joins The New Stack to talk about what happens after coding agents start opening pull requests faster than anyone can review them. He explains how he first saw the bo...

7 Sep 26min

How to find failures without drowning in tracing data

Traces provide a detailed view of a request’s journey through data, microservices and applications, helping SREs pinpoint where failures occur and resolve issues faster. But while tracing can reduce d...

3 Sep 31min

Why CPUs still matter in the age of AI agents

As AI evolves from conversational chatbots to autonomous agents, CPUs are becoming an increasingly important part of the infrastructure equation. In this episode, The New Stack speaks with Bhumik Pate...

11 Aug 26min

Why Doist Says Less AI Can Deliver More

Doist CTO Gonçalo Silva says AI is reshaping software development, but success depends on restraint rather than rapid feature expansion. Instead of chasing every AI capability, Doist prioritizes “subt...

31 Juli 35min