Is Apache Spark Too Costly? An Amazon Engineer Tells His Story

Is Apache Spark Too Costly? An Amazon Engineer Tells His Story

Is Apache Spark too costly? Amazon Principal Engineer Patrick Ames tackled this question during an interview with The New Stack Makers, sharing insights into transitioning from Spark to Ray for managing large-scale data. Ames, described as a "go-to" engineer for exabyte-scale projects, emphasized a goal-driven approach to solving complex engineering problems, from simplifying daily chores to optimizing software solutions.

Initially, Spark was chosen at Amazon for its simplicity and open-source flexibility, allowing efficient merging of data with minimal SQL code. The team leveraged Spark in a decoupled architecture over S3 storage, scaling it to handle thousands of jobs daily. However, as data volumes grew to hundreds of terabytes and beyond, Spark’s limitations became apparent. Long processing times and high costs prompted a search for alternatives.

Enter Ray—a unified framework designed for scaling AI and Python applications. After experimentation, Ames and his team noted significant efficiency improvements, driving the shift from Spark to Ray to meet scalability and cost-efficiency needs.

Learn more from The New Stack about Apache Spark and Ray:

Amazon to Save Millions Moving From Apache Spark to Ray

How Ray, a Distributed AI Framework, Helps Power ChatGPT

Join our community of newsletter subscribers to stay on top of the news and at the top of your game.

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(300)

Why MotherDuck refuses to fork DuckDB

Why MotherDuck refuses to fork DuckDB

At a recent MCP developer summit, The New Stack spoke with Till Döhmen, AI lead atMotherDuck, about the company’s growing role in the evolving DuckDB ecosystem. Backed by investors includingTomasz Tun...

27 Mai 27min

JetBrains is selling independence as the rest of AI coding picks sides

JetBrains is selling independence as the rest of AI coding picks sides

JetBrains is positioning itself as the last major independent AI coding-tool vendor in a market increasingly tied to hyperscalers and foundation model labs. Speaking at Google Cloud Next, JetBrains VP...

21 Mai 26min

Why Block handed Goose to the Linux Foundation

Why Block handed Goose to the Linux Foundation

What began as an internal developer tool atBlockhas evolved into a broader open-source initiative with industry backing. Goose, Block’s AI coding agent, followed a path similar to Amazon’s transformat...

15 Mai 19min

Fivetran's CPO: closed data stacks won't survive the agent era

Fivetran's CPO: closed data stacks won't survive the agent era

At Google Cloud Next 2026, Fivetran Chief Product Officer Anjan Kundavaram argued that enterprise data systems are unprepared for the scale of AI-driven analytics. Unlike humans, AI agents can generat...

13 Mai 22min

The new FinOps problem isn't cloud bills

The new FinOps problem isn't cloud bills

At Google Cloud Next 2026, Finout co-founder and CEO Roi Ravhon and Google Cloud FinOps lead Pathik Sharma discussed how FinOps is rapidly evolving for the AI era. Ravhon argued that while cloud FinOp...

12 Mai 28min

How Microsoft is governing thousands of Kubernetes clusters without manual intervention

How Microsoft is governing thousands of Kubernetes clusters without manual intervention

Managing Kubernetes at fleet scale introduces significant complexity, especially as organizations expand from a few clusters to hundreds or thousands across cloud, on-premises, and edge environments. ...

7 Mai 25min

Why long-running AI agents break on HTTP and how Ably is fixing it

Why long-running AI agents break on HTTP and how Ably is fixing it

In this episode ofThe New Stack Makers, Matthew O’Riordan, CEO of Ably, explains how infrastructure originally built for human collaboration is now well-suited for long-running AI agents. While Ably i...

6 Mai 31min

Why the Linux Foundation adopted MCP, with Jim Zemlin and Mazin Gilbert

Why the Linux Foundation adopted MCP, with Jim Zemlin and Mazin Gilbert

Agentic AI is advancing rapidly, with open-source projects racing to keep pace with real-world deployment. To accelerate progress, the Linux Foundation consolidated key technologies—Model Context Prot...

6 Mai 32min

Populært innen Politikk og nyheter

giver-og-gjengen-vg
aftenpodden
aftenpodden-usa
popradet
forklart
stopp-verden
fotballpodden-2
rss-gukild-johaug
det-store-bildet
nokon-ma-ga
hanna-de-heldige
lydartikler-fra-aftenposten
dine-penger-pengeradet
rss-espen-lee-usensurert
rss-ness
aftenbla-bla
rss-penger-polser-og-politikk
chit-chat-med-helle
e24-podden
frokostshowet-pa-p5