The New Stack Podcast12 Sep 2023

Kinesis, Kafka and Amazon Managed Service for Apache Flink

Apache Flink is an open-source framework and distributed processing engine designed for data analytics. It excels at handling tasks such as data joins, aggregations, and ETL (Extract, Transform, Load) operations. Moreover, it supports advanced real-time techniques like complex event processing.

In this episode, Deepthi Mohan and Nagesh Honnalii from AWS discussed Apache Flink and the Amazon Managed Service for Apache Flink (MSF) with our host, Alex Williams. MSF is a service that caters to customers with varying infrastructure preferences. Some prefer complete control, while others want AWS to handle all infrastructure-related aspects.

Use cases for MSF can be grouped into three categories. First, there's streaming ETL, which involves tasks like log aggregation for later auditing. Second, it supports real-time analytics, enabling customers to create dashboards for tasks like fraud detection. Third, it handles complex event processing, where data from multiple sources is joined and aggregated to extract meaningful insights.

The origins of MSF trace back to the evolution of real-time data services within AWS. In 2013, AWS introduced Amazon Kinesis, while the open-source community developed Apache Kafka. These services paved the way for MSF by highlighting the need for real-time data processing.

To provide more flexibility, AWS launched Kinesis Data Analytics in 2016, allowing customers to write code in JVM-based languages like Java and Scala. In 2018, AWS decided to incorporate Apache Flink into its Kinesis Data Analytics offering, leading to the birth of MSF.

Today, thousands of customers use MSF, and AWS continues to enhance its offerings in the real-time data processing space, including the launch of Amazon MSK (Managed Streaming for Apache Kafka). To align with its foundation on Flink, AWS rebranded Kinesis Data Analytics for Apache Flink to Amazon Managed Service for Apache Flink, making it clearer for customers.

Learn more from The New Stack about AWS and Apache Flink:

Apache Flink for Real Time Data Analysis

Apache Flink for Unbounded Data Streams

3 Reasons Why You Need Apache Flink for Stream Processing

Denne episoden er hentet fra en åpen RSS-feed og er ikke publisert av Podme. Den kan derfor inneholde annonser.

Episoder(300)

Why CPUs still matter in the age of AI agents

As AI evolves from conversational chatbots to autonomous agents, CPUs are becoming an increasingly important part of the infrastructure equation. In this episode, The New Stack speaks with Bhumik Pate...

11 Aug 26min

Why Doist Says Less AI Can Deliver More

Doist CTO Gonçalo Silva says AI is reshaping software development, but success depends on restraint rather than rapid feature expansion. Instead of chasing every AI capability, Doist prioritizes “subt...

31 Jul 35min

Why your company should (try to) build its own AI SRE

As AI coding agents accelerate software development, they also create new challenges for site reliability engineers (SREs), who are increasingly responsible for debugging systems that no single human ...

30 Jul 30min

Nvidia

In this episode with The New Stack Agents, Frederic Lardinois, NVIDIA’s Joey Conway says advances in AI over the past year have dramatically improved the capabilities of local models, making them prac...

23 Jul 32min

Meet Brain, the AI that decides when Azure is officially down

In this episode, Mark Russinovich, CTO of Microsoft Azure revealed Brain, the AI-powered AIOps system that continuously monitors Azure’s health, detects incidents, identifies root causes, and increasi...

14 Jul 19min

What comes after attention? This startup says it already knows.

Subquadratic is beginning to back up its ambitious claims with benchmarks and third-party validation for its SubQ 1.1 Small model, which uses its proprietary Sparse Attention (SSA) architecture to dra...

7 Jul 20min

“The harness is where the hard work is”: Harness bets on agents that enterprises can trust in production

Harness has introduced Autonomous Worker Agents, a new capability that allows enterprises to replace rigid CI/CD pipeline scripts with AI agents that can deploy applications, run tests, and perform se...

2 Jul 19min

Public cloud vs. on-prem: Summit on where each workload belongs

More than two decades after AWS helped usher in the public cloud era, many organizations are reassessing whether a cloud-first strategy still delivers the cost and operational benefits it once promise...

25 Jun 36min