(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

In this podcast, Krish explores the various connectors available in Confluent Cloud. He starts by recapping the previous podcasts and the basics of Confluent Cloud. Krish then focuses on connectors, explaining their value and why they can reduce the need for writing code. He explores different connectors, such as the data gen source connector and the MongoDB Atlas connectors. Krish also discusses different data formats, including Avro, Protobuf, and JSON. He briefly touches on implementing custom connectors. Krish explores the topic of connectors in Confluent Cloud. He discusses the process of creating connectors and the different types of connectors available. Krish also delves into configuring connectors and defining configuration parameters. He explores the concept of custom connector configuration and the use of connector properties files. Krish then explores existing connectors, such as the HTTP source and sync connectors, and discusses the process of publishing custom connectors. He concludes by mentioning the Confluent CLI for managing connectors.

Takeaways

  • Connectors in Confluent Cloud provide value by reducing the need for writing code.
  • Different connectors are available for various data sources and destinations, such as MongoDB, Amazon S3, and Elasticsearch.
  • Data formats like Avro, Protobuf, and JSON can be used with connectors.
  • Implementing custom connectors allows for more flexibility and integration with specific systems.
  • Connectors enable seamless data integration and propagation between different systems. Connectors in Confluent Cloud allow for seamless integration with various systems and services.
  • Custom connectors can be created and published to Confluent Cloud.
  • Configuration parameters for connectors can be defined and managed.
  • The Confluent CLI provides a command-line interface for managing connectors.

Chapters

00:00 Introduction

00:35 Recap of Previous Podcasts

01:05 Focus on Connectors in Confluent Cloud

02:16 Exploring Data Gen Source Connector

03:43 Different Formats: Avro, Protobuf, JSON

08:07 Differences Between Avro and Protobuf

10:03 Exploring Other Connectors

11:14 Using MongoDB Atlas Connectors

12:08 Testing Different Formats with Connectors

13:36 Handling Avro Format with Consumer

16:58 Exploring More Connectors: Snowflake, Amazon S3, Elasticsearch

20:33 Implementing Custom Connectors

27:31 Exploring More Connectors: Salesforce, Oracle, Jira

35:16 Exploring More Connectors: SQL Server, MySQL

38:43 Implementing Custom Connectors

43:24 Exploring More Connectors: Kafka, File

46:20 Understanding Connector Implementation

49:06 Creating Custom Connectors

50:00 Summary and Conclusion

50:59 Creating Connectors

52:04 Configuring Connectors

54:00 Custom Connector Configuration

56:08 Defining Configuration Parameters

57:38 Configuration Properties

59:49 Self-Managed Connectors

01:00: 27 Connector Properties File

01:01:28 Creating Custom Connectors

01:02: 09 Publishing Custom Connectors

01:03: 37Existing Connectors

01:04: 14HTTP Source Connector

01:06:40 HTTP Sync Connector

01:08:34 Other Connectors

01:10:34 Managing Connectors

01:12:14 Confluent CLI

Snowpal Products

Jaksot(412)

Augmented Reality: A use case that will benefit infrequent travelers such as myself!

Augmented Reality: A use case that will benefit infrequent travelers such as myself!

Have you ever wanted to be somewhere but didn't find the time to get yourself there? Have you ever wondered why you had to save up measly vacation hours and use them prudently? Have you ever wanted to bring vacation home, and make it an essential part of everyday life? If so, you may appreciate my random thoughts here that will surely become true given advancements in Augmented Reality. #projectmanagement Manage your projects on https://snowpal.com. Keep it simple.

17 Loka 20217min

Flutter upgrade issues that prevented deployment to both a real iOS device and a simulator

Flutter upgrade issues that prevented deployment to both a real iOS device and a simulator

Every time I do a flutter upgrade, I tend to keep my fingers crossed. Some days, it goes seamlessly but other days, not so much. It was one of those days a couple of days back. In this podcast, I share the issues I ran into along with the fixes. #projectmanagement Manage your projects on https://snowpal.com

27 Syys 20219min

Selling Software: Know your users, know your direct competitors, but also your indirect customers

Selling Software: Know your users, know your direct competitors, but also your indirect customers

Who are your competitors? Only those who are in your line of business? It's important to know our competitors, and we all know that. But, who are our competitors? It's not just the ones in your line of business. It is everyone who's after your clients' dollar. #projectmanagement Manage your projects on https://snowpal.com!

7 Syys 20216min

Mono repos vs Poly repos: Which one works best (for you)?

Mono repos vs Poly repos: Which one works best (for you)?

Monorepos have made a sort of a comeback, shall we say. Should you use them? Do they work? What should you watch out for? And most importantly, how do they compare to Poly-repos? #projectmanagement Manage projects on https://snowpal.com

27 Elo 202111min

Mobile App UI/UX: Design your app to suit your customer needs (I used a poorly designed app recently)

Mobile App UI/UX: Design your app to suit your customer needs (I used a poorly designed app recently)

It takes an effort, a rather significant one, to get things right. You want to build a secure, scalable, cool mobile app but it can't be at the expense of User Experience. It definitely can't. I'll share a recent horrendous experience with a mobile banking app. #projectmanagement Manage projects on https://snowpal.com!

8 Elo 20217min

Mobile first vs Web first vs (a 3rd option): Product Strategy

Mobile first vs Web first vs (a 3rd option): Product Strategy

I am sure you've heard of a mobile-first strategy and a web-first strategy but there's a 3rd option. Let's talk a bit about that. #projectmanagement Manage your projects on https://snowpal.com!

28 Heinä 20215min

Software Design vs Implementation: Where are you spending most of your time?

Software Design vs Implementation: Where are you spending most of your time?

Enough designing, start implementing already! If you find yourself in endless discussions about how to design something, one thing is for certain. You are overthinking and under delivering! #projectmanagement Manage your projects on https://snowpal.com!

28 Heinä 20211min

API Directory Structure: Define it correctly to improve backend development productivity

API Directory Structure: Define it correctly to improve backend development productivity

And by correctly, I literally mean the way it works best for you and your company. There is no single correct answer. #projectmanagement Manage projects on https://snowpal.com!

2 Heinä 20216min