(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

In this podcast, Krish explores the various connectors available in Confluent Cloud. He starts by recapping the previous podcasts and the basics of Confluent Cloud. Krish then focuses on connectors, explaining their value and why they can reduce the need for writing code. He explores different connectors, such as the data gen source connector and the MongoDB Atlas connectors. Krish also discusses different data formats, including Avro, Protobuf, and JSON. He briefly touches on implementing custom connectors. Krish explores the topic of connectors in Confluent Cloud. He discusses the process of creating connectors and the different types of connectors available. Krish also delves into configuring connectors and defining configuration parameters. He explores the concept of custom connector configuration and the use of connector properties files. Krish then explores existing connectors, such as the HTTP source and sync connectors, and discusses the process of publishing custom connectors. He concludes by mentioning the Confluent CLI for managing connectors.

Takeaways

  • Connectors in Confluent Cloud provide value by reducing the need for writing code.
  • Different connectors are available for various data sources and destinations, such as MongoDB, Amazon S3, and Elasticsearch.
  • Data formats like Avro, Protobuf, and JSON can be used with connectors.
  • Implementing custom connectors allows for more flexibility and integration with specific systems.
  • Connectors enable seamless data integration and propagation between different systems. Connectors in Confluent Cloud allow for seamless integration with various systems and services.
  • Custom connectors can be created and published to Confluent Cloud.
  • Configuration parameters for connectors can be defined and managed.
  • The Confluent CLI provides a command-line interface for managing connectors.

Chapters

00:00 Introduction

00:35 Recap of Previous Podcasts

01:05 Focus on Connectors in Confluent Cloud

02:16 Exploring Data Gen Source Connector

03:43 Different Formats: Avro, Protobuf, JSON

08:07 Differences Between Avro and Protobuf

10:03 Exploring Other Connectors

11:14 Using MongoDB Atlas Connectors

12:08 Testing Different Formats with Connectors

13:36 Handling Avro Format with Consumer

16:58 Exploring More Connectors: Snowflake, Amazon S3, Elasticsearch

20:33 Implementing Custom Connectors

27:31 Exploring More Connectors: Salesforce, Oracle, Jira

35:16 Exploring More Connectors: SQL Server, MySQL

38:43 Implementing Custom Connectors

43:24 Exploring More Connectors: Kafka, File

46:20 Understanding Connector Implementation

49:06 Creating Custom Connectors

50:00 Summary and Conclusion

50:59 Creating Connectors

52:04 Configuring Connectors

54:00 Custom Connector Configuration

56:08 Defining Configuration Parameters

57:38 Configuration Properties

59:49 Self-Managed Connectors

01:00: 27 Connector Properties File

01:01:28 Creating Custom Connectors

01:02: 09 Publishing Custom Connectors

01:03: 37Existing Connectors

01:04: 14HTTP Source Connector

01:06:40 HTTP Sync Connector

01:08:34 Other Connectors

01:10:34 Managing Connectors

01:12:14 Confluent CLI

Snowpal Products

Jaksot(412)

(Part 2/2) Podcast Collaboration: Introducing my first 2 guests

(Part 2/2) Podcast Collaboration: Introducing my first 2 guests

(Part 2) 2 podcast collaborations in - time to share a few thoughts! My first podcast was with ⁠Matthew Davidchuk⁠, a brilliant engineer. And no surprises - 90 minutes later, I walked out a slightly better engineer. My second podcast was with ⁠Magali Pelissier⁠, a brilliant product leader. And no surprises again - 120 minutes later, I walked out a slightly better product manager. (The podcasts are in "post production", so should be published shortly) Look out for more podcasts with Matthew and Magali, as the first podcast has turned into a series! Plus, my next guest is a very special one from New Delhi, and we will be discussing a 3rd topic (hint: there may well be a hands-on, LIVE coding component to that conversation). #snowpal #podcast

30 Marras 20234min

(Part 1/2) Podcast Collaboration: Introducing my first 2 guests

(Part 1/2) Podcast Collaboration: Introducing my first 2 guests

(Part 1) 2 podcast collaborations in - time to share a few thoughts! My first podcast was with ⁠Matthew Davidchuk⁠, a brilliant engineer. And no surprises - 90 minutes later, I walked out a slightly better engineer. My second podcast was with ⁠Magali Pelissier⁠, a brilliant product leader. And no surprises again - 120 minutes later, I walked out a slightly better product manager. (The podcasts are in "post production", so should be published shortly) Look out for more podcasts with Matthew and Magali, as the first podcast has turned into a series! Plus, my next guest is a very special one from New Delhi, and we will be discussing a 3rd topic (hint: there may well be a hands-on, LIVE coding component to that conversation). #snowpal #podcast

30 Marras 20235min

Porting over 250 podcasts from one platform to another

Porting over 250 podcasts from one platform to another

Scalability isn't limited to applications in production. No matter what you do, you want to ensure that it is a scalable approach. Short of it, the exact same activity is going to take a lot longer. Besides, what's interesting is that making even tiny changes to your approach tends to go a long way in enhancing overall productivity. In this podcast, I'll share the approach we took to port hundreds of videos from one platform to another, and why we were able to accomplish that in shy of 10 hours. #snowpal aws.snowpal.com learn.snowpal.com

26 Marras 202312min

Do you love what you do? Do you feel fulfilled at work?

Do you love what you do? Do you feel fulfilled at work?

And are you doing more of it? Here's my take on a Venn Diagram you've most likely seen that illustrates the implications of our career choices beautifully. #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 20236min

ETA: What does your team base their estimates on? And how accurate are they?

ETA: What does your team base their estimates on? And how accurate are they?

It is one thing to provide an ETA for any given task, but it is completely another to provide a meaningful, reasonably accurate, and accountable ETA. Ask for ETAs but take them with a grain of salt. Focus more on solving the problem efficiently and less on the actual time it will take to solve that. #projectmanagement Manage your projects on https://snowpal.com. Keep it simple.

24 Marras 20238min

Waterfall vs Scrum vs Kanban vs Agile

Waterfall vs Scrum vs Kanban vs Agile

If you want to learn the theoretical differences between these terms, you are better off checking it out on the web. Just do a search and you'll find a lot of hits. But, if you want to get a sense of the more practical differences, and how you should apply it in the context of your specific project, then you may want to give this podcast a listen. #projectmanagement Manage your projects on https://snowpal.com. Keep it simple.

24 Marras 202313min

Adding Scoped Bookmarks on Snowpal Pitch using the Relations Feature!

Adding Scoped Bookmarks on Snowpal Pitch using the Relations Feature!

Pitch has always supported a Linking feature that allows you to structure your content any which way you want (aka, create content once, use it any number of times). Relations is a new feature that is another feather on the cap, and it works beautifully whether you use it with Linking, or simply by itself. #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 20235min

Building a Mobile App - Part I (Starting to build a native mobile app, and wondering which way to go?)

Building a Mobile App - Part I (Starting to build a native mobile app, and wondering which way to go?)

If you are starting to build a native mobile app and are a little unsure of how to proceed, this podcast (the first in this series) might offer you some guidance. #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 20233min