(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

In this podcast, Krish explores the various connectors available in Confluent Cloud. He starts by recapping the previous podcasts and the basics of Confluent Cloud. Krish then focuses on connectors, explaining their value and why they can reduce the need for writing code. He explores different connectors, such as the data gen source connector and the MongoDB Atlas connectors. Krish also discusses different data formats, including Avro, Protobuf, and JSON. He briefly touches on implementing custom connectors. Krish explores the topic of connectors in Confluent Cloud. He discusses the process of creating connectors and the different types of connectors available. Krish also delves into configuring connectors and defining configuration parameters. He explores the concept of custom connector configuration and the use of connector properties files. Krish then explores existing connectors, such as the HTTP source and sync connectors, and discusses the process of publishing custom connectors. He concludes by mentioning the Confluent CLI for managing connectors.

Takeaways

  • Connectors in Confluent Cloud provide value by reducing the need for writing code.
  • Different connectors are available for various data sources and destinations, such as MongoDB, Amazon S3, and Elasticsearch.
  • Data formats like Avro, Protobuf, and JSON can be used with connectors.
  • Implementing custom connectors allows for more flexibility and integration with specific systems.
  • Connectors enable seamless data integration and propagation between different systems. Connectors in Confluent Cloud allow for seamless integration with various systems and services.
  • Custom connectors can be created and published to Confluent Cloud.
  • Configuration parameters for connectors can be defined and managed.
  • The Confluent CLI provides a command-line interface for managing connectors.

Chapters

00:00 Introduction

00:35 Recap of Previous Podcasts

01:05 Focus on Connectors in Confluent Cloud

02:16 Exploring Data Gen Source Connector

03:43 Different Formats: Avro, Protobuf, JSON

08:07 Differences Between Avro and Protobuf

10:03 Exploring Other Connectors

11:14 Using MongoDB Atlas Connectors

12:08 Testing Different Formats with Connectors

13:36 Handling Avro Format with Consumer

16:58 Exploring More Connectors: Snowflake, Amazon S3, Elasticsearch

20:33 Implementing Custom Connectors

27:31 Exploring More Connectors: Salesforce, Oracle, Jira

35:16 Exploring More Connectors: SQL Server, MySQL

38:43 Implementing Custom Connectors

43:24 Exploring More Connectors: Kafka, File

46:20 Understanding Connector Implementation

49:06 Creating Custom Connectors

50:00 Summary and Conclusion

50:59 Creating Connectors

52:04 Configuring Connectors

54:00 Custom Connector Configuration

56:08 Defining Configuration Parameters

57:38 Configuration Properties

59:49 Self-Managed Connectors

01:00: 27 Connector Properties File

01:01:28 Creating Custom Connectors

01:02: 09 Publishing Custom Connectors

01:03: 37Existing Connectors

01:04: 14HTTP Source Connector

01:06:40 HTTP Sync Connector

01:08:34 Other Connectors

01:10:34 Managing Connectors

01:12:14 Confluent CLI

Snowpal Products

Jaksot(411)

While it is completely natural to look for help online, do not expect solutions to be handed to you on a silver platter!

While it is completely natural to look for help online, do not expect solutions to be handed to you on a silver platter!

Every problem is unique even when it is not. So, while it is not just natural but also prudent to look for help online from the larger community when you are stuck and need some guidance, I find that too often developers look for solutions to be handed to them on a silver platter. That's not going to happen!   Manage Projects on https://snowpal.com!

26 Huhti 20211min

Attention to Details is always important but more so when you are doing UI/UX work

Attention to Details is always important but more so when you are doing UI/UX work

## Attention to Detail We all recognize the fact the pixel pushing is time consuming, and if you are a smallish team trying to move mountains, it could be a bit of a pain as it consumes a lot of time.  But, one thing is for certain. You are going to save time (and a lot of it) if you did it right the first time around rather than add TODOs in the code to revisit them later.  We've made a number of UI/UX enhancements to our Web Platform, and we hope to deploy them this weekend. Check them out on https://snowpal.com! Managing projects should be quick, simple and fun. Not complex, boring or overwhelming!

9 Huhti 20211min

How much should the past drive the future?

How much should the past drive the future?

One school of thought is if you've done a certain way a certain way in the past and it worked, there's no good reason to change it. Another school of thought is that one should always question how we may have approached a certain problem in the past and continually look for ways to improve it. While I don't know if one is better than the other, I know where I stand! Manage your projects on https://snowpal.com!

7 Huhti 20211min

Integrated Development Environments (or IDEs): How important are they?

Integrated Development Environments (or IDEs): How important are they?

IDEs play a huge role in not just our productivity as developers but also our happiness levels (given the amount of time we literally spend using them). What can I say about them in under 100 seconds that you may not already know? :)   Manage your projects on https://snowpal.com.

6 Huhti 20211min

(Tech Lightyear Series) Software Development in under 100 seconds

(Tech Lightyear Series) Software Development in under 100 seconds

Of the many pieces of (constructive) feedback I've received over the years (in both personal & professional life), one that has been a common occurrence is "Can you be more succinct?". And while I've always tried to do that (and also want to believe I've made some good strides in that aspect), I realize I've ways to go. So, here is another attempt (read: challenge) at getting closer to that finish line. I'll be doing a series that I've ambitiously named Tech Lightyear (as in, how much can we learn if we traveled at the speed of light!). And the requirement I've set myself is: no more than 90-100 seconds, tops. Whatever I want to say, that's how long I get. And we shall see how goes!   Manage your projects on https://snowpal.com.

5 Huhti 20211min

Code Commenting: Mandatory or Optional?

Code Commenting: Mandatory or Optional?

We all agree (I think!) that code commenting is essential. But, how much of it do we need, and how do we go about it? And why it is stack agnostic.   Manage your projects on https://snowpal.com.

4 Huhti 20214min

Firebase Messaging - Sending push notifications on a Flutter App

Firebase Messaging - Sending push notifications on a Flutter App

Setting up push notifications on a mobile app takes a bit of time the first time, and then onwards, you shouldn't have to do much. Ideally.   Manage your projects on https://snowpal.com.

31 Maalis 20217min

(Feat. Special Guest!) Pros & Cons of Technology Choices (Part 3/3)

(Feat. Special Guest!) Pros & Cons of Technology Choices (Part 3/3)

(Part 3/3) When you are building a new app, you are bound to assess a variety of options available to you. Some of these choices are going to stick around with you for a while so it is imperative to make the right choices (with the information available at that point).   Manage your projects on https://snowpal.com.

28 Maalis 20217min