(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

In this podcast, Krish explores the various connectors available in Confluent Cloud. He starts by recapping the previous podcasts and the basics of Confluent Cloud. Krish then focuses on connectors, explaining their value and why they can reduce the need for writing code. He explores different connectors, such as the data gen source connector and the MongoDB Atlas connectors. Krish also discusses different data formats, including Avro, Protobuf, and JSON. He briefly touches on implementing custom connectors. Krish explores the topic of connectors in Confluent Cloud. He discusses the process of creating connectors and the different types of connectors available. Krish also delves into configuring connectors and defining configuration parameters. He explores the concept of custom connector configuration and the use of connector properties files. Krish then explores existing connectors, such as the HTTP source and sync connectors, and discusses the process of publishing custom connectors. He concludes by mentioning the Confluent CLI for managing connectors.

Takeaways

  • Connectors in Confluent Cloud provide value by reducing the need for writing code.
  • Different connectors are available for various data sources and destinations, such as MongoDB, Amazon S3, and Elasticsearch.
  • Data formats like Avro, Protobuf, and JSON can be used with connectors.
  • Implementing custom connectors allows for more flexibility and integration with specific systems.
  • Connectors enable seamless data integration and propagation between different systems. Connectors in Confluent Cloud allow for seamless integration with various systems and services.
  • Custom connectors can be created and published to Confluent Cloud.
  • Configuration parameters for connectors can be defined and managed.
  • The Confluent CLI provides a command-line interface for managing connectors.

Chapters

00:00 Introduction

00:35 Recap of Previous Podcasts

01:05 Focus on Connectors in Confluent Cloud

02:16 Exploring Data Gen Source Connector

03:43 Different Formats: Avro, Protobuf, JSON

08:07 Differences Between Avro and Protobuf

10:03 Exploring Other Connectors

11:14 Using MongoDB Atlas Connectors

12:08 Testing Different Formats with Connectors

13:36 Handling Avro Format with Consumer

16:58 Exploring More Connectors: Snowflake, Amazon S3, Elasticsearch

20:33 Implementing Custom Connectors

27:31 Exploring More Connectors: Salesforce, Oracle, Jira

35:16 Exploring More Connectors: SQL Server, MySQL

38:43 Implementing Custom Connectors

43:24 Exploring More Connectors: Kafka, File

46:20 Understanding Connector Implementation

49:06 Creating Custom Connectors

50:00 Summary and Conclusion

50:59 Creating Connectors

52:04 Configuring Connectors

54:00 Custom Connector Configuration

56:08 Defining Configuration Parameters

57:38 Configuration Properties

59:49 Self-Managed Connectors

01:00: 27 Connector Properties File

01:01:28 Creating Custom Connectors

01:02: 09 Publishing Custom Connectors

01:03: 37Existing Connectors

01:04: 14HTTP Source Connector

01:06:40 HTTP Sync Connector

01:08:34 Other Connectors

01:10:34 Managing Connectors

01:12:14 Confluent CLI

Snowpal Products

Jaksot(410)

Here is how to succeed in your first job

Here is how to succeed in your first job

Landed your first Software Development job? - First of all, Congratulations! - Next, if you are feeling a tiny bit anxious, don't be.   My thoughts (based on my own experience) on how you could go about it to be successful.   ## Be Organized. Be Happy. Be on Pitch.   1. Our SaaS Product - https://pitch.snowpal.com 2. LIVE (Web-only) demo - https://lnkd.in/eE9dvjw 3. App Store: https://lnkd.in/eZ6U_du 4. Play Store: https://lnkd.in/ekYfkxS

23 Loka 20204min

Need to hop on another call?

Need to hop on another call?

If you've ever had to get off the current call to hop on another call, you will *not* be able to relate to this video ;)   1. Our SaaS Product - https://pitch.snowpal.com 2. LIVE (Web-only) demo - https://consulting.snowpal.com/pitch 3. App Store: https://apps.apple.com/us/app/snowpal/id1502153924 4. Play Store: https://play.google.com/store/apps/details?id=com.snowpal.pitch   Be Organized. Be Happy. Be on Pitch.

23 Loka 20202min

On any given day, how much time do you spend in meetings?

On any given day, how much time do you spend in meetings?

Now that a lot of the world is working remotely, there are probably even more meetings than there already were, and there used to a lot to begin with!   Keep an eye on the time spent in meetings. i. Do you need to have as many meetings as you do? ii. Can they have fewer people? iii. Do you have a sense for the collective human hours spent in them?   ## Snowpal Pitch   You can track these things on our SaaS platform as well!   1. Our SaaS Product - https://pitch.snowpal.com 2. LIVE (Web-only) demo - https://consulting.snowpal.com/pitch 3. App Store: https://apps.apple.com/us/app/snowpal/id1502153924 4. Play Store: https://play.google.com/store/apps/details?id=com.snowpal.pitch   Be Organized. Be Happy. Be on Pitch.

10 Loka 20207min

Process Efficiency - keep an eye on the drops that are wasted

Process Efficiency - keep an eye on the drops that are wasted

As they say, every drop makes an ocean. I've noticed time and again that organizations don't keep an eye out for the "little" time that is wasted here and there. It all adds up and no matter how big you are, it is bound to come back and bite you unless you have a concrete plan to handle it.

6 Loka 20207min

Monolithic Apps vs Micro Services - compare and contrast with an example

Monolithic Apps vs Micro Services - compare and contrast with an example

While there may be some arguments about what exactly constitutes a micro service, let's take a simple use case and try to understand it better.

27 Syys 202011min

Optimize the performance of your Web and Mobile Apps (Series: 4/N)

Optimize the performance of your Web and Mobile Apps (Series: 4/N)

You want your pages to render as fast as they can. Most times, it is easier to bump up your hardware but that merely hides the problem, it doesn't fix it.

27 Syys 202014min

Optimize your application performance so your pages render faster (Series: 3/N)

Optimize your application performance so your pages render faster (Series: 3/N)

You want your pages to render as fast as they can. Most times, it is easier to bump up your hardware but that merely hides the problem, it doesn't fix it.

19 Syys 202011min

MongoDB Aggregate Query for faster retrieval of data

MongoDB Aggregate Query for faster retrieval of data

Let's take an example MongoDB Aggregate Query to understand certain fundamentals of aggregate queries.

15 Syys 20207min