(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

In this podcast, Krish explores the various connectors available in Confluent Cloud. He starts by recapping the previous podcasts and the basics of Confluent Cloud. Krish then focuses on connectors, explaining their value and why they can reduce the need for writing code. He explores different connectors, such as the data gen source connector and the MongoDB Atlas connectors. Krish also discusses different data formats, including Avro, Protobuf, and JSON. He briefly touches on implementing custom connectors. Krish explores the topic of connectors in Confluent Cloud. He discusses the process of creating connectors and the different types of connectors available. Krish also delves into configuring connectors and defining configuration parameters. He explores the concept of custom connector configuration and the use of connector properties files. Krish then explores existing connectors, such as the HTTP source and sync connectors, and discusses the process of publishing custom connectors. He concludes by mentioning the Confluent CLI for managing connectors.

Takeaways

  • Connectors in Confluent Cloud provide value by reducing the need for writing code.
  • Different connectors are available for various data sources and destinations, such as MongoDB, Amazon S3, and Elasticsearch.
  • Data formats like Avro, Protobuf, and JSON can be used with connectors.
  • Implementing custom connectors allows for more flexibility and integration with specific systems.
  • Connectors enable seamless data integration and propagation between different systems. Connectors in Confluent Cloud allow for seamless integration with various systems and services.
  • Custom connectors can be created and published to Confluent Cloud.
  • Configuration parameters for connectors can be defined and managed.
  • The Confluent CLI provides a command-line interface for managing connectors.

Chapters

00:00 Introduction

00:35 Recap of Previous Podcasts

01:05 Focus on Connectors in Confluent Cloud

02:16 Exploring Data Gen Source Connector

03:43 Different Formats: Avro, Protobuf, JSON

08:07 Differences Between Avro and Protobuf

10:03 Exploring Other Connectors

11:14 Using MongoDB Atlas Connectors

12:08 Testing Different Formats with Connectors

13:36 Handling Avro Format with Consumer

16:58 Exploring More Connectors: Snowflake, Amazon S3, Elasticsearch

20:33 Implementing Custom Connectors

27:31 Exploring More Connectors: Salesforce, Oracle, Jira

35:16 Exploring More Connectors: SQL Server, MySQL

38:43 Implementing Custom Connectors

43:24 Exploring More Connectors: Kafka, File

46:20 Understanding Connector Implementation

49:06 Creating Custom Connectors

50:00 Summary and Conclusion

50:59 Creating Connectors

52:04 Configuring Connectors

54:00 Custom Connector Configuration

56:08 Defining Configuration Parameters

57:38 Configuration Properties

59:49 Self-Managed Connectors

01:00: 27 Connector Properties File

01:01:28 Creating Custom Connectors

01:02: 09 Publishing Custom Connectors

01:03: 37Existing Connectors

01:04: 14HTTP Source Connector

01:06:40 HTTP Sync Connector

01:08:34 Other Connectors

01:10:34 Managing Connectors

01:12:14 Confluent CLI

Snowpal Products

Jaksot(411)

How to set up Campaigns in Facebook, Pinterest, App Store, etc.

How to set up Campaigns in Facebook, Pinterest, App Store, etc.

Well, there's a lot to learn here, and I've hardly scratched the surface. But my hope is that it at least gives you a starting point.

19 Tammi 20219min

I do a lot of our Software Design work while walking!

I do a lot of our Software Design work while walking!

When I am sitting in front of my machine, I want to be coding (or doing something that I really need the machine for). Design work doesn't necessarily fall in that category for me - at least not all the time.

15 Tammi 20216min

Should your Product Manager have to be cognizant of your Architecture?

Should your Product Manager have to be cognizant of your Architecture?

The correct answer is "No". Or, is it? It really depends on a few different factors. Let's talk about a few pros and cons of either approach.

14 Tammi 20214min

When we embarked on building a product, we expected some challenges. But not a few others.

When we embarked on building a product, we expected some challenges. But not a few others.

I knew we would have engineering challenges and that didn't bother me (having done this for a while). But what kept me awake were the roadblocks that were bound to be there that I had little clue about!

12 Tammi 20219min

The True Cost of Meetings - it is more than what you think

The True Cost of Meetings - it is more than what you think

How much time are you spending in meetings? And are you sure you needed to be in these meetings?

10 Tammi 20219min

One approach to create UX Designs is to draw them out quickly without relying on tools

One approach to create UX Designs is to draw them out quickly without relying on tools

If you can draw, this approach will work really well! But, even if you are like me and cannot really draw, it will still save you a ton of time if you manually drew out the UI pages to share with your development team.

31 Joulu 202010min

User Experience(UX) Designs for Web and Mobile Apps

User Experience(UX) Designs for Web and Mobile Apps

While it would save time (which is priceless for everyone, not to mention smaller startups where it is even more valuable!) to use UX designs created for the Web for Mobile Apps, and vice-versa, it is hardly ever possible (if a seamless User Experience across devices is at the top of your priority list).

26 Joulu 20207min

How to deal with the issue of Bot Sign ups?

How to deal with the issue of Bot Sign ups?

Well, as technology improves, we get smarter as humans. But, so do bots. There's any number of things you can, and should, do to ensure your app is not hit with bot (aka: fake) sign ups.

26 Joulu 20202min