(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

In this podcast, Krish explores the various connectors available in Confluent Cloud. He starts by recapping the previous podcasts and the basics of Confluent Cloud. Krish then focuses on connectors, explaining their value and why they can reduce the need for writing code. He explores different connectors, such as the data gen source connector and the MongoDB Atlas connectors. Krish also discusses different data formats, including Avro, Protobuf, and JSON. He briefly touches on implementing custom connectors. Krish explores the topic of connectors in Confluent Cloud. He discusses the process of creating connectors and the different types of connectors available. Krish also delves into configuring connectors and defining configuration parameters. He explores the concept of custom connector configuration and the use of connector properties files. Krish then explores existing connectors, such as the HTTP source and sync connectors, and discusses the process of publishing custom connectors. He concludes by mentioning the Confluent CLI for managing connectors.

Takeaways

  • Connectors in Confluent Cloud provide value by reducing the need for writing code.
  • Different connectors are available for various data sources and destinations, such as MongoDB, Amazon S3, and Elasticsearch.
  • Data formats like Avro, Protobuf, and JSON can be used with connectors.
  • Implementing custom connectors allows for more flexibility and integration with specific systems.
  • Connectors enable seamless data integration and propagation between different systems. Connectors in Confluent Cloud allow for seamless integration with various systems and services.
  • Custom connectors can be created and published to Confluent Cloud.
  • Configuration parameters for connectors can be defined and managed.
  • The Confluent CLI provides a command-line interface for managing connectors.

Chapters

00:00 Introduction

00:35 Recap of Previous Podcasts

01:05 Focus on Connectors in Confluent Cloud

02:16 Exploring Data Gen Source Connector

03:43 Different Formats: Avro, Protobuf, JSON

08:07 Differences Between Avro and Protobuf

10:03 Exploring Other Connectors

11:14 Using MongoDB Atlas Connectors

12:08 Testing Different Formats with Connectors

13:36 Handling Avro Format with Consumer

16:58 Exploring More Connectors: Snowflake, Amazon S3, Elasticsearch

20:33 Implementing Custom Connectors

27:31 Exploring More Connectors: Salesforce, Oracle, Jira

35:16 Exploring More Connectors: SQL Server, MySQL

38:43 Implementing Custom Connectors

43:24 Exploring More Connectors: Kafka, File

46:20 Understanding Connector Implementation

49:06 Creating Custom Connectors

50:00 Summary and Conclusion

50:59 Creating Connectors

52:04 Configuring Connectors

54:00 Custom Connector Configuration

56:08 Defining Configuration Parameters

57:38 Configuration Properties

59:49 Self-Managed Connectors

01:00: 27 Connector Properties File

01:01:28 Creating Custom Connectors

01:02: 09 Publishing Custom Connectors

01:03: 37Existing Connectors

01:04: 14HTTP Source Connector

01:06:40 HTTP Sync Connector

01:08:34 Other Connectors

01:10:34 Managing Connectors

01:12:14 Confluent CLI

Snowpal Products

Jaksot(412)

(Part 1/2): How important is Technology Stack?

(Part 1/2): How important is Technology Stack?

(Part 1/2): How important is Technology Stack when it comes to: - Learning - Job Satisfaction - Job Security - Career Growth #snowpal #projectmanagementKeep it simple. Keep it on https://snowpal.com.

16 Elo 20227min

(Part 2/2) What does it mean to do things at scale?

(Part 2/2) What does it mean to do things at scale?

(Part 2/2) What does it mean to do things at scale? If you are thinking this podcast is about (Performance &) Scalability, think again! #snowpal #projectmanagementKeep it simple. Keep it on https://snowpal.com.

24 Heinä 20225min

(Part 1/2) What does it mean to do things at scale?

(Part 1/2) What does it mean to do things at scale?

(Part 1/2) What does it mean to do things at scale? If you are thinking this podcast is about (Performance &) Scalability, think again! #snowpal #projectmanagementKeep it simple. Keep it on https://snowpal.com.

24 Heinä 20226min

(Part 2/2) Repo-level project management using GitHub Projects

(Part 2/2) Repo-level project management using GitHub Projects

(Part 2/2) A little bit about GitHub Projects. #snowpal #projectmanagementKeep it simple. Keep it on https://snowpal.com.

21 Heinä 20228min

(Part 1/2) Repo-level project management using GitHub Projects

(Part 1/2) Repo-level project management using GitHub Projects

(Part 1/2) A little bit about GitHub Projects. #snowpal #projectmanagementKeep it simple. Keep it on https://snowpal.com.

21 Heinä 202210min

(Part 2/2) GitHub Codespaces: What is it, and how can we use it?

(Part 2/2) GitHub Codespaces: What is it, and how can we use it?

(Part 2/2) A little bit about GitHub Codespaces. #snowpal #projectmanagement Keep it simple. Keep it on https://snowpal.com.

21 Heinä 202218min

(Part 1/2) GitHub Codespaces: What is it, and how can we use it?

(Part 1/2) GitHub Codespaces: What is it, and how can we use it?

(Part 1/2) A little bit about GitHub Codespaces. #snowpal #projectmanagement Keep it simple. Keep it on https://snowpal.com.

21 Heinä 202210min

Implement menus: How hard can it possibly be?

Implement menus: How hard can it possibly be?

Say someone asked you to implement a "tiny" feature on an iPhone Mail App.Here's the (first part of the) requirement as Business provided it (to the dev team):- User can swipe right to left and when they do, 2 menu options should appear.- When you click on one of them, it should allow you to forward the email and when you click the other, it should archive it.Sounds simple enough?It should be simple till you get to the "the devil is in the detail" part. So, before I implement it, here are some questions I would have as a developer:- Can the user only swipe from right to left, or can they do vice-versa? And if they did that, what should happen?- If the user starts swiping but doesn't complete the action till they dragged it all the way to the very left, what should happen?- If the user swiped & performed an action but realized they were on the wrong item so wanted to undo, will they be able to do it? And if so, how?- Can they perform bulk actions? If so, is that a mutually exclusive action agnostic to the individual swipe?- After they swipe on one of the emails, say if they swiped on another. Will the subsequent action need to reset the state so the first one is now un-swiped?- Do we need to support multiple email accounts where each of them is rendered as a separate category? And if we did that, what should happen when a user swipes on one of the emails in each of those sections? Will that result in a reset of all states (of all other sections)?- And there's a few more I can think of off the top of my head but you get the idea.So, now, let me ask again:How simple is the design and implementation now? Did you have a change of heart and will you need to adjust your story points?"How hard can it possibly be?" :)#snowpal #projectmanagementKeep it simple. Keep it on https://snowpal.com. And just so that you can keep it simple, we do everything in our capacity to remove all the complexities. Rome was not built in a day as they say, and Snowpal 2.0 certainly wasn't either. We have built it brick by brick for your pleasure. Or, should I say, "swipe by swipe"? :)

21 Heinä 20227min