(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

In this podcast, Krish explores the various connectors available in Confluent Cloud. He starts by recapping the previous podcasts and the basics of Confluent Cloud. Krish then focuses on connectors, explaining their value and why they can reduce the need for writing code. He explores different connectors, such as the data gen source connector and the MongoDB Atlas connectors. Krish also discusses different data formats, including Avro, Protobuf, and JSON. He briefly touches on implementing custom connectors. Krish explores the topic of connectors in Confluent Cloud. He discusses the process of creating connectors and the different types of connectors available. Krish also delves into configuring connectors and defining configuration parameters. He explores the concept of custom connector configuration and the use of connector properties files. Krish then explores existing connectors, such as the HTTP source and sync connectors, and discusses the process of publishing custom connectors. He concludes by mentioning the Confluent CLI for managing connectors.

Takeaways

  • Connectors in Confluent Cloud provide value by reducing the need for writing code.
  • Different connectors are available for various data sources and destinations, such as MongoDB, Amazon S3, and Elasticsearch.
  • Data formats like Avro, Protobuf, and JSON can be used with connectors.
  • Implementing custom connectors allows for more flexibility and integration with specific systems.
  • Connectors enable seamless data integration and propagation between different systems. Connectors in Confluent Cloud allow for seamless integration with various systems and services.
  • Custom connectors can be created and published to Confluent Cloud.
  • Configuration parameters for connectors can be defined and managed.
  • The Confluent CLI provides a command-line interface for managing connectors.

Chapters

00:00 Introduction

00:35 Recap of Previous Podcasts

01:05 Focus on Connectors in Confluent Cloud

02:16 Exploring Data Gen Source Connector

03:43 Different Formats: Avro, Protobuf, JSON

08:07 Differences Between Avro and Protobuf

10:03 Exploring Other Connectors

11:14 Using MongoDB Atlas Connectors

12:08 Testing Different Formats with Connectors

13:36 Handling Avro Format with Consumer

16:58 Exploring More Connectors: Snowflake, Amazon S3, Elasticsearch

20:33 Implementing Custom Connectors

27:31 Exploring More Connectors: Salesforce, Oracle, Jira

35:16 Exploring More Connectors: SQL Server, MySQL

38:43 Implementing Custom Connectors

43:24 Exploring More Connectors: Kafka, File

46:20 Understanding Connector Implementation

49:06 Creating Custom Connectors

50:00 Summary and Conclusion

50:59 Creating Connectors

52:04 Configuring Connectors

54:00 Custom Connector Configuration

56:08 Defining Configuration Parameters

57:38 Configuration Properties

59:49 Self-Managed Connectors

01:00: 27 Connector Properties File

01:01:28 Creating Custom Connectors

01:02: 09 Publishing Custom Connectors

01:03: 37Existing Connectors

01:04: 14HTTP Source Connector

01:06:40 HTTP Sync Connector

01:08:34 Other Connectors

01:10:34 Managing Connectors

01:12:14 Confluent CLI

Snowpal Products

Jaksot(412)

Building a Mobile App - Part II (React Native, Flutter, and some others)

Building a Mobile App - Part II (React Native, Flutter, and some others)

Even if you've decided to build a cross-platform native mobile app, there are still a few decisions to be made. In this podcast, I discuss a tiny bit about React Native and Flutter (2 very popular frameworks). #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 202311min

Aggregations in Mongo - Much faster and the best (and maybe only?) alternative in many cases

Aggregations in Mongo - Much faster and the best (and maybe only?) alternative in many cases

If you are using MongoDB and an Object Data Modeling library like Mongoose, or Mongoid, you may find yourselves in situations where your queries are taking a little too long. Come Aggregations to your rescue! #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 202310min

Make your Git commits frequent so you run into fewer merge conflicts

Make your Git commits frequent so you run into fewer merge conflicts

Say, your team is working on a number of features, and there is a steady stream of Pull Requests at any given time. The likelihood and complexity of merge conflicts depends both on the size of your team, and the scope of your features. In this podcast, I discuss a few options to mitigate merge conflicts. #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 20238min

Heroku Cloud Application Platform - why it will help your TTM (Time to Market)

Heroku Cloud Application Platform - why it will help your TTM (Time to Market)

If you want complete control over your servers, you would choose (something like) Amazon EC2 and start with creating a new Machine Image. But, what if your interest primarily lied in building your app, and solving your user's problems ASAP, and you didn't want to spend much, if any, time on setting up and configuring servers? #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 20238min

Building a Brand New Application? Looking for a way to reduce your ramp up time?

Building a Brand New Application? Looking for a way to reduce your ramp up time?

When you get started on a new project, it is natural to feel a tad overwhelmed. After all, it may be a new team, new technology stack, new processes, and new business problems - so it is alright to feel a little restless. But there is a way to feel at home by the end of Day 1.   #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 202312min

Development Efficiency - Are you bouncing your servers too often?

Development Efficiency - Are you bouncing your servers too often?

If it takes you 5 minutes to test 10 lines of code, there is a problem. If it takes you 3 minutes to bounce your (local) server, and that's the only way for you to test your code, there is another problem.  #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 202312min

Code Quality - How can you assess the quality of a codebase before reviewing a single line of code?

Code Quality - How can you assess the quality of a codebase before reviewing a single line of code?

Whether you are joining a new team, or a new developer is joining your team, the expectation is quite the same - "a good quality codebase that will help us understand the product without having to bug the rest of the team". How can we work towards building such a codebase? #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 20239min

Backwards Compatibility Issues - a few ways to ensure that you do not run into them

Backwards Compatibility Issues - a few ways to ensure that you do not run into them

You cannot afford to make Backwards Compatibility an after thought. As your system grows, and your user base grows, you are going to continually add new features and services, and some of them will require fundamental changes to your semantics and data structure. Given that, backwards compatibility can get rather challenging but there are steps you can take to mitigate potential issues. #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 20239min