(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

In this podcast, Krish explores the various connectors available in Confluent Cloud. He starts by recapping the previous podcasts and the basics of Confluent Cloud. Krish then focuses on connectors, explaining their value and why they can reduce the need for writing code. He explores different connectors, such as the data gen source connector and the MongoDB Atlas connectors. Krish also discusses different data formats, including Avro, Protobuf, and JSON. He briefly touches on implementing custom connectors. Krish explores the topic of connectors in Confluent Cloud. He discusses the process of creating connectors and the different types of connectors available. Krish also delves into configuring connectors and defining configuration parameters. He explores the concept of custom connector configuration and the use of connector properties files. Krish then explores existing connectors, such as the HTTP source and sync connectors, and discusses the process of publishing custom connectors. He concludes by mentioning the Confluent CLI for managing connectors.

Takeaways

  • Connectors in Confluent Cloud provide value by reducing the need for writing code.
  • Different connectors are available for various data sources and destinations, such as MongoDB, Amazon S3, and Elasticsearch.
  • Data formats like Avro, Protobuf, and JSON can be used with connectors.
  • Implementing custom connectors allows for more flexibility and integration with specific systems.
  • Connectors enable seamless data integration and propagation between different systems. Connectors in Confluent Cloud allow for seamless integration with various systems and services.
  • Custom connectors can be created and published to Confluent Cloud.
  • Configuration parameters for connectors can be defined and managed.
  • The Confluent CLI provides a command-line interface for managing connectors.

Chapters

00:00 Introduction

00:35 Recap of Previous Podcasts

01:05 Focus on Connectors in Confluent Cloud

02:16 Exploring Data Gen Source Connector

03:43 Different Formats: Avro, Protobuf, JSON

08:07 Differences Between Avro and Protobuf

10:03 Exploring Other Connectors

11:14 Using MongoDB Atlas Connectors

12:08 Testing Different Formats with Connectors

13:36 Handling Avro Format with Consumer

16:58 Exploring More Connectors: Snowflake, Amazon S3, Elasticsearch

20:33 Implementing Custom Connectors

27:31 Exploring More Connectors: Salesforce, Oracle, Jira

35:16 Exploring More Connectors: SQL Server, MySQL

38:43 Implementing Custom Connectors

43:24 Exploring More Connectors: Kafka, File

46:20 Understanding Connector Implementation

49:06 Creating Custom Connectors

50:00 Summary and Conclusion

50:59 Creating Connectors

52:04 Configuring Connectors

54:00 Custom Connector Configuration

56:08 Defining Configuration Parameters

57:38 Configuration Properties

59:49 Self-Managed Connectors

01:00: 27 Connector Properties File

01:01:28 Creating Custom Connectors

01:02: 09 Publishing Custom Connectors

01:03: 37Existing Connectors

01:04: 14HTTP Source Connector

01:06:40 HTTP Sync Connector

01:08:34 Other Connectors

01:10:34 Managing Connectors

01:12:14 Confluent CLI

Snowpal Products

Jaksot(412)

Custom Attribution API: Build forms really quick

Custom Attribution API: Build forms really quick

One of the recent APIs we released is the Custom Attribution API, and the Postman Collection for it can be found on custom-attribution-api.snowpal.com. #snowpal

14 Marras 20232min

If you are not enjoying what you are doing, is it worth doing it at all?

If you are not enjoying what you are doing, is it worth doing it at all?

This particular podcast is less about technology and more about enjoying what you are doing. Do you love what you do? #snowpal #apidevelopment Subscribe to our APIs on aws.snowpal.com.

14 Marras 20238min

Rome was not built in a day. Nor will your next feature be.

Rome was not built in a day. Nor will your next feature be.

Too often, I see developers look a little flustered when they embark on a journey to build something new. Some of the questions/comments that repeatedly pop up: "There's a lot to do". "Will I be able to do it?". "Is the deadline reasonable?" Let's talk about this a bit, and see how we should approach this. #snowpal Hone your skills on learn.snowpal.com.

14 Marras 20239min

(Part 2/2) AWS Summit, DC: My thoughts (on workshops, sessions, etc).

(Part 2/2) AWS Summit, DC: My thoughts (on workshops, sessions, etc).

(Part 2/2) I attended the AWS Summit in DC last week, and here are some of my thoughts. #snowpal #projectmanagement #apidevelopment #softwaredevelopment  Integrate Snowpal APIs. Subscribe on AWS Marketplace. Get help from Snowpal Professional Services. Learn. Succeed. Grow. Hone your skills on Snowpal Education. Manage personal and professional projects on Snowpal. Download from App Store and/or Play Store.

12 Kesä 20235min

(Part 1/2) AWS Summit, DC: My thoughts (on workshops, sessions, etc).

(Part 1/2) AWS Summit, DC: My thoughts (on workshops, sessions, etc).

(Part 1/2) I attended the AWS Summit in DC last week, and here are some of my thoughts. #snowpal #projectmanagement #apidevelopment #softwaredevelopment Integrate Snowpal APIs. Subscribe on AWS Marketplace. Get help from Snowpal Professional Services. Learn. Succeed. Grow. Hone your skills on Snowpal Education. Manage personal and professional projects on Snowpal. Download from App Store and/or Play Store.

12 Kesä 20239min

PowerSync: Have a need to keep your data in sync with on-device databases?

PowerSync: Have a need to keep your data in sync with on-device databases?

PowerSync is a Cloud Service that helps keep your data in synch with your Web/Mobile clients -- as long as you use a SQL Database, and perhaps Flutter as a framework (not sure if they support other frameworks). ----more----I have not used this yet but it certainly piqued my interest so I met with their team recently, and here are my initial thoughts. For more information, check out https://www.powersync.co. ----more---- #snowpal #projectmanagement #apidevelopment #softwaredevelopment Integrate Snowpal APIs. Subscribe on AWS Marketplace. Get help from Snowpal Professional Services. Learn. Succeed. Grow. Hone your skills on Snowpal Education. Manage personal and professional projects on Snowpal. Download from App Store and/or Play Store.

15 Maalis 202316min

Importance of Design Decisions in Software Development: It can hardly ever be overstated.

Importance of Design Decisions in Software Development: It can hardly ever be overstated.

While all design decisions are important, some tend to have a longstanding effect compared to others. And it isn't always easy to tell which ones those are. So, how should you address those? #snowpal #projectmanagement #apidevelopment #softwaredevelopment Integrate Snowpal APIs. Subscribe on AWS Marketplace. Get help from Snowpal Professional Services. Learn. Succeed. Grow. Hone your skills on Snowpal Education. Manage personal and professional projects on Snowpal. Download from App Store and/or Play Store.

7 Maalis 20237min

Testing API on Postman: Some of the features Postman offers (and should you use them?)

Testing API on Postman: Some of the features Postman offers (and should you use them?)

Do your development and/or testing teams have an interest in testing APIs on Postman? If so, how do they go about it? #snowpal #projectmanagement #developer #api Integrate with Snowpal. Subscribe to Snowpal API. Manage personal and professional projects on https://snowpal.com

7 Maalis 20236min