(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

In this podcast, Krish explores the various connectors available in Confluent Cloud. He starts by recapping the previous podcasts and the basics of Confluent Cloud. Krish then focuses on connectors, explaining their value and why they can reduce the need for writing code. He explores different connectors, such as the data gen source connector and the MongoDB Atlas connectors. Krish also discusses different data formats, including Avro, Protobuf, and JSON. He briefly touches on implementing custom connectors. Krish explores the topic of connectors in Confluent Cloud. He discusses the process of creating connectors and the different types of connectors available. Krish also delves into configuring connectors and defining configuration parameters. He explores the concept of custom connector configuration and the use of connector properties files. Krish then explores existing connectors, such as the HTTP source and sync connectors, and discusses the process of publishing custom connectors. He concludes by mentioning the Confluent CLI for managing connectors.

Takeaways

  • Connectors in Confluent Cloud provide value by reducing the need for writing code.
  • Different connectors are available for various data sources and destinations, such as MongoDB, Amazon S3, and Elasticsearch.
  • Data formats like Avro, Protobuf, and JSON can be used with connectors.
  • Implementing custom connectors allows for more flexibility and integration with specific systems.
  • Connectors enable seamless data integration and propagation between different systems. Connectors in Confluent Cloud allow for seamless integration with various systems and services.
  • Custom connectors can be created and published to Confluent Cloud.
  • Configuration parameters for connectors can be defined and managed.
  • The Confluent CLI provides a command-line interface for managing connectors.

Chapters

00:00 Introduction

00:35 Recap of Previous Podcasts

01:05 Focus on Connectors in Confluent Cloud

02:16 Exploring Data Gen Source Connector

03:43 Different Formats: Avro, Protobuf, JSON

08:07 Differences Between Avro and Protobuf

10:03 Exploring Other Connectors

11:14 Using MongoDB Atlas Connectors

12:08 Testing Different Formats with Connectors

13:36 Handling Avro Format with Consumer

16:58 Exploring More Connectors: Snowflake, Amazon S3, Elasticsearch

20:33 Implementing Custom Connectors

27:31 Exploring More Connectors: Salesforce, Oracle, Jira

35:16 Exploring More Connectors: SQL Server, MySQL

38:43 Implementing Custom Connectors

43:24 Exploring More Connectors: Kafka, File

46:20 Understanding Connector Implementation

49:06 Creating Custom Connectors

50:00 Summary and Conclusion

50:59 Creating Connectors

52:04 Configuring Connectors

54:00 Custom Connector Configuration

56:08 Defining Configuration Parameters

57:38 Configuration Properties

59:49 Self-Managed Connectors

01:00: 27 Connector Properties File

01:01:28 Creating Custom Connectors

01:02: 09 Publishing Custom Connectors

01:03: 37Existing Connectors

01:04: 14HTTP Source Connector

01:06:40 HTTP Sync Connector

01:08:34 Other Connectors

01:10:34 Managing Connectors

01:12:14 Confluent CLI

Snowpal Products

Jaksot(412)

API Gateway (Part 2/12): Publishing Snowpal API (Postman Workspace, Readme Documentation)

API Gateway (Part 2/12): Publishing Snowpal API (Postman Workspace, Readme Documentation)

(Part 2/12): Setting up Readme documentation, and creating manageable Postman workspaces. At Snowpal, we are close enough to launching our first API so other technology businesses can consume it, thereby focusing more on their core business problems and delivering quicker.   In this API Gateway series, I'll be sharing a lot of those details. If you have an interest in doing something similar, I hope it benefits you. #snowpal #projectmanagement Manage personal and professional projects on https://snowpal.com.

18 Tammi 20239min

API Gateway (Part 1/12): Publishing Snowpal API (for other businesses to consume)

API Gateway (Part 1/12): Publishing Snowpal API (for other businesses to consume)

(Part 1/12): You may have heard about API-First strategies. What does it mean to build your APIs first? What is an API Gateway?  At Snowpal, we are close enough to launching our first API so other technology businesses can consume it, thereby focusing more on their core business problems and delivering quicker.   In this API Gateway series, I'll be sharing a lot of those details. If you have an interest in doing something similar, I hope it benefits you. #snowpal #projectmanagement Manage personal and professional projects on https://snowpal.com.

17 Tammi 20236min

Thinking about purchasing an UI Template for a quick turnaround? Think again!

Thinking about purchasing an UI Template for a quick turnaround? Think again!

There's no dearth of UI templates and a lot of them certainly look promising but is the reality a bit different? #snowpal #projectmanagement Manage personal and professional projects on https://snowpal.com.

31 Joulu 20229min

Does your SDLC include this very critical step?

Does your SDLC include this very critical step?

You were given some Business Requirements. You understood them, converted them to Technical Requirements, designed it nicely, implemented it beautifully and tested it phenomenally well. At this point, you think you are ready to deploy those changes to Production. Sure about that? #snowpal #projectmanagement Manage personal and professional projects on https://snowpal.com.

30 Joulu 20224min

API Specification: Do Postman Collections with nested directories carry over seamlessly to OpenAPI?

API Specification: Do Postman Collections with nested directories carry over seamlessly to OpenAPI?

Well, we've seen that they do not. We created nested directories in our Postman Collections only to notice that they didn't convert gracefully to OpenAPI when transformed. #snowpal #projectmanagement Manage personal and professional projects on https://snowpal.com.

30 Joulu 20224min

OAuth 2.0 Implementation to support Apple, Microsoft, Facebook and Google

OAuth 2.0 Implementation to support Apple, Microsoft, Facebook and Google

We recently implemented Microsoft Sign In, and had added support for Google Sign In a few weeks earlier. And we've had support for Facebook and Apple Sign In from Day 1. They all support OAuth 2.0 yet we've had to deal with some nuances. #snowpal #projectmanagement Manage personal and professional projects on https://snowpal.com.

30 Joulu 20228min

Cool User Interface vs Stable User Interface: Are they NOT mutually exclusive items?

Cool User Interface vs Stable User Interface: Are they NOT mutually exclusive items?

It's nice to have a cool UI. Sure, who likes dated stuff! But, it is important that by making your UI cooler, you don't end up making it unstable. And if you are wondering why they have anything to do with each other, listen in! #snowpal #projectmanagement Manage personal and professional projects on https://snowpal.com.

30 Joulu 20229min

Transforming REST Endpoints defined in YAML files to Postman Workspace

Transforming REST Endpoints defined in YAML files to Postman Workspace

We had a need to dynamically create Postman Workspaces (collections and endpoints) as part of the API Gateway Product we are currently working on. So, we wrote a script to convert endpoints defined in YAML files to Postman workspaces. #snowpal #projectmanagement Manage personal and professional projects on https://snowpal.com.

30 Joulu 20229min