(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

In this podcast, Krish explores the various connectors available in Confluent Cloud. He starts by recapping the previous podcasts and the basics of Confluent Cloud. Krish then focuses on connectors, explaining their value and why they can reduce the need for writing code. He explores different connectors, such as the data gen source connector and the MongoDB Atlas connectors. Krish also discusses different data formats, including Avro, Protobuf, and JSON. He briefly touches on implementing custom connectors. Krish explores the topic of connectors in Confluent Cloud. He discusses the process of creating connectors and the different types of connectors available. Krish also delves into configuring connectors and defining configuration parameters. He explores the concept of custom connector configuration and the use of connector properties files. Krish then explores existing connectors, such as the HTTP source and sync connectors, and discusses the process of publishing custom connectors. He concludes by mentioning the Confluent CLI for managing connectors.

Takeaways

  • Connectors in Confluent Cloud provide value by reducing the need for writing code.
  • Different connectors are available for various data sources and destinations, such as MongoDB, Amazon S3, and Elasticsearch.
  • Data formats like Avro, Protobuf, and JSON can be used with connectors.
  • Implementing custom connectors allows for more flexibility and integration with specific systems.
  • Connectors enable seamless data integration and propagation between different systems. Connectors in Confluent Cloud allow for seamless integration with various systems and services.
  • Custom connectors can be created and published to Confluent Cloud.
  • Configuration parameters for connectors can be defined and managed.
  • The Confluent CLI provides a command-line interface for managing connectors.

Chapters

00:00 Introduction

00:35 Recap of Previous Podcasts

01:05 Focus on Connectors in Confluent Cloud

02:16 Exploring Data Gen Source Connector

03:43 Different Formats: Avro, Protobuf, JSON

08:07 Differences Between Avro and Protobuf

10:03 Exploring Other Connectors

11:14 Using MongoDB Atlas Connectors

12:08 Testing Different Formats with Connectors

13:36 Handling Avro Format with Consumer

16:58 Exploring More Connectors: Snowflake, Amazon S3, Elasticsearch

20:33 Implementing Custom Connectors

27:31 Exploring More Connectors: Salesforce, Oracle, Jira

35:16 Exploring More Connectors: SQL Server, MySQL

38:43 Implementing Custom Connectors

43:24 Exploring More Connectors: Kafka, File

46:20 Understanding Connector Implementation

49:06 Creating Custom Connectors

50:00 Summary and Conclusion

50:59 Creating Connectors

52:04 Configuring Connectors

54:00 Custom Connector Configuration

56:08 Defining Configuration Parameters

57:38 Configuration Properties

59:49 Self-Managed Connectors

01:00: 27 Connector Properties File

01:01:28 Creating Custom Connectors

01:02: 09 Publishing Custom Connectors

01:03: 37Existing Connectors

01:04: 14HTTP Source Connector

01:06:40 HTTP Sync Connector

01:08:34 Other Connectors

01:10:34 Managing Connectors

01:12:14 Confluent CLI

Snowpal Products

Avsnitt(412)

Go SDK and Go Recipes: A first look at our approach.

Go SDK and Go Recipes: A first look at our approach.

A quick first look at our approach behind the Go SDK we are working on, which will also include a set of recipes. #snowpal #projectmanagement #developer #api Integrate with Snowpal. Subscribe to Snowpal API. Manage personal and professional projects on https://snowpal.com

7 Mars 20236min

There’s always a few ways to learn something new: Tutorials, Training, Classes, Lessons (some work slightly better than others!)

There’s always a few ways to learn something new: Tutorials, Training, Classes, Lessons (some work slightly better than others!)

Continuous Learning is key to most fields and Software Development is no different. But, how you go about doing it might end up making all the difference. #snowpal #projectmanagement #developer #api Integrate with Snowpal. Subscribe to Snowpal API. Manage personal and professional projects on https://snowpal.com

7 Mars 20237min

Why I think the hiring process is broken. And lacking. Do you agree?

Why I think the hiring process is broken. And lacking. Do you agree?

A simple example to prove that the Hiring Process in general is quite broken. And lacking.  #snowpal #projectmanagement #developer #api Integrate with Snowpal. Subscribe to Snowpal API. Manage personal and professional projects on https://snowpal.com

3 Mars 20239min

Development Efficiency is not optional. And it is no less important than Production Stability.

Development Efficiency is not optional. And it is no less important than Production Stability.

What in the world could Development Efficiency have anything to do with Production Stability? Well, when you have to do one thing, you can do it the not so efficient way and get away with it. Maybe even when you are doing 5 things. But, what if you need to do 10 things? Or, 25 things? Or a 100 things? #snowpal #projectmanagement #developer #api Integrate with Snowpal. Subscribe to Snowpal API. Manage personal and professional projects on https://snowpal.com

3 Mars 20237min

Dependencies on 3rd party libraries and systems, and ones that are far from perfect: What to look out for

Dependencies on 3rd party libraries and systems, and ones that are far from perfect: What to look out for

When you are building something, you are going to depend on other systems regardless of what you are building. And those systems are going to be far from perfect. So, what happens when you run into issues? #snowpal #projectmanagement #developer #api Integrate with Snowpal. Subscribe to Snowpal API. Manage personal and professional projects on https://snowpal.com

3 Mars 20239min

API Hubs and Development Environments (or lack thereof)

API Hubs and Development Environments (or lack thereof)

API Hubs do not support the notion of environments. But, you need more than one environment for the usual reasons. So, how do you go about achieving this? #snowpal #projectmanagement #developer #api Integrate with Snowpal. Subscribe to Snowpal API. Manage personal and professional projects on https://snowpal.com

3 Mars 20239min

Creating a GO SDK. What does it entail.

Creating a GO SDK. What does it entail.

Snowpal GO SDK: Now that we've published our API on several API Hubs, we've embarked on an effort to creating our first SDK. In GO. #snowpal #projectmanagement #developer #api #golang #sdk Integrate with Snowpal. Subscribe to Snowpal API. Manage personal and professional projects on https://snowpal.com

3 Mars 20235min

DNS Records: Redirects using URL Record Type

DNS Records: Redirects using URL Record Type

When you want to set up URL redirection and are added a record in your DNS Server, watch out for this SSL nuance. #snowpal #projectmanagement #developer #api Integrate with Snowpal. Subscribe to Snowpal API. Manage personal and professional projects on https://snowpal.com

3 Mars 20236min

Populärt inom Teknik

uppgang-och-fall
market-makers
elbilsveckan
natets-morka-sida
rss-elektrikerpodden
skogsforum-podcast
bli-saker-podden
rss-laddstationen-med-elbilen-i-sverige
developers-mer-an-bara-kod
rss-uppgang-och-fall
rss-technokratin
hej-bruksbil
rss-veckans-ai
har-vi-akt-till-mars-an
rss-upplyst-entreprenordirektor
rss-bakom-boken
bilar-med-sladd
rss-fabriken-2
rss-rapporterat
rss-badfluence