(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors
Software Development, Finance and AI18 Tammi 2024

(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

In this podcast, Krish explores the various connectors available in Confluent Cloud. He starts by recapping the previous podcasts and the basics of Confluent Cloud. Krish then focuses on connectors, explaining their value and why they can reduce the need for writing code. He explores different connectors, such as the data gen source connector and the MongoDB Atlas connectors. Krish also discusses different data formats, including Avro, Protobuf, and JSON. He briefly touches on implementing custom connectors. Krish explores the topic of connectors in Confluent Cloud. He discusses the process of creating connectors and the different types of connectors available. Krish also delves into configuring connectors and defining configuration parameters. He explores the concept of custom connector configuration and the use of connector properties files. Krish then explores existing connectors, such as the HTTP source and sync connectors, and discusses the process of publishing custom connectors. He concludes by mentioning the Confluent CLI for managing connectors.

Takeaways

  • Connectors in Confluent Cloud provide value by reducing the need for writing code.
  • Different connectors are available for various data sources and destinations, such as MongoDB, Amazon S3, and Elasticsearch.
  • Data formats like Avro, Protobuf, and JSON can be used with connectors.
  • Implementing custom connectors allows for more flexibility and integration with specific systems.
  • Connectors enable seamless data integration and propagation between different systems. Connectors in Confluent Cloud allow for seamless integration with various systems and services.
  • Custom connectors can be created and published to Confluent Cloud.
  • Configuration parameters for connectors can be defined and managed.
  • The Confluent CLI provides a command-line interface for managing connectors.

Chapters

00:00 Introduction

00:35 Recap of Previous Podcasts

01:05 Focus on Connectors in Confluent Cloud

02:16 Exploring Data Gen Source Connector

03:43 Different Formats: Avro, Protobuf, JSON

08:07 Differences Between Avro and Protobuf

10:03 Exploring Other Connectors

11:14 Using MongoDB Atlas Connectors

12:08 Testing Different Formats with Connectors

13:36 Handling Avro Format with Consumer

16:58 Exploring More Connectors: Snowflake, Amazon S3, Elasticsearch

20:33 Implementing Custom Connectors

27:31 Exploring More Connectors: Salesforce, Oracle, Jira

35:16 Exploring More Connectors: SQL Server, MySQL

38:43 Implementing Custom Connectors

43:24 Exploring More Connectors: Kafka, File

46:20 Understanding Connector Implementation

49:06 Creating Custom Connectors

50:00 Summary and Conclusion

50:59 Creating Connectors

52:04 Configuring Connectors

54:00 Custom Connector Configuration

56:08 Defining Configuration Parameters

57:38 Configuration Properties

59:49 Self-Managed Connectors

01:00: 27 Connector Properties File

01:01:28 Creating Custom Connectors

01:02: 09 Publishing Custom Connectors

01:03: 37Existing Connectors

01:04: 14HTTP Source Connector

01:06:40 HTTP Sync Connector

01:08:34 Other Connectors

01:10:34 Managing Connectors

01:12:14 Confluent CLI

Snowpal Products

Jaksot(412)

AWS Lambda and Layers: How to handle dependency errors

AWS Lambda and Layers: How to handle dependency errors

We have begun to use Lambda for one of our projects, and one of the very first issues we ran into with a rather simple Lambda function is dependency related. Here's the problem, and how we resolved it. #projectmanagement Manage your projects on https://snowpal.com. Keep it simple.

18 Helmi 20229min

Taking some cues from Test Cricket and using them to build good software!

Taking some cues from Test Cricket and using them to build good software!

### Sessions in Test Cricket   Borrowing a few concepts from Test Cricket, and trying to relate that to Software Development. Here's my take!   #projectmanagement   Manage your sessions better. Do it on https://snowpal.com!

18 Helmi 20229min

Myths of Software Development (Part 3/3)

Myths of Software Development (Part 3/3)

(Part 3/3) When it comes to Software Development(much like everything else), there are myths. And while technologies and stacks have changed frequently, I've hardly seen many of these myths go away (not for good, at least). In this 3-part series (yes, it took me that many!), I try to break those myths. #projectmanagement Manage your projects on https://snowpal.com. Keep it simple.

18 Helmi 20229min

Myths of Software Development (Part 2/3)

Myths of Software Development (Part 2/3)

(Part 2/3) When it comes to Software Development(much like everything else), there are myths. And while technologies and stacks have changed frequently, I've hardly seen many of these myths go away (not for good, at least). In this 3-part series (yes, it took me that many!), I try to break those myths. #projectmanagement Manage your projects on https://snowpal.com. Keep it simple.

18 Helmi 20229min

Myths of Software Development (Part 1/3)

Myths of Software Development (Part 1/3)

(Part 1/3) When it comes to Software Development(much like everything else), there are myths. And while technologies and stacks have changed frequently, I've hardly seen many of these myths go away (not for good, at least). In this 3-part series (yes, it took me that many!), I try to break those myths. #projectmanagement Manage your projects on https://snowpal.com. Keep it simple.

18 Helmi 20229min

It’s important that you develop across stacks (wide and deep)

It’s important that you develop across stacks (wide and deep)

By wide, I mean Polyglot development. By deep, I mean Full stack development. In today's day and age, it is all the more important that you don't end up being a single-language developer. It's a bit too risky. #projectmanagement Manage your projects on https://snowpal.com. Keep it simple.

6 Helmi 20226min

Where to focus your energies when building an MVP?

Where to focus your energies when building an MVP?

When you start out to build an MVP, or a new feature in an existing app, how you go about approaching the problem in the initial days or weeks will make a ton of difference.   #projectmanagement   Manage projects on https://snowpal.com. Keep it simple.

5 Helmi 202210min

Are you sure the requirements are complete, and you understand them?

Are you sure the requirements are complete, and you understand them?

If your requirements aren't complete, there is a problem. Sounds obvious? It sure is but, unfortunately, this is one of those recurring issues in Software Development that happens all too often and before you know it, your team is developing software to a specification that is not even close to being complete. #projectmanagement Manage your projects on https://snowpal.com. Keep it simple.

13 Tammi 20226min