(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

In this podcast, Krish explores the various connectors available in Confluent Cloud. He starts by recapping the previous podcasts and the basics of Confluent Cloud. Krish then focuses on connectors, explaining their value and why they can reduce the need for writing code. He explores different connectors, such as the data gen source connector and the MongoDB Atlas connectors. Krish also discusses different data formats, including Avro, Protobuf, and JSON. He briefly touches on implementing custom connectors. Krish explores the topic of connectors in Confluent Cloud. He discusses the process of creating connectors and the different types of connectors available. Krish also delves into configuring connectors and defining configuration parameters. He explores the concept of custom connector configuration and the use of connector properties files. Krish then explores existing connectors, such as the HTTP source and sync connectors, and discusses the process of publishing custom connectors. He concludes by mentioning the Confluent CLI for managing connectors.

Takeaways

  • Connectors in Confluent Cloud provide value by reducing the need for writing code.
  • Different connectors are available for various data sources and destinations, such as MongoDB, Amazon S3, and Elasticsearch.
  • Data formats like Avro, Protobuf, and JSON can be used with connectors.
  • Implementing custom connectors allows for more flexibility and integration with specific systems.
  • Connectors enable seamless data integration and propagation between different systems. Connectors in Confluent Cloud allow for seamless integration with various systems and services.
  • Custom connectors can be created and published to Confluent Cloud.
  • Configuration parameters for connectors can be defined and managed.
  • The Confluent CLI provides a command-line interface for managing connectors.

Chapters

00:00 Introduction

00:35 Recap of Previous Podcasts

01:05 Focus on Connectors in Confluent Cloud

02:16 Exploring Data Gen Source Connector

03:43 Different Formats: Avro, Protobuf, JSON

08:07 Differences Between Avro and Protobuf

10:03 Exploring Other Connectors

11:14 Using MongoDB Atlas Connectors

12:08 Testing Different Formats with Connectors

13:36 Handling Avro Format with Consumer

16:58 Exploring More Connectors: Snowflake, Amazon S3, Elasticsearch

20:33 Implementing Custom Connectors

27:31 Exploring More Connectors: Salesforce, Oracle, Jira

35:16 Exploring More Connectors: SQL Server, MySQL

38:43 Implementing Custom Connectors

43:24 Exploring More Connectors: Kafka, File

46:20 Understanding Connector Implementation

49:06 Creating Custom Connectors

50:00 Summary and Conclusion

50:59 Creating Connectors

52:04 Configuring Connectors

54:00 Custom Connector Configuration

56:08 Defining Configuration Parameters

57:38 Configuration Properties

59:49 Self-Managed Connectors

01:00: 27 Connector Properties File

01:01:28 Creating Custom Connectors

01:02: 09 Publishing Custom Connectors

01:03: 37Existing Connectors

01:04: 14HTTP Source Connector

01:06:40 HTTP Sync Connector

01:08:34 Other Connectors

01:10:34 Managing Connectors

01:12:14 Confluent CLI

Snowpal Products

Jaksot(412)

Native Mobile App - Design Patterns & Separation of concerns (Part 1)

Native Mobile App - Design Patterns & Separation of concerns (Part 1)

(Part 1) Separation of concerns is important to keep your native mobile app modular and becomes especially important when your app integrates with 100s of APIs. You want to ensure that your app is resilient to changes to the API Spec.  #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 20239min

Native Mobile App - Design Patterns & Separation of concerns (Part 2)

Native Mobile App - Design Patterns & Separation of concerns (Part 2)

(Part 2) Separation of concerns is important to keep your native mobile app modular and becomes especially important when your app integrates with 100s of APIs. You want to ensure that your app is resilient to changes to the API Spec.  #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 20239min

Databases & Denormalization - Faster queries for faster retrieval

Databases & Denormalization - Faster queries for faster retrieval

Whether you are designing a NoSQL Database or a traditional RDBMS Database, one way to optimize to ensure faster retrieval is to denormalize but that's not the best (or even the first) solution. You want to try out a few other things before resorting to denormalization. #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 20239min

Reusable UI components (React and beyond) - both presentational & functional

Reusable UI components (React and beyond) - both presentational & functional

When building UI screens (on mobile apps) or pages (on web apps), you want to focus your energies on your specific business problems. What you don't want to do is spend time reinventing the wheel. #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 20239min

Caching - What to Cache, and When to Cache (Part 3)

Caching - What to Cache, and When to Cache (Part 3)

Caching is a broad topic (what isn't?). In this podcast, I've only tried to scratch the surface. How do you determine what to cache? And once you do, how do you go about it? #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 20239min

Caching - Are you optimizing before identifying the problem? (Part 2)

Caching - Are you optimizing before identifying the problem? (Part 2)

If your pages take too long to render, caching may be a solution but before you go all out and start implementing or improving it, ask yourself if there are other fundamental problems that need addressing first (which led to the issues in the first place).   #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 20238min

When looking to make performance improvements, it is natural to look for that ”single” big problem. But, that’s hardly ever the case.

When looking to make performance improvements, it is natural to look for that ”single” big problem. But, that’s hardly ever the case.

I've worked on making performance improvements across all tiers, and in a variety of ways within any given tier, during the course of my career and one thing I can say with a good degree of certainty is that it gets more challenging as you get closer to your goal. Reducing a page rendering time from 3 seconds to 1 second is a whole lot easier than reducing it to 900ms, then to 800ms, and then to 50ms!   #snowpal aws.snowpal.com

24 Marras 20238min

Keep your code DRY. As in, super DRY! Code duplication is a maintenance nightmare.

Keep your code DRY. As in, super DRY! Code duplication is a maintenance nightmare.

You have to make a conscious effort to keep your code base DRY. Every time you make a change, add a feature, or fix a bug, make an effort to find duplication and to determine design patterns. This will play a huge role in keeping your code DRY, and maintainable in the long run. #snowpal aws.snowpal.com learn.snowpal.com

24 Marras 202315min