(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

(Part 4/4) Confluent Cloud (Managed Kafka as a Service) - What is a connector & How to create Custom Connectors

In this podcast, Krish explores the various connectors available in Confluent Cloud. He starts by recapping the previous podcasts and the basics of Confluent Cloud. Krish then focuses on connectors, explaining their value and why they can reduce the need for writing code. He explores different connectors, such as the data gen source connector and the MongoDB Atlas connectors. Krish also discusses different data formats, including Avro, Protobuf, and JSON. He briefly touches on implementing custom connectors. Krish explores the topic of connectors in Confluent Cloud. He discusses the process of creating connectors and the different types of connectors available. Krish also delves into configuring connectors and defining configuration parameters. He explores the concept of custom connector configuration and the use of connector properties files. Krish then explores existing connectors, such as the HTTP source and sync connectors, and discusses the process of publishing custom connectors. He concludes by mentioning the Confluent CLI for managing connectors.

Takeaways

  • Connectors in Confluent Cloud provide value by reducing the need for writing code.
  • Different connectors are available for various data sources and destinations, such as MongoDB, Amazon S3, and Elasticsearch.
  • Data formats like Avro, Protobuf, and JSON can be used with connectors.
  • Implementing custom connectors allows for more flexibility and integration with specific systems.
  • Connectors enable seamless data integration and propagation between different systems. Connectors in Confluent Cloud allow for seamless integration with various systems and services.
  • Custom connectors can be created and published to Confluent Cloud.
  • Configuration parameters for connectors can be defined and managed.
  • The Confluent CLI provides a command-line interface for managing connectors.

Chapters

00:00 Introduction

00:35 Recap of Previous Podcasts

01:05 Focus on Connectors in Confluent Cloud

02:16 Exploring Data Gen Source Connector

03:43 Different Formats: Avro, Protobuf, JSON

08:07 Differences Between Avro and Protobuf

10:03 Exploring Other Connectors

11:14 Using MongoDB Atlas Connectors

12:08 Testing Different Formats with Connectors

13:36 Handling Avro Format with Consumer

16:58 Exploring More Connectors: Snowflake, Amazon S3, Elasticsearch

20:33 Implementing Custom Connectors

27:31 Exploring More Connectors: Salesforce, Oracle, Jira

35:16 Exploring More Connectors: SQL Server, MySQL

38:43 Implementing Custom Connectors

43:24 Exploring More Connectors: Kafka, File

46:20 Understanding Connector Implementation

49:06 Creating Custom Connectors

50:00 Summary and Conclusion

50:59 Creating Connectors

52:04 Configuring Connectors

54:00 Custom Connector Configuration

56:08 Defining Configuration Parameters

57:38 Configuration Properties

59:49 Self-Managed Connectors

01:00: 27 Connector Properties File

01:01:28 Creating Custom Connectors

01:02: 09 Publishing Custom Connectors

01:03: 37Existing Connectors

01:04: 14HTTP Source Connector

01:06:40 HTTP Sync Connector

01:08:34 Other Connectors

01:10:34 Managing Connectors

01:12:14 Confluent CLI

Snowpal Products

Episoder(411)

GitHub Projects - How we manage our repo-specific issues

GitHub Projects - How we manage our repo-specific issues

We use our own product (obviously!) to give us a federated view of all the projects that our ecosystem comprises of but at a code repository level, we use GitHub's Project feature to manage repo-specific issues. Here's how we do it.

1 Des 20202min

Snowpal Pitch: GitHub Branch Management

Snowpal Pitch: GitHub Branch Management

There is no single answer to what branching strategies work for you when it comes to GitHub. We've tried more than a few ourselves but eventually arrived at one that works quite well for us.

28 Nov 20203min

Unit Testing vs Functional Testing vs Integration Testing: What are some differences?

Unit Testing vs Functional Testing vs Integration Testing: What are some differences?

While it's important to not be bogged down by terminologies, it is also imperative that there is consensus within any given team so there is no confusion.

28 Nov 20205min

TypeScript & ECMAScript6

TypeScript & ECMAScript6

It is hard to overstate the value strong typing brings to the table, especially in the context of teams that move real fast (and have fun doing so)!

22 Nov 20202min

SSL Connect Error: Upgrade to heroku-20

SSL Connect Error: Upgrade to heroku-20

If you upgraded to heroku-20, and ran into SSL errors, this quick video (less than 2 minutes is very, very quick for someone for who brevity ain't always a forte!) may help you. It took me 3 hours to figure this out, and if I can save you that time, it would be worth it, right!

20 Nov 20201min

Code Refactoring: Few things to consider

Code Refactoring: Few things to consider

Refactoring your code periodically is a good idea. But, when you do, exercise some caution.

17 Nov 202045s

(Part 2/2) Software Testing in a Startup

(Part 2/2) Software Testing in a Startup

If you are an ambitious startup (like us!) that tries to churn out a lot of new features on a weekly basis, you are always focused on the next cool thing to do. But, needless to mention, a stable platform in Production is imperative to building user trust. Here are some of the things we do to ensure uncompromising stability.

17 Nov 20203min

(Part 1/2) Software Testing in a Startup

(Part 1/2) Software Testing in a Startup

If you are an ambitious startup (like us!) that tries to churn out a lot of new features on a weekly basis, you are always focused on the next cool thing to do. But, needless to mention, a stable platform in Production is imperative to building user trust. Here are some of the things we do to ensure uncompromising stability.

10 Nov 20209min

Populært innen Teknologi

lydartikler-fra-aftenposten
romkapsel
rss-avskiltet
tomprat-med-gunnar-tjomlid
teknisk-sett
energi-og-klima
rss-impressions-2
teknologi-og-mennesker
shifter
hans-petter-og-co
fornybaren
rss-polypod
pedagogisk-intelligens
rss-alt-som-gar-pa-strom
i-loopen
smart-forklart
elektropodden
kunstig-intelligens-med-morten-goodwin
rss-digitaliseringspadden
rss-heis