> For the complete documentation index, see [llms.txt](https://chris-lomeli.gitbook.io/tiny-engines/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://chris-lomeli.gitbook.io/tiny-engines/walk-throughs/001-kafka-intro.md).

# kafka walkthroughs

{% hint style="info" %}
Code is stored at: [Github](https://github.com/tiny-engines-code/kafka-course-material)
{% endhint %}

## 01-hello-kafka

This is a version of the Confluent sample program, except that the configurations are in a separate file. We won't spend a lot of time on this except to run the program and make sure that we can all produce records to Kafka.

* Run the program against the training server
* Run the program against Confluent
* IConfluentect the data in Confluent using the Control Center or the Kafka-utils tool
* Change the record count and run again

![](https://835605371-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MirRN1Vsv3GcZ0OdXtu%2Fsync%2F67e210caad7e2bb5174e0720b773d8193c5a23f8.png?generation=1631147611097358\&alt=media)

## 02-hello-kafka-again

We'll dig into the basics of producing and consuming simple text messages to Kafka.

* Walk through the Producer configurations
* Introduction to the KafkaProducer class
* Walk through the Consumer properties
* Introduction to the KafkaConsumer class
* Run the program 
* Change the record count and run again

![](https://835605371-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MirRN1Vsv3GcZ0OdXtu%2Fsync%2F2e11d4292199205aa7e26537e4ba359badbd236a.png?generation=1631147615371775\&alt=media)

## 03-hello-admin

Use the AdminClient functionality to get and create topics.\
We've also separated configuration from logical code.

* Walk through the admin code
* Run the program and create a new topic in the training environment

![](https://835605371-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MirRN1Vsv3GcZ0OdXtu%2Fsync%2Fc2b1551b950c31d176233e334f397d4c37de58a8.png?generation=1631147610143067\&alt=media)

## 04-hello-json

Introduce the concept of serializers and how they are implemented in Kafka.

* IConfluentect the Json serializer and the changes to Producer and consumer configurations
* Walk through the generator pojo and factory
* Run the program for the Training broker
* IConfluentect the data in Confluent using the Control Center or the Kafka-utils tool
* Run the program 

![](https://835605371-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MirRN1Vsv3GcZ0OdXtu%2Fsync%2Fd3d24cbf8a05379abcc67052f8fab10c91d88fc2.png?generation=1631147608467650\&alt=media)

## 05-hello-json (perfs)

Quickly walk through the executor thread setup - the important takeaway here is that we share the producer across threads.

* IConfluentect the threaded producer

![](https://835605371-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MirRN1Vsv3GcZ0OdXtu%2Fsync%2F8bf1f77d54d953baef2d77345fc75a90c49610f9.png?generation=1631147607125083\&alt=media)

## 06-hello-avro

Why Avro?

* Explain automated 'sinks' like KafkaConnect that require schema
* Walk through a KafkaConnect example

Schema Registry

* Emphasis that Kafka is only storing binary schema-less data
* Explain the general concept of schema
* Explain the schema registry
* Schema naming strategy

Confluent Schema-Registry API

* Generic vs Specific records
* Explain Specific vs Generic records
* Quickly review a Specific record implementation
* Deep dive into the generic record implementation
* Run the program against the Confluent broker
* Kafka serialization internals

Optionally walk through KafkaAvro serializer logic

![](https://835605371-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MirRN1Vsv3GcZ0OdXtu%2Fsync%2F420bb7bc95aa7b4fe53101470a79938ba09fea6b.png?generation=1631147613139851\&alt=media)

## 07-hello-loadgen

The same basic material as version 06, but replace the User pojo with a NotificationStatus pojo, and substitute executor threads with a reactive approach (just for fun).

![](https://835605371-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MirRN1Vsv3GcZ0OdXtu%2Fsync%2F90a777be510530558b094119b69e9447f7c174f3.png?generation=1631147616000056\&alt=media)

## 08-kafka-spring (json)

Spring offers an implementation of Kafka that hides some boilerplate code. Revisit the 04-hello-json solution and re-write using spring-kafka

SpringCloud offers an even simpler generic interface

![](https://835605371-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MirRN1Vsv3GcZ0OdXtu%2Fsync%2Fa2bb91f660c09994ab26c05cc660e475e03820bf.png?generation=1631147612110763\&alt=media)

## 09-kafka-spring (avro)

Revisit the 06-hello-avro solution and re-write using spring-kafka

* Walk through all of the Spring producer code
* Run the program

![](https://835605371-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-MirRN1Vsv3GcZ0OdXtu%2Fsync%2F6b7903c855f1b7b68c2e8b13d3352db6adcbcd46.png?generation=1631147610380063\&alt=media)