Tag: Kafka

Connect Spring Cloud Stream With Kafka Binder to Azure Event Hub

Robert Lohr4 Comments

In two previous blog posts, I explained how to create a Kafka consumer and producer with the Spring Cloud Stream framework. In the Famous Last Words section of the producer, I already hinted at the notion of utilizing this technology for connecting to Azure Event Hub. While doing so, I discovered an error in one of Microsoft’s examples that has cost me about two days of work. I show you how to avoid the dreaded “Node -1 disconnected” error.

In this tutorial, I explain how to use the exact same code to connect to Azure Event Hub using a Shared Access Signature Token (connection string) and a Service Principal.

I have good news and bad news. Which one first? The bad? Okay, here we go:

There will not be any code in this tutorial, only YAML configuration.

Now to the good part:

There will not be any code in this tutorial, only YAML configuration.

This is the beauty of Spring Cloud Stream. Granted, I am not even swapping the binder for an Azure-native variant. So why would there be any code changes? But let me say this: I briefly plugged in the Event Hub Binder without changing the code in my research on getting this to work. Even the updates to the config were minimal. A few Event Hub-specific settings, especially the Storage Account for checkpoints, and that was it.

Enough foreplay; let me explain what you likely came here for.

Read More »

Produce Messages With Spring Cloud Stream Kafka

Robert Lohr2 Comments

Update:

In a recent post, I explained how to create a Kafka Consumer application with Spring Boot using Spring Cloud Stream with the Kafka Binder. In this installment, I explain how to build the other side of the Kafka connection: the producer.

The main concepts are the same. The most significant change is that instead of a Consumer<T>, you implement a Supplier<T>.

Read More »

Consume Messages With Spring Cloud Stream Kafka

Robert Lohr2 Comments

Update:

Spring Cloud Stream is a very complex topic and a remarkable piece of technology. It builds on other intricate Spring technologies like Spring Integration and Spring Cloud Function, and when you add Apache Kafka to the mix, you have a steep learning curve on your hands.

There is a lot of documentation to read and comprehend, and I do not think it helps that your first interaction with the technology is by showing off. Here is the sample in the “Introducing Spring Cloud Stream” section.

@SpringBootApplication
public class SampleApplication {
    public static void main(String[] args) {
        SpringApplication.run(SampleApplication.class, args);
    }
    @Bean
    public Function<String, String> uppercase() {
        return value -> value.toUpperCase();
    }
}

That supposedly is a fully functioning application. The uppercase() method consumes and produces simultaneously, essentially turning it into a way software can pleasure itself. To understand this example, you must know about all the Spring Boot auto-configuration magic happening in the background. Otherwise, it is an opaque magical, and indecipherable showpiece.

This post will show a practical example of a simple consumer application receiving messages from a Kafka cluster. This was my use case, and while the documentation contains a ton of helpful information, it only succeeded in confusing me at first, coming to the technology with fresh eyes. As a result, it took me a long time to put together all the pieces before I understood what was going on.

Read More »