标签 kafka 下的文章 - 搜虎博客 - 心有猛虎，细嗅蔷薇

首页 » kafka

kafka报错：java.lang.IllegalArgumentException: Magic v1 does not support record headers

作者: itquanquan
时间: 2019-07-15
分类: bug飞
评论

这个报错是因为kafka server的版本过低，建议更换kafka server

完整报错

java.lang.IllegalArgumentException: Magic v1 does not support record headers
    at org.apache.kafka.common.record.MemoryRecordsBuilder.appendWithOffset(MemoryRecordsBuilder.java:403)
    at org.apache.kafka.common.record.MemoryRecordsBuilder.appendWithOffset(MemoryRecordsBuilder.java:442)
    at org.apache.kafka.common.record.MemoryRecordsBuilder.append(MemoryRecordsBuilder.java:499)
    at org.apache.kafka.common.record.MemoryRecordsBuilder.append(MemoryRecordsBuilder.java:522)
    at org.apache.kafka.clients.producer.internals.ProducerBatch.tryAppend(ProducerBatch.java:107)
    at org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:222)
    at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:814)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:768)
    at org.springframework.kafka.core.DefaultKafkaProducerFactory$CloseSafeProducer.send(DefaultKafkaProducerFactory.java:384)
    at brave.kafka.clients.TracingProducer.send(TracingProducer.java:106)
    at org.springframework.kafka.core.KafkaTemplate.doSend(KafkaTemplate.java:368)
    at org.springframework.kafka.core.KafkaTemplate.send(KafkaTemplate.java:183)
    at tv.rr.messaging.KafkaMessageProducer.sendMessage(KafkaMessageProducer.java:38)
    at tv.rr.messaging.KafkaMessageProducer$$FastClassBySpringCGLIB$$8cc6427f.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
    at org.springframework.cloud.sleuth.instrument.async.TraceAsyncAspect.traceBackgroundThread(TraceAsyncAspect.java:65)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
    at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
    at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:115)
    at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
    at java.util.concurrent.FutureTask.run(FutureTask.java)
    at org.springframework.cloud.sleuth.instrument.async.TraceRunnable.run(TraceRunnable.java:62)
    at java.lang.Thread.run(Thread.java:748)

kafka 报错解决kafka.common.TopicExistsException: Topic "XXX" already exists.

作者: itquanquan
时间: 2018-11-19
分类: bug飞
评论

kafka 报错：kafka.common.TopicExistsException: Topic "XXX" already exists.怀疑是kafka和zookeeper不一致导致，在${kafka_home}/config/server.properties 中加入配置

delete.topic.enable=true

问题解决

kafka报错Can't convert value of class java.lang.String to class org.apache.kafka.common.serialization.ByteArraySerializer specified in value.serializer

作者: itquanquan
时间: 2018-10-16
分类: 随记
评论

kafka 发消息报错

org.apache.kafka.common.errors.SerializationException: Can't convert value of class java.lang.String to class org.apache.kafka.common.serialization.ByteArraySerializer specified in value.serializer

Caused by: java.lang.ClassCastException: java.lang.String cannot be cast to [B
    at org.apache.kafka.common.serialization.ByteArraySerializer.serialize(ByteArraySerializer.java:21)
    at org.apache.kafka.common.serialization.ExtendedSerializer$Wrapper.serialize(ExtendedSerializer.java:65)
    at org.apache.kafka.common.serialization.ExtendedSerializer$Wrapper.serialize(ExtendedSerializer.java:55)
    at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:791)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:768)
    at org.springframework.kafka.core.DefaultKafkaProducerFactory$CloseSafeProducer.send(DefaultKafkaProducerFactory.java:363)
    at brave.kafka.clients.TracingProducer.send(TracingProducer.java:79)
    at org.springframework.kafka.core.KafkaTemplate.doSend(KafkaTemplate.java:355)
    at org.springframework.kafka.core.KafkaTemplate.send(KafkaTemplate.java:182)
    at tv.rr.thor.bus.kafka.producer.KafkaMsgProducer.send(KafkaMsgProducer.java:34)
    at tv.rr.thor.bus.kafka.producer.KafkaMsgProducer$$FastClassBySpringCGLIB$$65ccb95e.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:746)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
    at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:88)
    at org.springframework.cloud.sleuth.instrument.async.TraceAsyncAspect.traceBackgroundThread(TraceAsyncAspect.java:65)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:644)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:633)
    at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:70)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)
    at tv.rr.thor.bus.kafka.producer.KafkaMsgProducer$$EnhancerBySpringCGLIB$$d0f2998e.send(<generated>)
    at tv.rr.thor.bus.ThorBusApplicationTests.testKafka(ThorBusApplicationTests.java:29)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
    at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
    at org.springframework.test.context.junit4.statements.RunBeforeTestExecutionCallbacks.evaluate(RunBeforeTestExecutionCallbacks.java:73)
    at org.springframework.test.context.junit4.statements.RunAfterTestExecutionCallbacks.evaluate(RunAfterTestExecutionCallbacks.java:83)
    at org.springframework.test.context.junit4.statements.RunBeforeTestMethodCallbacks.evaluate(RunBeforeTestMethodCallbacks.java:75)
    at org.springframework.test.context.junit4.statements.RunAfterTestMethodCallbacks.evaluate(RunAfterTestMethodCallbacks.java:86)
    at org.springframework.test.context.junit4.statements.SpringRepeat.evaluate(SpringRepeat.java:84)
    at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:251)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:97)
    at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
    at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
    at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
    at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
    at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
    at org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
    at org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
    at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
    at org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:190)

代码如下


@Component
public class KafkaMsgProducer {

    @Autowired
    private KafkaTemplate<Long, String> kafkaTemplate;



    private static final Logger log = LoggerFactory.getLogger(KafkaMsgProducer.class);

    @Async
    public void send(String topic, Object msg) {

        ListenableFuture<SendResult<Long, String>> future = kafkaTemplate.send(topic, JSON.toJSONString(msg));
        future.addCallback(new ListenableFutureCallback<SendResult<Long, String>>() {
            @Override
            public void onFailure(Throwable throwable) {
                log.warn("Sending message to Kafka failed, topic {}, msg: {}", topic, JSON.toJSONString(msg));
            }

            @Override
            public void onSuccess(SendResult<Long, String> season) {
                log.info("Sending message to Kafka finished, topic {}, msg: {}", topic, JSON.toJSONString(msg));
            }
        });

    }
}

原因分析：java value的序列化和反序列化没有配置，配置如下

spring:
  kafka:
    producer:
      value-serializer: org.apache.kafka.common.serialization.StringSerializer
#      key-serializer: org.apache.kafka.common.serialization.LongSerializer
    consumer:
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer

还有有的项目不需要配置，怀疑是版本问题，默认序列化器有的是String,有的不是

kafka报错could not be established. Broker may not be available.

作者: itquanquan
时间: 2018-09-30
分类: bug飞
评论

原因查出来是kafka gg了，进程不在了,注意这条命令也会生成一个进程执行，所以搜索出来的结果如果只有一条，表示kafka进程不在了哦。

 ps -ef | grep kafka

重启kafka，问题解决

bin/kafka-server-start.sh config/server.properties

kafka报错：java.lang.IllegalArgumentException: Magic v1 does not support record headers

kafka 报错解决kafka.common.TopicExistsException: Topic "XXX" already exists.

kafka 报错：kafka.common.TopicExistsException: Topic "XXX" already exists.怀疑是kafka和zookeeper不一致导致，在${kafka_home}/config/server.properties 中加入配置

kafka报错Can't convert value of class java.lang.String to class org.apache.kafka.common.serialization.ByteArraySerializer specified in value.serializer

kafka 发消息报错

代码如下

原因分析：java value的序列化和反序列化没有配置，配置如下

还有有的项目不需要配置，怀疑是版本问题，默认序列化器有的是String,有的不是

kafka报错could not be established. Broker may not be available.

最新文章

其它