Flink: java.lang.NoSuchMethodError: AvroSchemaConverter

Question

I am trying to connect to Kafka. When I run a simple JAR file, I get the following error:

java.lang.NoSuchMethodError: org.apache.flink.formats.avro.typeutils.AvroSchemaConverter.convertToSchema(Lorg/apache/flink/table/types/logical/LogicalType;)Lorg/apache/avro/Schema

Yes, I am aware that NoSuchMethod can potentially mean that there is a conflict between versions; however, there is no Flink instance running on YARN cluster.

I have also tried to play with pom.xml but no luck

<properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <flink.version>1.13.2</flink.version>
    <java.version>1.8</java.version>
    <scala.binary.version>2.12</scala.binary.version>
    <confluent.version>5.2.1</confluent.version>
    <maven.compiler.source>${java.version}</maven.compiler.source>
    <maven.compiler.target>${java.version}</maven.compiler.target>
</properties>

<dependencies>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-sql-connector-kafka_${scala.binary.version}</artifactId>
        <version>${flink.version}</version>
    </dependency>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-table-planner-blink_${scala.binary.version}</artifactId>
        <version>${flink.version}</version>
    </dependency>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-clients_${scala.binary.version}</artifactId>
        <version>${flink.version}</version>
    </dependency>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-avro-confluent-registry</artifactId>
        <version>${flink.version}</version>
    </dependency>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-sql-avro</artifactId>
        <version>${flink.version}</version>
    </dependency>

    <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro</artifactId>
        <version>1.8.2</version>
    </dependency>


    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-statebackend-rocksdb_${scala.binary.version}</artifactId>
        <version>${flink.version}</version>
    </dependency>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-connector-jdbc_${scala.binary.version}</artifactId>
        <version>${flink.version}</version>
    </dependency>

    <!-- Flink Dependency -->
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-connector-hive_${scala.binary.version}</artifactId>
        <version>${flink.version}</version>
    </dependency>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-table-api-java-bridge_${scala.binary.version}</artifactId>
        <version>${flink.version}</version>
    </dependency>

    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-table-planner_${scala.binary.version}</artifactId>
        <version>${flink.version}</version>
    </dependency>

    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>1.7.5</version>
    </dependency>
    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
        <version>1.7.5</version>
    </dependency>
    <dependency>
        <groupId>commons-logging</groupId>
        <artifactId>commons-logging</artifactId>
        <version>1.2</version>
    </dependency>

</dependencies>

<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-jar-plugin</artifactId>
            <version>2.5</version>
            <configuration>
                <archive>
                    <manifest>
                        <mainClass>${main.class}</mainClass>
                    </manifest>
                </archive>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-compiler-plugin</artifactId>
            <version>3.1</version>
            <configuration>
                <source>${java.version}</source>
                <target>${java.version}</target>
            </configuration>
        </plugin>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>3.2.4</version>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <filters>
                            <filter>
                                <artifact>*:*</artifact>
                                <excludes>
                                    <exclude>META-INF/*.SF</exclude>
                                    <exclude>META-INF/*.DSA</exclude>
                                    <exclude>META-INF/*.RSA</exclude>
                                </excludes>
                            </filter>
                        </filters>
                        <transformers>
                            <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>com.azercell.bigdata.flink_data_calculator.flinkBalance</mainClass>
                            </transformer>
                            <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                                <resource>reference.conf</resource>
                            </transformer>
                        </transformers>
                        <relocations>
                            <relocation>
                                <pattern>org.codehaus.plexus.util</pattern>
                                <shadedPattern>org.shaded.plexus.util</shadedPattern>
                                <excludes>
                                    <exclude>org.codehaus.plexus.util.xml.Xpp3Dom</exclude>
                                    <exclude>org.codehaus.plexus.util.xml.pull.*</exclude>
                                </excludes>
                            </relocation>
                        </relocations>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

The versions are indicated above. I was able to connect to Hive; it's when I am trying to connect to Kafka, the problem arises.

Dominik Wosiński · Accepted Answer · 2021-11-18 15:44:26Z

1

If I recall correctly Flink 1.13.2 has switched to Apache Avro 1.10.0, so that's quite probably the issue You are facing since You are trying to use the 1.8.2 avro lib.

answered Nov 18, 2021 at 15:44

Dominik Wosiński

3,8941 gold badge11 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

eemamedo Over a year ago

That's correct. I will be posting a solution in a bit. However, you are correct in the root-cause of the problem.

eemamedo · Accepted Answer · 2021-11-19 11:52:35Z

0

The problem was with dependency. The trick was to either remove Avro 1.11.0 from Flink, or remove Avro 1.7.7 from Hive. I ended up removing Avro 1.11.0 and replacing it with a different Avro version:

<dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-sql-avro-confluent-registry</artifactId>
        <version>${flink.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.flink</groupId>
        <artifactId>flink-avro</artifactId>
        <version>1.13.2</version>
        <exclusions>
            <exclusion>
                <groupId>org.apache.avro</groupId>
                <artifactId>*</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro</artifactId>
        <version>1.8.2-cdh6.1.1</version>
    </dependency>

Another way that will also work is to use hive-exec::core and hive-metastore as dependencies.

edited Nov 19, 2021 at 11:52

answered Nov 19, 2021 at 10:44

eemamedo

3372 gold badges6 silver badges17 bronze badges

1 Comment

Community Over a year ago

Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center.

Collectives™ on Stack Overflow

Flink: java.lang.NoSuchMethodError: AvroSchemaConverter

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related