3

I have the following setup:

$ export |grep HADOOP
declare -x HADOOP_HOME="/home/jesaremi/hadoop-3.1.3-bin"
declare -x HADOOP_OPTIONAL_TOOLS="hadoop-azure"

$ echo $PATH
/home/jesaremi/hadoop-3.1.3-bin/bin:/home/jesaremi/hadoop-3.1.3-bin/sbin:/opt/protobuf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/jesaremi/.dotnet/tools:/opt/HP_Fortify/HP_Fortify_SCA_and_Apps_4.30/bin:/opt/HP_Fortify/HP_Fortify_SCA_and_Apps_4.30/bin


$ ls -l /home/jesaremi/hadoop-3.1.3-bin/share/hadoop/common/
total 7452
-rw-rw-r-- 1 jesaremi jesaremi  303043 Dec 19 17:30 hadoop-azure-3.1.3.jar
-rw-rw-r-- 1 jesaremi jesaremi 4092533 Dec 17 09:42 hadoop-common-3.1.3-SNAPSHOT.jar
-rw-rw-r-- 1 jesaremi jesaremi 2877294 Dec 17 09:42 hadoop-common-3.1.3-SNAPSHOT-tests.jar
-rw-rw-r-- 1 jesaremi jesaremi  130001 Dec 17 09:42 hadoop-kms-3.1.3-SNAPSHOT.jar
-rw-rw-r-- 1 jesaremi jesaremi  201637 Dec 17 09:42 hadoop-nfs-3.1.3-SNAPSHOT.jar
drwxrwxr-x 2 jesaremi jesaremi    4096 Dec 17 09:42 jdiff
drwxrwxr-x 2 jesaremi jesaremi    4096 Dec 17 09:42 lib
drwxrwxr-x 2 jesaremi jesaremi    4096 Dec 17 09:42 sources
drwxrwxr-x 3 jesaremi jesaremi    4096 Dec 17 09:42 webapps

Yet when I try to amount an ABFS file system I get this:

$ hdfs dfsadmin -addMount abfs://[email protected]/ /mnt/abfs/main "fs.azure.abfs.account.name=test1.dfs.core.windows.net,fs.azure.account.key.test1.dfs.core.windows.net=<key removed>"
addMount: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem not found

1 Answer 1

5

afraid that HADOOP_OPTIONAL_TOOLS env var isn't enough; you'll need to get hadoop-azure JAR and some others into common/lib

from share/hadoop/tools/lib copy hadoop-azure jar, azure-* and, if it's there, wildfly-openssl.jar into share/hadoop/common/lib

The cloudstore JAR is with diagnostics as it tells you which JAR is missing, e.g.

bin/hadoop jar cloudstore-0.1-SNAPSHOT.jar storediag abfs://[email protected]/
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.