Hive Metastore HA with DBTokenStore: Failed to initialize master key

This article describes my little adventure around a startup error with the Hive Metastore. It shall be reproducable with any secure installation, meaning with Kerberos, with high availability enabled and with the storage of the delegation token in a database. The version of Hive is the 1.2 packaged inside the Hortonworks 2.4.2 distribution.

Storage for delegation token is defined by the hive.cluster.delegation.token.store.class property. The available choices are Zookeeper, the Metastore database and memory. Both Cloudera and Hortonworks recommend using the database as org.apache.hadoop.hive.thrift.DBTokenStore.

The error in question occurs at the launch of the Metastore and bears the following signature:

java.io.IOException: Failed to initialize master key
    at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:213)
    at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:96)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6031)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5945)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:211)
    ... 9 more
Caused by: org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addMasterKey(java.lang.String)
    at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:158)
    at org.apache.hadoop.hive.thrift.DBTokenStore.addMasterKey(DBTokenStore.java:42)
    at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.logUpdateMasterKey(TokenStoreDelegationTokenSecretManager.java:193)
    at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:335)
    ... 14 more
Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addMasterKey(java.lang.String)
    at java.lang.Class.getMethod(Class.java:1670)
    at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:148)
    ... 17 more
Exception in thread "main" java.io.IOException: Failed to initialize master key
    at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:213)
    at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.startDelegationTokenSecretManager(HiveDelegationTokenManager.java:96)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6031)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5945)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.startThreads(TokenStoreDelegationTokenSecretManager.java:211)
    ... 9 more
Caused by: org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addMasterKey(java.lang.String)
    at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:158)
    at org.apache.hadoop.hive.thrift.DBTokenStore.addMasterKey(DBTokenStore.java:42)
    at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.logUpdateMasterKey(TokenStoreDelegationTokenSecretManager.java:193)
    at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.updateCurrentKey(AbstractDelegationTokenSecretManager.java:335)
    ... 14 more
Caused by: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addMasterKey(java.lang.String)
    at java.lang.Class.getMethod(Class.java:1670)
    at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnRawStore(DBTokenStore.java:148)
    ... 17 more

The message printed into stdout is quite clear:

java.io.IOException: Failed to initialize master key Caused by: org.apache.hadoop.hive.thrift.DelegationTokenStore$TokenStoreException: java.lang.NoSuchMethodException: org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.addMasterKey

Let’s dive into the source code. Hortonworks publishes on GitHub the source code of all the components of its distribution. Each distribution version corresponds to an associated tag under Git. Hive for HDP 2.4.2 can be imported into our local workstation with the commands (the link is now broken because the branch has been erased by Cloudera):

git clone https://github.com/hortonworks/hive-release/blob/HDP-2.4.2.0-tag
git checkout HDP-2.4.2.0-tag

Indeed, there is no function HiveMetaStore$HMSHandler.addMasterKey (edit: source no longer available) in version 2.4.2. We can find the exact same function inside the ObjectStore (edit: source no longer available) class of the same package. In Hive version 1.3 like in version 2.1, the code didn’t seem to have changed. In the logs, we confirm that it is the ObjectStore class which is mentioned:

metastore.HiveMetaStore - 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore

So from a configuration point of view, the parameter is well transmitted from the configuration and everything should be OK. The error is called in class DBTokenStore line 42 (edit: source no longer available) by the code:

return (Integer)invokeOnRawStore("addMasterKey", new Object[]{s},String.class);

Then again on line 156 (edit: source no longer available) with the code:

return rawStore.getClass().getMethod(methName, paramTypes).invoke(rawStore, params)

So, let’s find out how rawStore is built:

it arrives through the DBTokenStore.init method
not very sure where init is called
but after a quick search, objectStore is instantiated in HiveAuthFactory by rawStore = baseHandler.getMS (); where baseHandler is HiveMetaStore.HMSHandler
- in HiveMetaStore.HMSHandler.getMS, the getMS method returns an instance of threadLocalMS.get() or newRawStore() if null
the latter, newRawStore , feeds on the rawStoreClassName property
this property is initialized from the configuration (edit: source no longer available) rawStoreClassName = hiveConf.getVar(HiveConf.ConfVars.METASTORE_RAW_STORE_IMPL);
he default value is declared in HiveConf (edit: source no longer available) METASTORE_RAW_STORE_IMPL (“hive.metastore.rawstore.impl”, “org.apache.hadoop.hive.metastore.ObjectStore”, …)
we return to the class ObjectStore, which contains our method getMasterKey, so what is happening?

By re-browsing the source code, we see that HiveAuthFactory, where rawstore is built and passed to the function startDelegationTokenSecretManager, applies to HiveServer2. Hmm, could there be an equivalent for Metastore?

A search with startDelegationTokenSecretManager leads us again to the HiveMetaStore. And once there, what can we read:

delegationTokenManager.startDelegationTokenSecretManager(
            conf, baseHandler, ServerMode.METASTORE);

With baseHandler being instantiated a little above by the code:

HMSHandler baseHandler = new HiveMetaStore.HMSHandler(
          "new db based metaserver", conf, false);

The signature of startDelegationTokenSecretManager is:

public void startDelegationTokenSecretManager(Configuration conf, Object hms, ServerMode smode) throws IOException

Any object can be passed as a second argument which is logical since Hive is using reflection.

Inspired by HiveAuthFactory, we can see how rawStore is itself obtained from baseHandler. So the parameter is not baseHandler but baseHandler.getMS ().

So, HiveMetaStore on line 6031 (edit: source no longer available) should look like:

delegationTokenManager.startDelegationTokenSecretManager(conf, baseHandler.getMS(), ServerMode.METASTORE);

Once the modification are applied, it is now time to compile all of this.

The command mvn clean package -DskipTests -Phadoop-2 is not working on the first attempt. It could actually not happen. Being interested only in the compilation of the jar associated with the Metastore, let us take the option of quickly correcting the compilation errors. Fortunately there will be only one. It consists of removing all references to the class CallerContext in Hadoop23Shims (edit: source no longer available). After that, Maven does not compile all the projects but goes far enough to generate the MetaStore jar.

In terms of deployment, everything is not so simple either. Replacing the jar in question is not enough because another previously loaded jar also contains the impacted classes. I took the solution to rewrite the environment variable HADOOP_CLASSPATH by prefixing it with our newly imported jar. In the file “/usr/hdp/current/hive-metastore/bin/ext/metastore.sh”, add:

export HADOOP_CLASSPATH="/usr/hdp/2.4.2.0-258/hive/lib/hive-metastore.jar:$HADOOP_CLASSPATH"

before the line:

export HADOOP_OPTS="$HIVE_METASTORE_HADOOP_OPTS $HADOOP_OPTS"

You can now execute the command hive --service metastore and the Metastore shall start.

At the time of this writing, it seems that this problem is still present in the master branch of Hive Metastore when this article was written. However, it was not present in the previous release HDP 2.4.0 (edit: source no longer available).

UPDATE: The links in the article might be broken.

Share this article