[mongodb-user] Failed to create external mongodb-based hive table

Discussion:

Farahzalina Yusof

2018-11-23 02:23:06 UTC

I tried to create mongodb-based hive table following the guidelines in
https://github.com/mongodb/mongo-hadoop/wiki/Hive-Usage

create external table mongoTest(
id String,age String,gender String,race String,custState String,purchaseDate
String,purchaseTime String,foodname String,restaurant String,foodtype String
,quantity String,totalPrice String,orderType String,rating String,servingType
String,characteristicString,restaurantType String,restaurantState String,priceRange
String,paymentMethod String,tableBooking String,onlineBooking String,deliveryService
String)
STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler'
WITH SERDEPROPERTIES('mongo.columns.mapping'='{"id":"_id","age":"age","gender":"gender","race":"race","custState":"custState
","purchaseDate":"purchaseDate","purchaseTime":"purchaseTime","foodname":"foodname","restaurant":"restaurant","foodtype":"foodtype","quantity":"quantity","totalPrice":"totalPrice","orderType":"orderType","rating":"rating","servingType":"servingType","characteristic":"characteristic","restaurantType":"restaurantType",
"restaurantState":"restaurantState","priceRange":"priceRange","paymentMethod":"paymentMethod","tableBooking":"tableBooking","onlineBooking":"onlineBooking","deliveryService":"deliveryService"}'
)
TBLPROPERTIES('mongo.uri'='mongodb://master:27017/bigdatadb.datasetv1');

but then i got this error:

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.
DDLTask. org/apache/hadoop/hive/serde2/SerDe
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/d38193ba-40d9-4762-bd31-2805d5d5dac1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Robert Cochran

2018-11-23 13:02:50 UTC

Permalink

Hi!

You don't give some important information:

What version is your Java JDK
What version is the MongoDB Java driver that you are using.
What version of the MongoDB server you are using.
What version of the MongoDB client you are using.
What operating system and version you are using for both the client and
server.
Provide some sample documents for use by those trying to test your issue.

However

If you carefully read the GitHub Documentation for the Hadoop Connector
<https://github.com/mongodb/mongo-hadoop#building> you will see it was only
tested on MongoDB version 2.2. That is really old. The connector itself
hasn't had any updates since April 2017, which is getting closer to 2 years
ago. So it hasn't kept up with recent MongoDB server releases.

To me, this doesn't seem to be an issue with the MongoDB server or client
at all. It is a Hadoop-related issue. Perhaps you can fork the Hadoop
Connector from GitHub and update it.

Thanks so much

Bob

Post by Farahzalina Yusof
I tried to create mongodb-based hive table following the guidelines in
https://github.com/mongodb/mongo-hadoop/wiki/Hive-Usage
create external table mongoTest(
id String,age String,gender String,race String,custState String,purchaseDate
String,purchaseTime String,foodname String,restaurant String,foodtype
String,quantity String,totalPrice String,orderType String,rating String,servingType
String,characteristicString,restaurantType String,restaurantState String,priceRange
String,paymentMethod String,tableBooking String,onlineBooking String,deliveryService
String)
STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler'
WITH SERDEPROPERTIES('mongo.columns.mapping'='{"id":"_id","age":"age","gender":"gender","race":"race","custState":"custState
","purchaseDate":"purchaseDate","purchaseTime":"purchaseTime","foodname":"foodname","restaurant":"restaurant","foodtype":"foodtype","quantity":"quantity","totalPrice":"totalPrice","orderType":"orderType","rating":"rating","servingType":"servingType","characteristic":"characteristic","restaurantType":"restaurantType",
"restaurantState":"restaurantState","priceRange":"priceRange","paymentMethod":"paymentMethod","tableBooking":"tableBooking","onlineBooking":"onlineBooking","deliveryService":"deliveryService"}'
)
TBLPROPERTIES('mongo.uri'='mongodb://master:27017/bigdatadb.datasetv1');
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec
.DDLTask. org/apache/hadoop/hive/serde2/SerDe

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/cded317d-d174-4967-aab9-62d41e430124%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Farahzalina Yusof

2018-11-26 01:56:21 UTC

Permalink

Hi tq for your response.

I'm using:
java jdk - 1.8.01
Mongo-java-driver - 3.2.1
Mongo-hadoop-core - 2.0.2
Mongo-hadoop-hive - 2.0.2
Ubuntu 16.04
Hadoop - 2.9.1
Mongodb - 4.0.4
Hive - 2.3.3

I also obtained the mongo-hadoop-connector thru
https://github.com/mongodb/mongo-hadoop

Post by Robert Cochran
Hi!
What version is your Java JDK
What version is the MongoDB Java driver that you are using.
What version of the MongoDB server you are using.
What version of the MongoDB client you are using.
What operating system and version you are using for both the client and
server.
Provide some sample documents for use by those trying to test your issue.
However
If you carefully read the GitHub Documentation for the Hadoop Connector
<https://github.com/mongodb/mongo-hadoop#building> you will see it was
only tested on MongoDB version 2.2. That is really old. The connector
itself hasn't had any updates since April 2017, which is getting closer to
2 years ago. So it hasn't kept up with recent MongoDB server releases.
To me, this doesn't seem to be an issue with the MongoDB server or client
at all. It is a Hadoop-related issue. Perhaps you can fork the Hadoop
Connector from GitHub and update it.
Thanks so much
Bob

You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/cded317d-d174-4967-aab9-62d41e430124%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/cded317d-d174-4967-aab9-62d41e430124%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CABLfj-9%3DO3sf%2BnwTdq2AEHXZ7wVVg9moaBUgvuuA0W8hvmhtFw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

'Wan Bachtiar' via mongodb-user

2018-11-26 04:56:54 UTC

Permalink

Hi Farahzalina,

Are there any other error messages or stack traces been printed to the Hive
logs?
Also what jars did you ADD (register) to Hive ?

See also FAQ: I get a ClassNotFoundException/NoClassDefFoundError when
using the connector. What do I do?
<https://github.com/mongodb/mongo-hadoop/wiki/FAQ#i-get-a-classnotfoundexceptionnoclassdeffounderror-when-using-the-connector-what-do-i-do>

Regards,
Wan.
â
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/5f2d7f30-0867-43f1-ae16-8611648d669d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Farahzalina Yusof

2018-11-26 05:06:53 UTC

Permalink

Hi Wan Bachtiar,
Thanks for your response

The only error printed is :
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. org/apache/hadoop/hive/serde2/SerDe

And i added the "mongo-java-driver", "mongo-hadoop-core", and
"mongo-hadoop-hive" jars to hive.

On Mon, Nov 26, 2018 at 12:57 PM 'Wan Bachtiar' via mongodb-user <

Post by 'Wan Bachtiar' via mongodb-user
Hi Farahzalina,
Are there any other error messages or stack traces been printed to the
Hive logs?
Also what jars did you ADD (register) to Hive ?
See also FAQ: I get a ClassNotFoundException/NoClassDefFoundError when
using the connector. What do I do?
<https://github.com/mongodb/mongo-hadoop/wiki/FAQ#i-get-a-classnotfoundexceptionnoclassdeffounderror-when-using-the-connector-what-do-i-do>
Regards,
Wan.
â
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/5f2d7f30-0867-43f1-ae16-8611648d669d%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/5f2d7f30-0867-43f1-ae16-8611648d669d%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CABLfj-_%2Bu07cZ7GNvCLR5AA2Af8yzqXRUTz3eDjFHmBocCTk5Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Farahzalina Yusof

2018-11-26 08:01:36 UTC

Permalink

btw, i succeed in building jars by running ./gradlew as suggested in the
https://github.com/mongodb/mongo-hadoop#building
and the build is succeed but i can't find the jars that being built against
what's specified in build.gradle

[image: build.PNG]

On Mon, Nov 26, 2018 at 1:06 PM Farahzalina Yusof <

Post by Farahzalina Yusof
Hi Wan Bachtiar,
Thanks for your response
FAILED: Execution Error, return code 1 from
org.apache.hadoop.hive.ql.exec.DDLTask. org/apache/hadoop/hive/serde2/SerDe
And i added the "mongo-java-driver", "mongo-hadoop-core", and
"mongo-hadoop-hive" jars to hive.
On Mon, Nov 26, 2018 at 12:57 PM 'Wan Bachtiar' via mongodb-user <

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CABLfj-_kqqsiL8zD_nXKyMnxCfvf3EFKNRf1SN-ZBTx2LSZ43A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Farahzalina Yusof

2018-11-27 08:47:36 UTC

Permalink

I solved my issue about creating external mongodb-based hive table. So now
the table is created and can present in the hdfs browser but now the new
issue is the table in hdfs is empty.
So how do i properly create the mongodb-based hive table so i can see the
data in mongodb thru hdfs?

On Mon, Nov 26, 2018 at 4:01 PM Farahzalina Yusof <

Post by Farahzalina Yusof
btw, i succeed in building jars by running ./gradlew as suggested in the
https://github.com/mongodb/mongo-hadoop#building
and the build is succeed but i can't find the jars that being built
against what's specified in build.gradle
[image: build.PNG]
On Mon, Nov 26, 2018 at 1:06 PM Farahzalina Yusof <

Post by 'Wan Bachtiar' via mongodb-user
Hi Farahzalina,
Are there any other error messages or stack traces been printed to the
Hive logs?
Also what jars did you ADD (register) to Hive ?
See also FAQ: I get a ClassNotFoundException/NoClassDefFoundError when
using the connector. What do I do?
<https://github.com/mongodb/mongo-hadoop/wiki/FAQ#i-get-a-classnotfoundexceptionnoclassdeffounderror-when-using-the-connector-what-do-i-do>
Regards,
Wan.
â
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/5f2d7f30-0867-43f1-ae16-8611648d669d%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/5f2d7f30-0867-43f1-ae16-8611648d669d%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CABLfj-_dBqAjCL%3DBCHAckXYZfxGqNu%2BV04NWEPd3sLRmq9r2Fw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.