Hi Martinus/Elif,
I
Post by Martinus MartinusHi Elif,
http://www.mongodb.org/display/DOCS/Import+Export+Tools.
I have never test that either. I guess the developers should know better
than me. :)
Thanks.
I created a collection 'in' in the 'test' db in mongos
show collections
in
system.indexes
and then ran the hadoop process
***@ip-10-252-31-236:/home/ubuntu/mongo-hadoop$ cd /usr/lib/hadoop-0.20/
***@ip-10-252-31-236:/usr/lib/hadoop-0.20$ bin/hadoop jar WordCount.jar
WordCount
Conf: Configuration: core-default.xml, core-site.xml
12/03/15 20:33:00 INFO security.UserGroupInformation: JAAS Configuration
already set up for Hadoop, not re-installing.
12/03/15 20:33:00 WARN mapred.JobClient: Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same.
12/03/15 20:33:01 INFO util.MongoSplitter: Calculate Splits Code ... Use
Shards? false, Use Chunks? true; Collection Sharded? false
12/03/15 20:33:01 INFO util.MongoSplitter: Creation of Input Splits is
enabled.
12/03/15 20:33:01 INFO util.MongoSplitter: Using Unsharded Split mode
(Calculating multiple splits though)
12/03/15 20:33:01 INFO util.MongoSplitter: Calculating unsharded input
splits on namespace 'test.in' with Split Key '{ "_id" : 1}' and a split
size of '8'mb per
12/03/15 20:33:01 INFO mapred.JobClient: Cleaning up the staging area
hdfs://master:54310/app/hadoop/tmp/mapred/staging/hduser/.staging/job_201203151831_0003
Exception in thread "main" java.lang.IllegalArgumentException: Error
calculating splits: { "serverUsed" :
"ec2-50-112-19-33.us-west-2.compute.amazonaws.com:27017" , "$err" :
"unrecognized command: splitVector" , "code" : 13390}
at
com.mongodb.hadoop.util.MongoSplitter.calculateUnshardedSplits(MongoSplitter.java:104)
at
com.mongodb.hadoop.util.MongoSplitter.calculateSplits(MongoSplitter.java:75)
at
com.mongodb.hadoop.MongoInputFormat.getSplits(MongoInputFormat.java:51)
at
org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:944)
at
org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:961)
at org.apache.hadoop.mapred.JobClient.access$500(JobClient.java:170)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:880)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1157)
at
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
at WordCount.main(WordCount.java:97)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:197)
and I still get the same error... any idea what am I doing wrong???
Thanks!!!
-k
Post by Martinus MartinusThanks Martinus. I did that and now WordCount is working. BUT how do I
import text to in collection? Right now I have empty in so empty out
collections.
I am assuming that "beyond_lies_the_wub.txt" is the sample input they
provide for this example.
But how do I import that into mongodb since it is not in json,csv or
tsv format?
thanks,
elif
Hi elif,
You need to made the collection named "in" inside your MongoDB database
from the Mongo shell and then you can run your WordCount.java example,
otherwise there is nothing to be map/reduce by mongo-hadoop.
Thanks.
How did you made the "in" collection later. I am getting the same
error and don't know how to proceed.
Are we supposed to import the beyond_lies_the_wub.txt to the mongodb
or we need to set it up as the input?
thanks.
Post by Martinus MartinusHi Eliot,
I knew where the problem is : I haven't made the "in" collection
when I
run
Post by Martinus Martinusthe program, so it gave me above error.
Thanks.
Merry Christmas.
On Mon, Dec 26, 2011 at 12:46 PM, Martinus Martinus
Post by Martinus MartinusHi Eliot,
I tried to used hadoop-mongo plugin using hadoop-0.20.2 and do I
need
to
Post by Martinus MartinusPost by Martinus Martinusadd external library for all of hadoop library? and when I tried
to
run the
Post by Martinus MartinusPost by Martinus MartinusConf: Configuration: core-default.xml, core-site.xml
11/12/26 12:42:46 INFO jvm.JvmMetrics: Initializing JVM Metrics
with
Post by Martinus MartinusPost by Martinus MartinusprocessName=JobTracker, sessionId=
11/12/26 12:42:46 WARN mapred.JobClient: Use GenericOptionsParser
for
Post by Martinus MartinusPost by Martinus Martinusparsing the arguments. Applications should implement Tool for the
same.
Post by Martinus MartinusPost by Martinus Martinus11/12/26 12:42:46 WARN mapred.JobClient: No job jar file set.
User
JobConf#setJar(String).
Post by Martinus MartinusPost by Martinus Martinus11/12/26 12:42:58 INFO util.MongoSplitter: Calculate Splits Code
...
Use
Post by Martinus MartinusPost by Martinus MartinusShards? false, Use Chunks? true; Collection Sharded? false
11/12/26 12:42:58 INFO util.MongoSplitter: Creation of Input
Splits is
Post by Martinus MartinusPost by Martinus Martinusenabled.
11/12/26 12:42:58 INFO util.MongoSplitter: Using Unsharded Split
mode
Post by Martinus MartinusPost by Martinus Martinus(Calculating multiple splits though)
11/12/26 12:42:58 INFO util.MongoSplitter: Calculating unsharded
input
Post by Martinus MartinusPost by Martinus Martinussplits on namespace 'test.in' with Split Key '{ "_id" : 1}' and a
split
Unable
com.mongodb.hadoop.util.MongoSplitter.calculateUnshardedSplits(MongoSplitter.java:106)
com.mongodb.hadoop.util.MongoSplitter.calculateSplits(MongoSplitter.java:75)
com.mongodb.hadoop.MongoInputFormat.getSplits(MongoInputFormat.java:51)
org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:885)
org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:779)
Post by Martinus MartinusPost by Martinus Martinusat org.apache.hadoop.mapreduce.Job.submit(Job.java:432)
at
org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447)
Post by Martinus MartinusPost by Martinus Martinusat WordCount.main(WordCount.java:76)
Would you be so kindly to tell me how to fix this problem?
Thanks.
On Wed, Dec 21, 2011 at 2:20 PM, Martinus Martinus <
Post by Martinus MartinusHi Eliot,
I have tried to built the jar file from the core folder inside
it,
file
below
Post by Martinus MartinusPost by Martinus MartinusPost by Martinus Martinus<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.5</source>
<target>1.5</target>
</configuration>
</plugin>
</plugins>
and it can be built using mvn package. It gaves me
mongo-hadoop-core-1.0-SNAPSHOT.jar on the target folder, but I
still
don't
Post by Martinus MartinusPost by Martinus MartinusPost by Martinus Martinusknow how to use this library along with mongodb inside eclipse.
Thanks.
Post by Eliot Horowitzhttps://github.com/mongodb/mongo-hadoop
On Tue, Dec 20, 2011 at 10:29 PM, Martinus Martinus
Post by Martinus MartinusHi,
I have hadoop cluster running and have my data inside mongodb
database. I
Post by Martinus Martinusalready write a java code to query data on mongodb using
mongodb-java
java
has
Google
.
Post by Martinus MartinusPost by Martinus MartinusPost by Martinus MartinusPost by Eliot HorowitzPost by Martinus MartinusTo unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/mongodb-user?hl=en.
--
You received this message because you are subscribed to the
Google
Post by Martinus MartinusPost by Martinus MartinusPost by Martinus MartinusPost by Eliot HorowitzGroups "mongodb-user" group.
To post to this group, send email to
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/mongodb-user?hl=en.
--
You received this message because you are subscribed to the Google
Groups
"mongodb-user" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/mongodb-user?hl=en.
--
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group, send email to
For more options, visit this group at
http://groups.google.com/group/mongodb-user?hl=en.
--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To view this discussion on the web visit https://groups.google.com/d/msg/mongodb-user/-/v5OWl03uLxQJ.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.