Discussion:
MongoDB Cursor exceeds size limit - query
(too old to reply)
PB
2018-07-24 16:07:35 UTC
Permalink
Hi, I'm trying to runt this code, but I always get document size exception,
when querying elements not in a list:

I also tried aggregation but I get a similar document size exception.

MongoClient mongoClient = new MongoClient(host, port);

MongoCollection<Document> col =
mongoClient.getDatabase(db).getCollection(collection;



int offset = 0;

int limit = 10000;

BasicDBObject whereQuery = new BasicDBObject();
whereQuery.put(FIELD1,"FIELD1");
whereQuery.put(FIELD2, "FIELD2");
whereQuery.put(FIELD3, "FIELD3");
whereQuery.put(FIELD4 new BasicDBObject("$nin", list));

Set<String> set= new HashSet<>();

while (offset < totalCount) {

*List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());*

for (Document doc : documents) {

set.add(doc.getString(field));

}

offset += limit;

}

//process last

List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());

for (Document doc : documents) {

set.add(doc.getString(field));

}
mongoClient.close();





When this line of code " *col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>()); "* runs I get this error:





org.bson.BsonMaximumSizeExceededException: Document size of 18339335 is
larger than maximum of 16793600. at
org.bson.BsonBinaryWriter.validateSize(BsonBinaryWriter.java:418) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.backpatchSize(BsonBinaryWriter.java:412) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.doWriteEndArray(BsonBinaryWriter.java:153) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.AbstractBsonWriter.writeEndArray(AbstractBsonWriter.java:350) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.LevelCountingBsonWriter.writeEndArray(LevelCountingBsonWriter.java:78)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:85) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:60)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:238)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:188)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandMessage.encodeMessageBodyWithMetadata(CommandMessage.java:136)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.encode(RequestMessage.java:138)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandMessage.encode(CommandMessage.java:57)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:244)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:444)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:200)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:242)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:233)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:136)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation$1.call(FindOperation.java:701) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation$1.call(FindOperation.java:695) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:457)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:401)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation.execute(FindOperation.java:695) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation.execute(FindOperation.java:83) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:179)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:132)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:86)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.forEach(MongoIterableImpl.java:110)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.into(MongoIterableImpl.java:122)
~[mongo-java-driver-3.8.0.jar:na]
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/61476e6e-fd3e-4498-affc-cd5087aebcd0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Robert Cochran
2018-07-24 21:24:15 UTC
Permalink
Hi!

I think you need to explain four (4) things:

1. What version of MongoDB are you running this query on?
2. What operating system are you running MongoDB on?
3. Can you show a few documents, such that the structure of each can be
understood.
4. Can you explain in writing the exact goal of the query? What are you
attempting to solve -- and how do you want to solve it, in words rather
than code?

Thanks

Bob
Post by PB
Hi, I'm trying to runt this code, but I always get document size
I also tried aggregation but I get a similar document size exception.
MongoClient mongoClient = new MongoClient(host, port);
MongoCollection<Document> col =
mongoClient.getDatabase(db).getCollection(collection;
int offset = 0;
int limit = 10000;
BasicDBObject whereQuery = new BasicDBObject();
whereQuery.put(FIELD1,"FIELD1");
whereQuery.put(FIELD2, "FIELD2");
whereQuery.put(FIELD3, "FIELD3");
whereQuery.put(FIELD4 new BasicDBObject("$nin", list));
Set<String> set= new HashSet<>();
while (offset < totalCount) {
*List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());*
for (Document doc : documents) {
set.add(doc.getString(field));
}
offset += limit;
}
//process last
List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());
for (Document doc : documents) {
set.add(doc.getString(field));
}
mongoClient.close();
When this line of code " *col.find(whereQuery).skip(offset).limit(limit).into(new
org.bson.BsonMaximumSizeExceededException: Document size of 18339335 is
larger than maximum of 16793600. at
org.bson.BsonBinaryWriter.validateSize(BsonBinaryWriter.java:418) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.backpatchSize(BsonBinaryWriter.java:412) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.doWriteEndArray(BsonBinaryWriter.java:153) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.AbstractBsonWriter.writeEndArray(AbstractBsonWriter.java:350) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.LevelCountingBsonWriter.writeEndArray(LevelCountingBsonWriter.java:78)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:85) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:60)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:238)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:188)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandMessage.encodeMessageBodyWithMetadata(CommandMessage.java:136)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.encode(RequestMessage.java:138)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandMessage.encode(CommandMessage.java:57)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:244)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:444)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:200)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:242)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:233)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:136)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation$1.call(FindOperation.java:701) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation$1.call(FindOperation.java:695) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:457)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:401)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation.execute(FindOperation.java:695) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation.execute(FindOperation.java:83) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:179)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:132)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:86)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.forEach(MongoIterableImpl.java:110)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.into(MongoIterableImpl.java:122)
~[mongo-java-driver-3.8.0.jar:na]
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/ffb876d7-67e7-42af-acbf-600bd6020097%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
PB
2018-07-24 21:57:58 UTC
Permalink
This post might be inappropriate. Click to display it.
Robert Cochran
2018-07-25 01:47:55 UTC
Permalink
Hi!

Thank you for posting these details. They are very helpful.

I believe the list you refer to of 967 989 elements is causing the query
document itself to become larger than 16 megabytes in size. That in turn is
causing your query to fail.

I'm not sure how to create a query that will satisfy your requirements. You
may have to split your one list into several smaller lists and find some
way of processing these lists in a progression of steps.

Now that I've ventured my opinion of what the issue is, let me also say
that I am not very experienced with MongoDB queries. So I could be giving
you incorrect information. I could be entirely wrong.

You might want to enclose the query in a try...catch...finally block to
ensure that the database connection is closed even if the query fails.

Thanks so much

Bob
Post by PB
Hi Bob,
1.
MongoDB server version: 4.0.0, spring-data-mongodb :2.0.8.RELEASE and
mongo-java-driver: 3.8.0
Spring version 5.0.7.RELEASE
2.
Windows OS
3.
My Mongo Database has a *collection* with 173 666 558 documents. Each of
{
"_id" : ObjectId("5b57527773ce360ce0ed4575"),
"field6" : "00:00:00",
"field7" : "00:00:00",
"field5" : "0000000000000000000",
"*field4*" : "3822",
"field8" : "3",
"*field1*" : "xxxxx",l
"*field2*" : "1.0",
"*field3*" : "5002",
"_class" : "clazz"
}
4.
I have a* String list with 967 989 elements*.
I want to know which documents in my collection (depicted in 3. ) have
field1, field2 and field3 equal to FiELD1, FIELD2 and FIELD3 query
parameters and also *field4 equal to a value not present in my list
with *967989 elements also given as a query parameter.
Post by Robert Cochran
Hi!
1. What version of MongoDB are you running this query on?
2. What operating system are you running MongoDB on?
3. Can you show a few documents, such that the structure of each can be
understood.
4. Can you explain in writing the exact goal of the query? What are you
attempting to solve -- and how do you want to solve it, in words rather
than code?
Thanks
Bob
Post by PB
Hi, I'm trying to runt this code, but I always get document size
I also tried aggregation but I get a similar document size exception.
MongoClient mongoClient = new MongoClient(host, port);
MongoCollection<Document> col =
mongoClient.getDatabase(db).getCollection(collection;
int offset = 0;
int limit = 10000;
BasicDBObject whereQuery = new BasicDBObject();
whereQuery.put(FIELD1,"FIELD1");
whereQuery.put(FIELD2, "FIELD2");
whereQuery.put(FIELD3, "FIELD3");
whereQuery.put(FIELD4 new BasicDBObject("$nin", list));
Set<String> set= new HashSet<>();
while (offset < totalCount) {
*List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());*
for (Document doc : documents) {
set.add(doc.getString(field));
}
offset += limit;
}
//process last
List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());
for (Document doc : documents) {
set.add(doc.getString(field));
}
mongoClient.close();
When this line of code " *col.find(whereQuery).skip(offset).limit(limit).into(new
org.bson.BsonMaximumSizeExceededException: Document size of 18339335 is
larger than maximum of 16793600. at
org.bson.BsonBinaryWriter.validateSize(BsonBinaryWriter.java:418) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.backpatchSize(BsonBinaryWriter.java:412) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.doWriteEndArray(BsonBinaryWriter.java:153) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.AbstractBsonWriter.writeEndArray(AbstractBsonWriter.java:350) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.LevelCountingBsonWriter.writeEndArray(LevelCountingBsonWriter.java:78)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:85) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:60)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:238)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:188)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandMessage.encodeMessageBodyWithMetadata(CommandMessage.java:136)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.encode(RequestMessage.java:138)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandMessage.encode(CommandMessage.java:57)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:244)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:444)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:200)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:242)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:233)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:136)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation$1.call(FindOperation.java:701) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation$1.call(FindOperation.java:695) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:457)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:401)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation.execute(FindOperation.java:695) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation.execute(FindOperation.java:83) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:179)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:132)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:86)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.forEach(MongoIterableImpl.java:110)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.into(MongoIterableImpl.java:122)
~[mongo-java-driver-3.8.0.jar:na]
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/e1e395f0-fbe8-42c2-8903-885d5963e4aa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
PB
2018-07-26 13:41:39 UTC
Permalink
Hi Bob,

Thank you for your answer.

Regarding the 16Mb limit, I agree with you on the size of the query
parameter list, it has more than twice the size of 16 Mb and that is
probably the cause.

If I run the query by steps partioning the list, I will always get
documents with field4 not present in the sublists.

So for now I still am stuck with this.

For managing database connections I use try-with-resources.


quarta-feira, 25 de Julho de 2018 às 02:47:55 UTC+1, Robert Cochran
Post by Robert Cochran
Hi!
Thank you for posting these details. They are very helpful.
I believe the list you refer to of 967 989 elements is causing the query
document itself to become larger than 16 megabytes in size. That in turn is
causing your query to fail.
I'm not sure how to create a query that will satisfy your requirements.
You may have to split your one list into several smaller lists and find
some way of processing these lists in a progression of steps.
Now that I've ventured my opinion of what the issue is, let me also say
that I am not very experienced with MongoDB queries. So I could be giving
you incorrect information. I could be entirely wrong.
You might want to enclose the query in a try...catch...finally block to
ensure that the database connection is closed even if the query fails.
Thanks so much
Bob
Post by PB
Hi Bob,
1.
MongoDB server version: 4.0.0, spring-data-mongodb :2.0.8.RELEASE and
mongo-java-driver: 3.8.0
Spring version 5.0.7.RELEASE
2.
Windows OS
3.
My Mongo Database has a *collection* with 173 666 558 documents. Each of
{
"_id" : ObjectId("5b57527773ce360ce0ed4575"),
"field6" : "00:00:00",
"field7" : "00:00:00",
"field5" : "0000000000000000000",
"*field4*" : "3822",
"field8" : "3",
"*field1*" : "xxxxx",l
"*field2*" : "1.0",
"*field3*" : "5002",
"_class" : "clazz"
}
4.
I have a* String list with 967 989 elements*.
I want to know which documents in my collection (depicted in 3. ) have
field1, field2 and field3 equal to FiELD1, FIELD2 and FIELD3 query
parameters and also *field4 equal to a value not present in my list
with *967989 elements also given as a query parameter.
Post by Robert Cochran
Hi!
1. What version of MongoDB are you running this query on?
2. What operating system are you running MongoDB on?
3. Can you show a few documents, such that the structure of each can be
understood.
4. Can you explain in writing the exact goal of the query? What are you
attempting to solve -- and how do you want to solve it, in words rather
than code?
Thanks
Bob
Post by PB
Hi, I'm trying to runt this code, but I always get document size
I also tried aggregation but I get a similar document size exception.
MongoClient mongoClient = new MongoClient(host, port);
MongoCollection<Document> col =
mongoClient.getDatabase(db).getCollection(collection;
int offset = 0;
int limit = 10000;
BasicDBObject whereQuery = new BasicDBObject();
whereQuery.put(FIELD1,"FIELD1");
whereQuery.put(FIELD2, "FIELD2");
whereQuery.put(FIELD3, "FIELD3");
whereQuery.put(FIELD4 new BasicDBObject("$nin", list));
Set<String> set= new HashSet<>();
while (offset < totalCount) {
*List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());*
for (Document doc : documents) {
set.add(doc.getString(field));
}
offset += limit;
}
//process last
List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());
for (Document doc : documents) {
set.add(doc.getString(field));
}
mongoClient.close();
When this line of code " *col.find(whereQuery).skip(offset).limit(limit).into(new
org.bson.BsonMaximumSizeExceededException: Document size of 18339335 is
larger than maximum of 16793600. at
org.bson.BsonBinaryWriter.validateSize(BsonBinaryWriter.java:418) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.backpatchSize(BsonBinaryWriter.java:412) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.doWriteEndArray(BsonBinaryWriter.java:153) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.AbstractBsonWriter.writeEndArray(AbstractBsonWriter.java:350) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.LevelCountingBsonWriter.writeEndArray(LevelCountingBsonWriter.java:78)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:85) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:60)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:238)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:188)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandMessage.encodeMessageBodyWithMetadata(CommandMessage.java:136)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.encode(RequestMessage.java:138)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandMessage.encode(CommandMessage.java:57)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:244)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:444)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:200)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:242)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:233)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:136)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation$1.call(FindOperation.java:701) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation$1.call(FindOperation.java:695) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:457)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:401)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation.execute(FindOperation.java:695) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation.execute(FindOperation.java:83) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:179)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:132)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:86)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.forEach(MongoIterableImpl.java:110)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.into(MongoIterableImpl.java:122)
~[mongo-java-driver-3.8.0.jar:na]
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/ba885885-e332-4a77-b37e-6032321eef5f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Robert Cochran
2018-07-26 23:18:44 UTC
Permalink
Hi PB,

In the sample document you describe above, are all the fields except the
_id field strings? Or are they some other data format, e.g. NumberDecimal,
ISODate, etc.?

Thanks

Bob
Post by PB
Hi Bob,
Thank you for your answer.
Regarding the 16Mb limit, I agree with you on the size of the query
parameter list, it has more than twice the size of 16 Mb and that is
probably the cause.
If I run the query by steps partioning the list, I will always get
documents with field4 not present in the sublists.
So for now I still am stuck with this.
For managing database connections I use try-with-resources.
quarta-feira, 25 de Julho de 2018 às 02:47:55 UTC+1, Robert Cochran
Post by Robert Cochran
Hi!
Thank you for posting these details. They are very helpful.
I believe the list you refer to of 967 989 elements is causing the query
document itself to become larger than 16 megabytes in size. That in turn is
causing your query to fail.
I'm not sure how to create a query that will satisfy your requirements.
You may have to split your one list into several smaller lists and find
some way of processing these lists in a progression of steps.
Now that I've ventured my opinion of what the issue is, let me also say
that I am not very experienced with MongoDB queries. So I could be giving
you incorrect information. I could be entirely wrong.
You might want to enclose the query in a try...catch...finally block to
ensure that the database connection is closed even if the query fails.
Thanks so much
Bob
Post by PB
Hi Bob,
1.
MongoDB server version: 4.0.0, spring-data-mongodb :2.0.8.RELEASE and
mongo-java-driver: 3.8.0
Spring version 5.0.7.RELEASE
2.
Windows OS
3.
My Mongo Database has a *collection* with 173 666 558 documents. Each
{
"_id" : ObjectId("5b57527773ce360ce0ed4575"),
"field6" : "00:00:00",
"field7" : "00:00:00",
"field5" : "0000000000000000000",
"*field4*" : "3822",
"field8" : "3",
"*field1*" : "xxxxx",l
"*field2*" : "1.0",
"*field3*" : "5002",
"_class" : "clazz"
}
4.
I have a* String list with 967 989 elements*.
I want to know which documents in my collection (depicted in 3. ) have
field1, field2 and field3 equal to FiELD1, FIELD2 and FIELD3 query
parameters and also *field4 equal to a value not present in my list
with *967989 elements also given as a query parameter.
Post by Robert Cochran
Hi!
1. What version of MongoDB are you running this query on?
2. What operating system are you running MongoDB on?
3. Can you show a few documents, such that the structure of each can be
understood.
4. Can you explain in writing the exact goal of the query? What are you
attempting to solve -- and how do you want to solve it, in words rather
than code?
Thanks
Bob
Post by PB
Hi, I'm trying to runt this code, but I always get document size
I also tried aggregation but I get a similar document size exception.
MongoClient mongoClient = new MongoClient(host, port);
MongoCollection<Document> col =
mongoClient.getDatabase(db).getCollection(collection;
int offset = 0;
int limit = 10000;
BasicDBObject whereQuery = new BasicDBObject();
whereQuery.put(FIELD1,"FIELD1");
whereQuery.put(FIELD2, "FIELD2");
whereQuery.put(FIELD3, "FIELD3");
whereQuery.put(FIELD4 new BasicDBObject("$nin", list));
Set<String> set= new HashSet<>();
while (offset < totalCount) {
*List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());*
for (Document doc : documents) {
set.add(doc.getString(field));
}
offset += limit;
}
//process last
List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());
for (Document doc : documents) {
set.add(doc.getString(field));
}
mongoClient.close();
When this line of code " *col.find(whereQuery).skip(offset).limit(limit).into(new
org.bson.BsonMaximumSizeExceededException: Document size of 18339335
is larger than maximum of 16793600. at
org.bson.BsonBinaryWriter.validateSize(BsonBinaryWriter.java:418) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.backpatchSize(BsonBinaryWriter.java:412) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.doWriteEndArray(BsonBinaryWriter.java:153) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.AbstractBsonWriter.writeEndArray(AbstractBsonWriter.java:350) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.LevelCountingBsonWriter.writeEndArray(LevelCountingBsonWriter.java:78)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:85) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:60)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:238)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:188)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandMessage.encodeMessageBodyWithMetadata(CommandMessage.java:136)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.encode(RequestMessage.java:138)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandMessage.encode(CommandMessage.java:57)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:244)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:444)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:200)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:242)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:233)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:136)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation$1.call(FindOperation.java:701) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation$1.call(FindOperation.java:695) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:457)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:401)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation.execute(FindOperation.java:695) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation.execute(FindOperation.java:83) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:179)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:132)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:86)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.forEach(MongoIterableImpl.java:110)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.into(MongoIterableImpl.java:122)
~[mongo-java-driver-3.8.0.jar:na]
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/0c36e579-add7-4cff-a21f-ab4be5b2ca5b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
paulo Leite
2018-07-27 09:43:13 UTC
Permalink
Hi Bob,

Yes, all the fields are of String type,

Regards,

PB
Post by Robert Cochran
Hi PB,
In the sample document you describe above, are all the fields except the
_id field strings? Or are they some other data format, e.g. NumberDecimal,
ISODate, etc.?
Thanks
Bob
Post by PB
Hi Bob,
Thank you for your answer.
Regarding the 16Mb limit, I agree with you on the size of the query
parameter list, it has more than twice the size of 16 Mb and that is
probably the cause.
If I run the query by steps partioning the list, I will always get
documents with field4 not present in the sublists.
So for now I still am stuck with this.
For managing database connections I use try-with-resources.
quarta-feira, 25 de Julho de 2018 às 02:47:55 UTC+1, Robert Cochran
Post by Robert Cochran
Hi!
Thank you for posting these details. They are very helpful.
I believe the list you refer to of 967 989 elements is causing the query
document itself to become larger than 16 megabytes in size. That in turn is
causing your query to fail.
I'm not sure how to create a query that will satisfy your requirements.
You may have to split your one list into several smaller lists and find
some way of processing these lists in a progression of steps.
Now that I've ventured my opinion of what the issue is, let me also say
that I am not very experienced with MongoDB queries. So I could be giving
you incorrect information. I could be entirely wrong.
You might want to enclose the query in a try...catch...finally block to
ensure that the database connection is closed even if the query fails.
Thanks so much
Bob
Post by PB
Hi Bob,
1.
MongoDB server version: 4.0.0, spring-data-mongodb :2.0.8.RELEASE and
mongo-java-driver: 3.8.0
Spring version 5.0.7.RELEASE
2.
Windows OS
3.
My Mongo Database has a *collection* with 173 666 558 documents. Each
{
"_id" : ObjectId("5b57527773ce360ce0ed4575"),
"field6" : "00:00:00",
"field7" : "00:00:00",
"field5" : "0000000000000000000",
"*field4*" : "3822",
"field8" : "3",
"*field1*" : "xxxxx",l
"*field2*" : "1.0",
"*field3*" : "5002",
"_class" : "clazz"
}
4.
I have a* String list with 967 989 elements*.
I want to know which documents in my collection (depicted in 3. ) have
field1, field2 and field3 equal to FiELD1, FIELD2 and FIELD3 query
parameters and also *field4 equal to a value not present in my list
with *967989 elements also given as a query parameter.
Post by Robert Cochran
Hi!
1. What version of MongoDB are you running this query on?
2. What operating system are you running MongoDB on?
3. Can you show a few documents, such that the structure of each can
be understood.
4. Can you explain in writing the exact goal of the query? What are
you attempting to solve -- and how do you want to solve it, in words rather
than code?
Thanks
Bob
Post by PB
Hi, I'm trying to runt this code, but I always get document size
I also tried aggregation but I get a similar document size exception.
MongoClient mongoClient = new MongoClient(host, port);
MongoCollection<Document> col = mongoClient.getDatabase(db).ge
tCollection(collection;
int offset = 0;
int limit = 10000;
BasicDBObject whereQuery = new BasicDBObject();
whereQuery.put(FIELD1,"FIELD1");
whereQuery.put(FIELD2, "FIELD2");
whereQuery.put(FIELD3, "FIELD3");
whereQuery.put(FIELD4 new BasicDBObject("$nin", list));
Set<String> set= new HashSet<>();
while (offset < totalCount) {
*List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());*
for (Document doc : documents) {
set.add(doc.getString(field));
}
offset += limit;
}
//process last
List<Document> documents = col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());
for (Document doc : documents) {
set.add(doc.getString(field));
}
mongoClient.close();
When this line of code " *col.find(whereQuery).skip(offset).limit(limit).into(new
org.bson.BsonMaximumSizeExceededException: Document size of 18339335
is larger than maximum of 16793600. at org.bson.BsonBinaryWriter.vali
dateSize(BsonBinaryWriter.java:418) ~[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.backpatchSize(BsonBinaryWriter.java:412) ~
[mongo-java-driver-3.8.0.jar:na] at org.bson.BsonBinaryWriter.doWr
na] at org.bson.AbstractBsonWriter.writeEndArray(AbstractBsonWriter.java:350)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connectio
n.LevelCountingBsonWriter.writeEndArray(LevelCountingBsonWriter.java:78)
~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonArrayCodec
.encode(BsonArrayCodec.java:85) ~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.EncoderContext
.encodeWithChildContext(EncoderContext.java:91) ~
[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCo
dec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCo
na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41)
~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.configuration.
LazyCodec.encode(LazyCodec.java:37) ~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCo
dec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCo
na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41)
~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.configuration.
LazyCodec.encode(LazyCodec.java:37) ~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:60)
~[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentWr
apperCodec.encode(BsonDocumentWrapperCodec.java:29) ~
[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.EncoderContext
.encodeWithChildContext(EncoderContext.java:91) ~
[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCo
dec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at org.bson.codecs.BsonDocumentCo
na] at org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connectio
n.RequestMessage.addDocument(RequestMessage.java:238) ~
[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connectio
n.RequestMessage.addDocument(RequestMessage.java:188) ~
[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connectio
n.CommandMessage.encodeMessageBodyWithMetadata(CommandMessage.java:136)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connectio
n.RequestMessage.encode(RequestMessage.java:138) ~
[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connectio
n.CommandMessage.encode(CommandMessage.java:57) ~
[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connectio
n.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:244)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connectio
n.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connectio
n.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:444)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connectio
n.CommandProtocolImpl.execute(CommandProtocolImpl.java:72) ~
[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connectio
n.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:200)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connectio
n.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connectio
n.DefaultServerConnection.command(DefaultServerConnection.java:131) ~
[mongo-java-driver-3.8.0.jar:na] at com.mongodb.internal.connectio
n.DefaultServerConnection.command(DefaultServerConnection.java:123) ~
[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.CommandO
perationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:242)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.CommandO
perationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:233)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.CommandO
perationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:136)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.FindOper
na] at com.mongodb.operation.FindOperation$1.call(FindOperation.java:695)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.Operatio
nHelper.withConnectionSource(OperationHelper.java:457) ~
[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.Operatio
nHelper.withConnection(OperationHelper.java:401) ~
[mongo-java-driver-3.8.0.jar:na] at com.mongodb.operation.FindOper
na] at com.mongodb.operation.FindOperation.execute(FindOperation.java:83)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.Mo
ngoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:179)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.Mo
ngoIterableImpl.execute(MongoIterableImpl.java:132) ~
[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.Mo
ngoIterableImpl.iterator(MongoIterableImpl.java:86) ~
[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.Mo
ngoIterableImpl.forEach(MongoIterableImpl.java:110) ~
[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.Mo
ngoIterableImpl.into(MongoIterableImpl.java:122) ~
[mongo-java-driver-3.8.0.jar:na]
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/
msgid/mongodb-user/0c36e579-add7-4cff-a21f-ab4be5b2ca5b%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/0c36e579-add7-4cff-a21f-ab4be5b2ca5b%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Cumprimentos,

Paulo Leite
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CA%2BSZT3QATLMZOoeU%3DKfA0n9SwRv6EVH29626398H0dzMtH%3Dhew%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Robert Cochran
2018-07-30 22:57:03 UTC
Permalink
Hi PB,

So all the fields in each document occur in the order shown and with the
names that are shown, is that correct? For example, after _id, we see
"field6", "field7", "field5"... in that order?

Thanks

Bob
Post by PB
Hi Bob,
Yes, all the fields are of String type,
Regards,
PB
Post by Robert Cochran
Hi PB,
In the sample document you describe above, are all the fields except the
_id field strings? Or are they some other data format, e.g. NumberDecimal,
ISODate, etc.?
Thanks
Bob
Post by PB
Hi Bob,
Thank you for your answer.
Regarding the 16Mb limit, I agree with you on the size of the query
parameter list, it has more than twice the size of 16 Mb and that is
probably the cause.
If I run the query by steps partioning the list, I will always get
documents with field4 not present in the sublists.
So for now I still am stuck with this.
For managing database connections I use try-with-resources.
quarta-feira, 25 de Julho de 2018 às 02:47:55 UTC+1, Robert Cochran
Post by Robert Cochran
Hi!
Thank you for posting these details. They are very helpful.
I believe the list you refer to of 967 989 elements is causing the
query document itself to become larger than 16 megabytes in size. That in
turn is causing your query to fail.
I'm not sure how to create a query that will satisfy your requirements.
You may have to split your one list into several smaller lists and find
some way of processing these lists in a progression of steps.
Now that I've ventured my opinion of what the issue is, let me also say
that I am not very experienced with MongoDB queries. So I could be giving
you incorrect information. I could be entirely wrong.
You might want to enclose the query in a try...catch...finally block to
ensure that the database connection is closed even if the query fails.
Thanks so much
Bob
Post by PB
Hi Bob,
1.
MongoDB server version: 4.0.0, spring-data-mongodb :2.0.8.RELEASE and
mongo-java-driver: 3.8.0
Spring version 5.0.7.RELEASE
2.
Windows OS
3.
My Mongo Database has a *collection* with 173 666 558 documents. Each
{
"_id" : ObjectId("5b57527773ce360ce0ed4575"),
"field6" : "00:00:00",
"field7" : "00:00:00",
"field5" : "0000000000000000000",
"*field4*" : "3822",
"field8" : "3",
"*field1*" : "xxxxx",l
"*field2*" : "1.0",
"*field3*" : "5002",
"_class" : "clazz"
}
4.
I have a* String list with 967 989 elements*.
I want to know which documents in my collection (depicted in 3. ) have
field1, field2 and field3 equal to FiELD1, FIELD2 and FIELD3 query
parameters and also *field4 equal to a value not present in my list
with *967989 elements also given as a query parameter.
Post by Robert Cochran
Hi!
1. What version of MongoDB are you running this query on?
2. What operating system are you running MongoDB on?
3. Can you show a few documents, such that the structure of each can
be understood.
4. Can you explain in writing the exact goal of the query? What are
you attempting to solve -- and how do you want to solve it, in words rather
than code?
Thanks
Bob
Post by PB
Hi, I'm trying to runt this code, but I always get document size
I also tried aggregation but I get a similar document size exception.
MongoClient mongoClient = new MongoClient(host, port);
MongoCollection<Document> col =
mongoClient.getDatabase(db).getCollection(collection;
int offset = 0;
int limit = 10000;
BasicDBObject whereQuery = new BasicDBObject();
whereQuery.put(FIELD1,"FIELD1");
whereQuery.put(FIELD2, "FIELD2");
whereQuery.put(FIELD3, "FIELD3");
whereQuery.put(FIELD4 new BasicDBObject("$nin", list));
Set<String> set= new HashSet<>();
while (offset < totalCount) {
*List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());*
for (Document doc : documents) {
set.add(doc.getString(field));
}
offset += limit;
}
//process last
List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());
for (Document doc : documents) {
set.add(doc.getString(field));
}
mongoClient.close();
When this line of code " *col.find(whereQuery).skip(offset).limit(limit).into(new
org.bson.BsonMaximumSizeExceededException: Document size of 18339335
is larger than maximum of 16793600. at
org.bson.BsonBinaryWriter.validateSize(BsonBinaryWriter.java:418) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.backpatchSize(BsonBinaryWriter.java:412) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.doWriteEndArray(BsonBinaryWriter.java:153) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.AbstractBsonWriter.writeEndArray(AbstractBsonWriter.java:350) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.LevelCountingBsonWriter.writeEndArray(LevelCountingBsonWriter.java:78)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:85) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:60)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:238)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:188)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandMessage.encodeMessageBodyWithMetadata(CommandMessage.java:136)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.encode(RequestMessage.java:138)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandMessage.encode(CommandMessage.java:57)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:244)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:444)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:200)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:242)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:233)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:136)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation$1.call(FindOperation.java:701) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation$1.call(FindOperation.java:695) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:457)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:401)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation.execute(FindOperation.java:695) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation.execute(FindOperation.java:83) ~
[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:179)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:132)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:86)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoIterableImpl.forEach(MongoIterableImpl.java:110)
~[mongo-java-driver-3.8.0.jar:na] at com.mongodb.client.internal.MongoIterableImpl.into(MongoIterableImpl.java:122)
~[mongo-java-driver-3.8.0.jar:na]
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/0c36e579-add7-4cff-a21f-ab4be5b2ca5b%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/0c36e579-add7-4cff-a21f-ab4be5b2ca5b%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Cumprimentos,
Paulo Leite
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/c546e444-ffd6-44af-9ed5-9320370aad7c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Ramon Fernandez' via mongodb-user
2018-07-25 13:39:51 UTC
Permalink
Please take a look at the Java Driver Quick Start Guide
<http://mongodb.github.io/mongo-java-driver/3.8/driver/getting-started/quick-start/>,
which has some useful information that should help you here, specially
around using cursors.

Cheers,
Ramón.
Post by PB
Hi, I'm trying to runt this code, but I always get document size
I also tried aggregation but I get a similar document size exception.
MongoClient mongoClient = new MongoClient(host, port);
MongoCollection<Document> col =
mongoClient.getDatabase(db).getCollection(collection;
int offset = 0;
int limit = 10000;
BasicDBObject whereQuery = new BasicDBObject();
whereQuery.put(FIELD1,"FIELD1");
whereQuery.put(FIELD2, "FIELD2");
whereQuery.put(FIELD3, "FIELD3");
whereQuery.put(FIELD4 new BasicDBObject("$nin", list));
Set<String> set= new HashSet<>();
while (offset < totalCount) {
*List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());*
for (Document doc : documents) {
set.add(doc.getString(field));
}
offset += limit;
}
//process last
List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());
for (Document doc : documents) {
set.add(doc.getString(field));
}
mongoClient.close();
When this line of code " *col.find(whereQuery).skip(offset).limit(limit).into(new
org.bson.BsonMaximumSizeExceededException: Document size of 18339335 is
larger than maximum of 16793600. at
org.bson.BsonBinaryWriter.validateSize(BsonBinaryWriter.java:418) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.backpatchSize(BsonBinaryWriter.java:412) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.doWriteEndArray(BsonBinaryWriter.java:153) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.AbstractBsonWriter.writeEndArray(AbstractBsonWriter.java:350) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.LevelCountingBsonWriter.writeEndArray(LevelCountingBsonWriter.java:78)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:85) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:60)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:238)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:188)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandMessage.encodeMessageBodyWithMetadata(CommandMessage.java:136)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.encode(RequestMessage.java:138)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandMessage.encode(CommandMessage.java:57)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:244)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:444)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:200)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:242)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:233)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:136)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation$1.call(FindOperation.java:701) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation$1.call(FindOperation.java:695) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:457)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:401)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation.execute(FindOperation.java:695) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation.execute(FindOperation.java:83) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:179)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:132)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:86)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.forEach(MongoIterableImpl.java:110)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.into(MongoIterableImpl.java:122)
~[mongo-java-driver-3.8.0.jar:na]
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/61476e6e-fd3e-4498-affc-cd5087aebcd0%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/61476e6e-fd3e-4498-affc-cd5087aebcd0%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAMW1i2VEFuFxdzutvZVA21SuBh%2B8z9VnPw_BrSO924SpeyFcxQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
PB
2018-07-26 13:42:24 UTC
Permalink
Hi Ramón,


Thanks, I had a look but did not help.


cheers

quarta-feira, 25 de Julho de 2018 às 14:40:45 UTC+1, Ramon Fernandez
Post by 'Ramon Fernandez' via mongodb-user
Please take a look at the Java Driver Quick Start Guide
<http://mongodb.github.io/mongo-java-driver/3.8/driver/getting-started/quick-start/>,
which has some useful information that should help you here, specially
around using cursors.
Cheers,
Ramón.
Post by PB
Hi, I'm trying to runt this code, but I always get document size
I also tried aggregation but I get a similar document size exception.
MongoClient mongoClient = new MongoClient(host, port);
MongoCollection<Document> col =
mongoClient.getDatabase(db).getCollection(collection;
int offset = 0;
int limit = 10000;
BasicDBObject whereQuery = new BasicDBObject();
whereQuery.put(FIELD1,"FIELD1");
whereQuery.put(FIELD2, "FIELD2");
whereQuery.put(FIELD3, "FIELD3");
whereQuery.put(FIELD4 new BasicDBObject("$nin", list));
Set<String> set= new HashSet<>();
while (offset < totalCount) {
*List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());*
for (Document doc : documents) {
set.add(doc.getString(field));
}
offset += limit;
}
//process last
List<Document> documents =
col.find(whereQuery).skip(offset).limit(limit).into(new
ArrayList<Document>());
for (Document doc : documents) {
set.add(doc.getString(field));
}
mongoClient.close();
When this line of code " *col.find(whereQuery).skip(offset).limit(limit).into(new
org.bson.BsonMaximumSizeExceededException: Document size of 18339335 is
larger than maximum of 16793600. at
org.bson.BsonBinaryWriter.validateSize(BsonBinaryWriter.java:418) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.backpatchSize(BsonBinaryWriter.java:412) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.BsonBinaryWriter.doWriteEndArray(BsonBinaryWriter.java:153) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.AbstractBsonWriter.writeEndArray(AbstractBsonWriter.java:350) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.LevelCountingBsonWriter.writeEndArray(LevelCountingBsonWriter.java:78)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:85) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonArrayCodec.encode(BsonArrayCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.configuration.LazyCodec.encode(LazyCodec.java:37) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:60)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentWrapperCodec.encode(BsonDocumentWrapperCodec.java:29)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.EncoderContext.encodeWithChildContext(EncoderContext.java:91)
~[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.writeValue(BsonDocumentCodec.java:136) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:115) ~
[mongo-java-driver-3.8.0.jar:na] at
org.bson.codecs.BsonDocumentCodec.encode(BsonDocumentCodec.java:41) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:238)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.addDocument(RequestMessage.java:188)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandMessage.encodeMessageBodyWithMetadata(CommandMessage.java:136)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.RequestMessage.encode(RequestMessage.java:138)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandMessage.encode(CommandMessage.java:57)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:244)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:444)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:200)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:123)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:242)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:233)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:136)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation$1.call(FindOperation.java:701) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation$1.call(FindOperation.java:695) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:457)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:401)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation.execute(FindOperation.java:695) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.operation.FindOperation.execute(FindOperation.java:83) ~
[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:179)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.execute(MongoIterableImpl.java:132)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.iterator(MongoIterableImpl.java:86)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.forEach(MongoIterableImpl.java:110)
~[mongo-java-driver-3.8.0.jar:na] at
com.mongodb.client.internal.MongoIterableImpl.into(MongoIterableImpl.java:122)
~[mongo-java-driver-3.8.0.jar:na]
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/61476e6e-fd3e-4498-affc-cd5087aebcd0%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/61476e6e-fd3e-4498-affc-cd5087aebcd0%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/744ef674-2b35-4e2d-ad47-853d02a3e916%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Wan Bachtiar' via mongodb-user
2018-08-02 08:01:06 UTC
Permalink
Regarding the 16Mb limit, I agree with you on the size of the query
parameter list, it has more than twice the size of 16 Mb and that is
probably the cause.

Hi PB,

Based on the error message, this is the reason why
org.bson.BsonBinaryWriter.validateSize() is throwing
BsonMaximumSizeExceededException. The total size of your query is greater
than the maximum BSON size allowed to be serialized and sent via the wire.

I would suggest to reconsider your data model
<https://docs.mongodb.com/manual/core/data-modeling-introduction/>, and
reassess how the application interacts with the data.

Alternatively, you could store the list of Field4 into another collection,
and utilise $lookup
<https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/> to
perform a left outer join between the two collections.

Regards,
Wan.
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/19f9329c-dfe8-4c3c-ab4d-86f35ecae449%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Continue reading on narkive:
Loading...