Discussion:
aggregation result exceeds maximum document size (16MB)
(too old to reply)
Prasanna Ganesh
2017-03-26 05:22:39 UTC
Permalink
Hi,

I have written a java program which takes distinct records from collection
called 'A' and query another collection called 'B' which is a audit table
in a batch of 1000 unique records from collection 'A'.

Currently i'm facing an issue in running below query on collection 'B'
after executing certain number of batches.For example out of 134 batches to
execute 133k records, it fails after 74 batches. Each batch contains set of
1000 records to query against collection B which might give 5000 recors
atleast.

I understand there is size restrictions with aggregation query of 16MB and
even mentioning allowDiskUse(true) in my query its failing with below
exceptions. I saw in another post to use *$out *to resolve this issue but
i'm facing difficulty in implementing it in my query because i want to
return the result set as a list.

Really appreciate any help on this issue.

http://stackoverflow.com/questions/29644587/how-could-i-write-aggregation-without-exceeds-maximum-document-size

Query:
List<ReportsDocument> list = new ArrayList<ReportsDocument>();

TypedAggregation aggregation = Aggregation.newAggregation(ReportsDocument.
class,
match(query1),sort(Sort.Direction.ASC, "controlDocument.accountNumber")

).withOptions(new org.springframework.data.mongodb.core.aggregation.
AggregationOptions.Builder().allowDiskUse(true).build());


AggregationResults<ReportsDocument> results = mongoTemplate.aggregate(
aggregation,VCTCMongoCollections.REPORTING_HISTORY,ReportsDocument.class);

list = results.getMappedResults();

return list;






The full response is { "ok" : 0.0, "errmsg" : "aggregation result exceeds
maximum document size (16MB)", "code" : 16389 }
at org.springframework.data.mongodb.core.MongoTemplate.
handleCommandError(MongoTemplate.java:2082)
at org.springframework.data.mongodb.core.MongoTemplate.aggregate(
MongoTemplate.java:1553)
at org.springframework.data.mongodb.core.MongoTemplate.aggregate(
MongoTemplate.java:1475)
at com.visa.ip.vctc.utility.OneTimeReportNAB.computeOneTimeReport(
OneTimeReportNAB.java:318)
at com.visa.ip.vctc.utility.OneTimeReportNAB.run(OneTimeReportNAB.
java:275)
at com.visa.ip.vctc.utility.OneTimeReportNAB.main(OneTimeReportNAB.
java:179)
Caused by: com.mongodb.MongoCommandException: Command failed with error
16389: 'aggregation result exceeds maximum document size (16MB)' on server
xxxxx:27017. The full response is { "ok" : 0.0, "errmsg" : "aggregation
result exceeds maximum document size (16MB)", "code" : 16389 }
at com.mongodb.CommandResult.getException(CommandResult.java:80)
at com.mongodb.CommandResult.throwOnError(CommandResult.java:94)
at org.springframework.data.mongodb.core.MongoTemplate.
handleCommandError(MongoTemplate.java:2076)
... 13 more
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/d8346645-f241-47a2-b7ab-b48cba97859d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Wan Bachtiar' via mongodb-user
2017-04-10 03:20:22 UTC
Permalink
com.mongodb.MongoCommandException: Command failed with error 16389:
‘aggregation result exceeds maximum document size (16MB)’

Hi Prasanna,

This is most likely due to the size of the aggregation query result itself
exceeding the 16MB limit. The option allowDiskUse is only to enable the
pipeline to write to temporary files during the stages. What you need is to
use cursor in order to return the results in iterable manner.

If you’re using MongoDB driver v3.4
<http://mongodb.github.io/mongo-java-driver/3.4/>, you can specify useCursor
on the AggregateIterable
<https://mongodb.github.io/mongo-java-driver/3.4/javadoc/com/mongodb/client/AggregateIterable.html>
object. for example:

AggregateIterable<Document> iterable = db.getCollection("collectionName").aggregate(aggPipeline).useCursor(true).allowDiskUse(true);

If you have further questions, please provide:

- Specific MongoDB version that you’re using
- Specific MongoDB Java version that you’re using

Regards,

Wan.
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/c660b36a-e598-4634-a1c5-a39bba5ad232%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Prashant Patil
2017-11-02 15:25:03 UTC
Permalink
Hi,

Attached is the screenshot of my eclipse environment, here you can see my
jar files.
I am using Mono-java-driver 2.1.4.

we are using spring mongo driver . Even i am facing this issue. Can you
please help me out with this?


Thanks,
Prashant.
Post by 'Wan Bachtiar' via mongodb-user
‘aggregation result exceeds maximum document size (16MB)’
Hi Prasanna,
This is most likely due to the size of the aggregation query result itself
exceeding the 16MB limit. The option allowDiskUse is only to enable the
pipeline to write to temporary files during the stages. What you need is to
use cursor in order to return the results in iterable manner.
If you’re using MongoDB driver v3.4
<http://mongodb.github.io/mongo-java-driver/3.4/>, you can specify
useCursor on the AggregateIterable
<https://mongodb.github.io/mongo-java-driver/3.4/javadoc/com/mongodb/client/AggregateIterable.html>
AggregateIterable<Document> iterable = db.getCollection("collectionName").aggregate(aggPipeline).useCursor(true).allowDiskUse(true);
- Specific MongoDB version that you’re using
- Specific MongoDB Java version that you’re using
Regards,
Wan.
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/3ea1d17d-eeeb-4cb7-a667-12c1f39174e9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Wan Bachtiar' via mongodb-user
2017-11-03 05:15:35 UTC
Permalink
Can you please help me out with this?

Hi Prashant,

I’m not entirely clear if you’re having the same issue as the original
question on this thread.
Could you please open a new discussion along with:

- MongoDB Server version
- The error message that you’re getting. (Actual error message, not
eclipse user-interface screenshot)

Could you also clarify the version of MongoDB Java driver that you’re using
?
The earliest version that is available at
http://mongodb.github.io/mongo-java-driver/ is v2.13.3 for mongo-java-driver.


Regards,
Wan.
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/dd821207-9188-419c-b653-ac817ec922e2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Prashant Patil
2017-11-03 06:02:01 UTC
Permalink
Hi Wan,

Thanks for the reply.

when i search for mongo in pom.xml i can see only the below dependency
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-mongodb</artifactId>
<version>1.9.2.RELEASE</version>
</dependency>c
And in maven dependency in eclipse these are the jars related to mongo
mongo-java-driver-2.14.0-sources.jar
spring-data-mongodb-1.9.2.RELEASE-sources.jar

I am not really sure what are the differences between the above 2 jar files.
and can you please tell me how to check mongo server version ? i m new to
this so m not understanding clearly.

It would be great help if you help me.


The error is :


com.mongodb.CommandFailureException: { "serverUsed" : "13.92.251.7:42525" ,
"ok" : 0.0 , "errmsg" : "aggregation result exceeds maximum document size
(16MB)" , "code" : 16389 , "codeName" : "Location16389"}
at com.mongodb.CommandResult.getException(CommandResult.java:76)
at com.mongodb.CommandResult.throwOnError(CommandResult.java:140)
at com.mongodb.DBCollection.aggregate(DBCollection.java:1798)
at com.mongodb.DBCollection.aggregate(DBCollection.java:1778)
at
com.flynava.jupiter.daoImpl.AnalystDaoImpl.getAnalystChannelRevenue(AnalystDaoImpl.java:237)
at
com.flynava.jupiter.serviceImpl.AnalystServiceImpl.getAnalystChannelFare(AnalystServiceImpl.java:136)
at
com.flynava.jupiter.controller.AnalystController.getAnalystChannelPie(AnalystController.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
at
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
at
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
at
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
at
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:154)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:199)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:110)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:57)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
at
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
at
org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
at
org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
at
org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:346)
at
org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:262)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:475)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:80)
at
org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:625)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342)
at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:498)
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:796)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1372)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Unknown Source)
Post by 'Wan Bachtiar' via mongodb-user
Can you please help me out with this?
Hi Prashant,
I’m not entirely clear if you’re having the same issue as the original
question on this thread.
- MongoDB Server version
- The error message that you’re getting. (Actual error message, not
eclipse user-interface screenshot)
Could you also clarify the version of MongoDB Java driver that you’re
using ?
The earliest version that is available at
http://mongodb.github.io/mongo-java-driver/ is v2.13.3 for
mongo-java-driver.
Regards,
Wan.
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/594fc544-b815-4099-9b9f-b10564a509bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Wan Bachtiar' via mongodb-user
2017-11-08 00:03:18 UTC
Permalink
I am not really sure what are the differences between the above 2 jar files.

Hi Prashant,

The spring-data-mongodb-1.9.2.RELEASE-sources.jar is a jar for Spring Data
for MongoDB <https://projects.spring.io/spring-data-mongodb/>. As Spring
Data for MongoDB provides integration between spring-data and MongoDB, it
also includes MongoDB Java Driver
<https://mongodb.github.io/mongo-java-driver/> i.e.
mongo-java-driver-2.14.0-sources.jar.

can you please tell me how to check mongo server version ?

You can use mongo shell <https://docs.mongodb.com/manual/mongo/> to connect
to your MongoDB instance, and execute db.serverBuildInfo().version
Alternatively, if you have access to the MongoDB server executable you can
execute mongod --version on the command line. The output of these two
commands will show you the server version.

im new to this so m not understanding clearly.

I would recommend to enroll in a free online course at MongoDB University
<https://university.mongodb.com/> to learn more about MongoDB. There’s also
a course related to MongoDB Java driver.

Since you’re starting out, I would also recommend to use the current stable
version of MongoDB (v3.4.10): MongoDB Download Center
<https://www.mongodb.com/download-center#community>. Similarly with Spring
Data MongoDB, you can utilise version 2.x which is compatible with MongoDB
server v3.4 and MongoDB Java driver v3.4.

“aggregation result exceeds maximum document size (16MB)”

The error message is related to the result of the aggregation pipeline
<https://docs.mongodb.com/manual/core/aggregation-pipeline/> exceeding the 16MB
BSON size limit
<https://docs.mongodb.com/manual/reference/limits/#BSON-Document-Size>.

There are two ways of solving this issue, the first option is to utilise
cursor option for the aggregation pipeline. For Spring Data MongoDB, this
option is only available since v2.0 (See DATAMONGO-1637
<https://jira.spring.io/browse/DATAMONGO-1637> for more information).

Depending on your use case, the second option is to write the output of
your aggregation pipeline to another collection. See aggregation operator
$out <https://docs.mongodb.com/manual/reference/operator/aggregation/out/>.

If you have further questions related to your MongoDB deployment or MongoDB
Java driver usage, please open a *new* thread discussion. However if you
have further question related to Spring Data MongoDB please post a question
on StackOverflow: spring-data-mongodb
<https://stackoverflow.com/questions/tagged/spring-data-mongodb> to reach
wider audience with spring-data expertise.

Regards,
Wan.
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/2bb0fb23-f46b-405c-abc7-af912137e59a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...