Discussion:
MongoDB NodeJS aggregations - stops finding data after a while
(too old to reply)
Rodolfo Perottoni
2018-03-23 07:24:15 UTC
Permalink
Bit of a tricky issue I'm facing and can't find any clue of what might be
going on.
My backend is made of an Express application exposing many endpoints that
interact with the MongoDB nodeJS driver.
I have set up a *cron *that calls one of my endpoints every 15 minutes, and
in this specific endpoints I have a huge aggregation pipeline being
executed.

The issue that I'm facing is that after 4-5 hours this query will simply
stop returning documents.

Important considerations:
- My crontab is running without any problem. This endpoint is sending a
message to a private slack channel and I can see it popping up every 15
minutes like I wanted.
- I know that my aggregation stopped working because this message that I
receive in Slack tells me how many documents were found by that query.
- I also log this aggregation's result to a separate collection, so the
connection with my DB is not a problem.

*CATCH 1: *I can copy this same aggregation command (with the Express app
still running) and put it into Robo 3T/Mongo Shell, execute it and *STILL*
get results out of it. It's as if my application simply can't see that data
in the database.
*CATCH 2: *As soon as I restart my express application this aggregation
will find all the data that has been "stuck" since the past few hours. This
makes me think that the MongoDB node driver might be the culprit.

I've found a guy with the same issue on StackOverflow but no one has
answered that question yet.
https://stackoverflow.com/questions/44144277/mongoose-mongodb-in-node-js-stops-returning-records-after-a-while
The only difference between this issue and mine is that I don't use
Mongoose.

Thanks everyone.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/febe016e-4c2f-4048-9a00-53e559b43521%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Kevin Adistambha' via mongodb-user
2018-04-03 05:29:59 UTC
Permalink
Hi

Could you please post more information about your issue:

I know that my aggregation stopped working because this message that I
receive in Slack tells me how many documents were found by that query.

My understanding is your end point continues to send a message to Slack
every 15 minutes but the count of documents returned by your aggregation
query is 0 after 4-5 hours. Is this correct?

I also log this aggregation’s result to a separate collection, so the
connection with my DB is not a problem.

How are you logging the result into a separate collection? Are you using
the $out operator?

I can copy this same aggregation command (with the Express app still
running) and put it into Robo 3T/Mongo Shell, execute it and STILL get
results out of it.

Please post the aggregation query in question, along with some example
documents. How long does the aggregation usually run for? Does it take more
than 15 minutes at some times?

As soon as I restart my express application this aggregation will find all
the data that has been “stuck” since the past few hours

Please post a small example of the endpoint code. It would be very helpful
if other people can reproduce the issue you’re seeing.

Could you also post the node driver version you’re using?

Are you seeing the same issue using a local deployment as well, or is it
only when you’re connecting to Atlas?

Best regards
Kevin
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/2a386c58-2e0a-4905-a359-7b712afb2e47%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Rodolfo Perottoni
2018-04-04 01:55:00 UTC
Permalink
Hi Kevin. Thanks for coming back to me.

My understanding is your end point continues to send a message to Slack
Post by 'Kevin Adistambha' via mongodb-user
every 15 minutes but the count of documents returned by your aggregation
query is 0 after 4-5 hours. Is this correct?
Correct.

How are you logging the result into a separate collection? Are you using
Post by 'Kevin Adistambha' via mongodb-user
the $out operator?
No, I'm manually logging it within my application's code. E.g. calling
*insertOne* in a separate logs collection.

Please post the aggregation query in question, along with some example
Post by 'Kevin Adistambha' via mongodb-user
documents. How long does the aggregation usually run for? Does it take more
than 15 minutes at some times?
You can find the aggregation attached to this reply. Unfortunately I can't
disclose the structure of my database objects.
And no, it never took more than 300-500ms to run.

Please post a small example of the endpoint code. It would be very helpful
Post by 'Kevin Adistambha' via mongodb-user
if other people can reproduce the issue you’re seeing.
While trying to fix this issue this was one of my first assumptions. So I
moved my whole query to an empty Express route just like the ones you see
on their docs to test whether it had something to do with my protected
routes or business logic. This is what it would like (I can't post the
original code), e.g.:

app.post('/route', async (req, res) => { const results = await collection.
aggregate(<pipeline>).toArray(); <my business logic that uses the "results"
object> });

I verified this issue happens on both the Node version 2.2 and 3.0;
I tried upgrading my Atlas server to M10 to guarantee this was not a free
tier limitation and the issue was still there after 4-5hs of server uptime.

On a side note, I already "translated" my aggregation pipeline into
individual queries (had to deliver this functionality ASAP) and everything
is working fine.
This is proof enough to me that the problem lives in the aggregation
framework.
Let me know if there's any other information that might help you on this.
Post by 'Kevin Adistambha' via mongodb-user
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/dd2d4fdb-d3cb-4207-8c78-e5aa640ff1a6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Rodolfo Perottoni
2018-04-04 01:56:29 UTC
Permalink
Pipeline attached to this message
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/f7161567-d69a-4210-ad78-b0d4cb214779%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Kevin Adistambha' via mongodb-user
2018-05-07 06:16:51 UTC
Permalink
Hi Rodolfo

Apologies for the delay in getting back to you, but nothing seems to jump
at me. However, the fact that it seems to freeze after a certain amount of
time while a manually run query would return something seems to suggest
that there is unlikely that anything is wrong with the query.

If you’re still having this issue and would like to dig deeper, I would
suggest to try:

- Try running the code by pointing the application toward a local
MongoDB instance instead of Atlas, and see if it also stops after 4-5
hours. If it does, then there could be something in the interaction of
Atlas and the Javascript code.
- Try running the periodic querying using another driver, e.g. the
Python driver to check if it also stops after 4-5 hours. If it does, then
the node driver is not the issue.
- After the application stops receiving results in 4-5 hours, does an
application restart immediately resume the stream of results? If it does,
then it could be some timeout, or some limitation being hit.
- Any other method you could think of to narrow down the issue will be
helpful.

Best regards
Kevin
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/0634fb4f-a7bf-4228-bf74-92a6aea92d4b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
miquel ferrando
2018-08-16 06:52:24 UTC
Permalink
Hi Rodolfo

Did you found how to solve that problem? I'm having the same issue but in
my case is with a find not an aggregate.

I also have an API to call my MongoDB (I use Mongoose) and it stops
responding even with an empty find.

In my case i saw that the issue ocurs to a certain collections while the
other ones still working and after restarting the API all starts working
again.

I think that the issue might be related to the driver.

Regards Miquel.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/f0a53dcb-3957-4334-8abd-0fdc4e03d2cf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Rodolfo Perottoni
2018-08-16 23:04:40 UTC
Permalink
Hi Miquel.
I haven't found a solution at all after 1 month of investigation and
frustration. I also had a chat with MongoDB through the phone and all they
did was push me an enterprise plan so I could get "enterprise-grade"
support.
We (business) decided not to spend more time on it and gave up on
aggregations - all of our business logic now runs with *find* queries. No
issues since then.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/b8a711a4-4101-48ec-b746-c060203bd14c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
miquel ferrando
2018-08-20 06:34:56 UTC
Permalink
Hi Rodolfo.

In our case that happends with finds :(

El viernes, 17 de agosto de 2018, 1:04:41 (UTC+2), Rodolfo Perottoni
Post by Rodolfo Perottoni
Hi Miquel.
I haven't found a solution at all after 1 month of investigation and
frustration. I also had a chat with MongoDB through the phone and all they
did was push me an enterprise plan so I could get "enterprise-grade"
support.
We (business) decided not to spend more time on it and gave up on
aggregations - all of our business logic now runs with *find* queries. No
issues since then.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/f72b1e29-150b-4960-a696-fa17b7f9cea2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Wan Bachtiar' via mongodb-user
2018-08-20 01:49:29 UTC
Permalink
app.post(‘/route’, async (req, res) => { const results = await
collection.aggregate().toArray(); });

Hi Rodolfo,

Do you wrap the await aggregation in a try/catch ?
Just in case there’s any error with your code before the return of the
endpoint response.

Also do you verify whether session always has a value ?
Do you check whether the conditional $filter for session (dynamic periodic
date range) always return a value ?

had a chat with MongoDB through the phone and all they did was push me an
enterprise plan so I could get “enterprise-grade” support.

I would think that is a reasonable approach, as you said that you can’t
disclose some of the proprietary code and environment. Signing an official
support contract would allow MongoDB support to look further into your use
case with some sort of NDA in place.

Best regards,
Wan.
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/994416ab-9048-4781-91fd-f6ac1246de12%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Rodolfo Perottoni
2018-08-20 02:17:06 UTC
Permalink
Hi Wan.
Post by 'Wan Bachtiar' via mongodb-user
Do you wrap the await aggregation in a try/catch ?
Just in case there’s any error with your code before the return of the
endpoint response.
​
Yep, the code had way more than just a aggregate().toArray(). I do have
every route on our API encapsulated in an error handling library and never
had any errors being thrown by the aggregation.

Also do you verify whether session always has a value ?
Post by 'Wan Bachtiar' via mongodb-user
Do you check whether the conditional $filter for session (dynamic
periodic date range) always return a value ?
I replicated this issue on a new Atlas environment during my tests and my
query kept returning values for only 2 to 3 hours. On my first message in
this discussion I've outlined how I could still execute the query on the
Mongo Shell and get results, hence why I'm pretty sure there's something
wrong with the NodeJS driver.


This issue has drove me nuts. Every time I had to explain it to someone the
outcome is always the same - doubtful or skeptical questions. And I get it,
it sounds like either I'm lying or just a beginner in this ecosystem. I'd
react the same way if someone told me such an issue exists.
After more than 20 hours working and debugging to ensure this is a NodeJS
driver issue, I'm gonna put a nail to this coffin by withdrawing myself
from this post and leaving it for posterity. I wish I had an enterprise
plan to get someone from MongoDB to look at this, but at this stage in our
business this is simply out of equation.

Thanks everyone for your input. Over and out.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/4e46604a-7a91-4a8d-b847-542294884138%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Daniele Tassone
2018-08-22 13:34:07 UTC
Permalink
Rodolfo,

can you post the MongoDB connection string? (without use/pass).

D.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/c9030f5e-c16d-413b-b665-c4c004700a1f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Wan Bachtiar' via mongodb-user
2018-08-21 06:24:57 UTC
Permalink
Every time I had to explain it to someone the outcome is always the same -
doubtful or skeptical questions. And I get it, it sounds like either I’m
lying or just a beginner in this ecosystem. I’d react the same way if
someone told me such an issue exists.

Hi Rodolfo,

This problem doesn’t appear to be trivially reproducible and without
further details it’s likely there won’t be any progress. It would be great
if you could provide a minimal, complete, and reproducible code example.

If this is a bug with MongoDB Node.JS driver, it would be preferable if a
bug report could come out from this discussion and fix it for others too.

In our case that happens with finds :(

Miquel,
To help focus on the details of your issue, please start a new discussion
with relevant details including:

- Mongoose version
- MongoDB Node.JS driver version
- A minimal, complete code that would be able to test/reproduce the issue
- MongoDB server version
- Any mongod log events around the time of the find not returning any
data

Regards,
Wan.
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/966ab207-a788-44d1-a07c-acfaaa435013%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
miquel ferrando
2018-08-21 08:11:59 UTC
Permalink
Hi Wan,
Those are the versions of my envirorment.

Mongoose:
I tried several versions of mongoose and it happends with all
mongoose 5.0.9 wich use (mongodb 3.0.4)
mongoose 5.2.7 & 5.2.8 (mongodb 3.1.1)

MongoDB server 3.6.3

Node.JS version 9.10.1

Server Ubuntu 16.04.4 LTS

About my scenario, it happends with several collections but i will descrive
one.
We have an API (Nose.JS & Mongoose) and some services (Node.JS) that calls
to our API.
In one of that services we have two process, one that insert documents with
an ReportRequestID that an Amazon API provides us and a ReportReadyDate
(about 1 minute later the date of the insert), and another proces (like an
infinite loop) that every minute, queries the DB asking for documents with
the ReportReadyDate < now, when it's ready, we procces the report and
delete the document, the query asking for the documents is the one that
fails, and the documents in the collection starts to stack (we can see that
througt Robo 3T and the mongoShell), and even if we call our API from
another service, even with an empty query, doesn't respond with any of that
documents until we restart the API.

I'm prety sure that the query is not the problem because it happends also
with other collections with much simpler queries (like find id == queryId)
but this is the case that I have studied the most because is one of the
most critical.

Our query it's constructed depending on a GET but in that case the
resulting query it's like that:

Mongoose Log:
amz_reports.find((
{
$and: [
{ reportId: { $exists: false } },
{
$or: [
{ readyDate: { $exists: false } },
{
readyDate: {
$lt: new Date('Tue, 21 Aug 2018 07:53:16 GMT')
}
}
]
},
{
$or: [
{ lastChecked: { $exists: false } },
{
lastChecked: {
$lt: new Date('Mon, 20 Aug 2018 07:53:16 GMT')
}
}
]
}
]
},
{ sort: { submitDate: 1 }, limit: 1, fields: {} }
)


And the code that calls it is like

amz_reports.find(query, project).limit(1).sort({ submitDate: 1 }).lean().
exec((err, results) => {
if (err) Promise.reject(err);
else Promise.resolve(results);
});

Thanks for your support Wan.

Regards, Miquel.
Post by 'Wan Bachtiar' via mongodb-user
Every time I had to explain it to someone the outcome is always the same -
doubtful or skeptical questions. And I get it, it sounds like either I’m
lying or just a beginner in this ecosystem. I’d react the same way if
someone told me such an issue exists.
Hi Rodolfo,
This problem doesn’t appear to be trivially reproducible and without
further details it’s likely there won’t be any progress. It would be great
if you could provide a minimal, complete, and reproducible code example.
If this is a bug with MongoDB Node.JS driver, it would be preferable if a
bug report could come out from this discussion and fix it for others too.
In our case that happens with finds :(
Miquel,
To help focus on the details of your issue, please start a new discussion
- Mongoose version
- MongoDB Node.JS driver version
- A minimal, complete code that would be able to test/reproduce the issue
- MongoDB server version
- Any mongod log events around the time of the find not returning any
data
Regards,
Wan.
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/6fc054fc-e007-4c70-8795-e54d4bd3073f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Wan Bachtiar' via mongodb-user
2018-08-27 06:21:22 UTC
Permalink
Hi Miquel,

Could you please start a new discussion thread (new post) to avoid mixing
with Rodolfo question about aggregation.

I’ve tried your setup briefly, but as mentioned previously this is not
trivial to reproduce.
In the new discussion post, please provide a minimal and complete code that
would be able to test the issue.

Regards,
Wan.
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/7bb2aba2-3cdb-4a21-9164-60e281c40776%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Continue reading on narkive:
Loading...