Discussion:
3.2.10 using significantly more RAM than CacheSizeGB
(too old to reply)
Nick Judson
2018-01-17 19:30:39 UTC
Permalink
This is occurring on a remote windows box with a cache size set to 1GB, but
where it appears to be using >3 GB. I have the metrics files but I don't
really want to post them to the MongoDB JIRA since this is an older
version. What is the best way to diagnose the memory usage? Thanks!
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/32641981-4d2d-4f0a-bb05-9c3f5b4698de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Kevin Adistambha' via mongodb-user
2018-01-22 00:45:28 UTC
Permalink
Hi Nick

Could you elaborate on how you determine that MongoDB is using >3 GB? Note
that by “cache size” I’m assuming you meant the WiredTiger cache size
setting.

Please note that WiredTiger cache size setting governs only the maximum
cache size of WiredTiger, and not the overall MongoDB memory use. Other
processes such as client connection, aggregation framework, queries, etc.
uses memory on top of WiredTiger cache usage. Therefore, a busy server
could use significantly more memory on top of the WiredTiger cache size
setting.

Having said that, as of MongoDB 3.6.2, there is currently no built-in
method to limit MongoDB memory usage, since MongoDB was designed to utilize
as much memory as possible to give you the best performance.

For diagnosing memory usage, you may be able to use Memory Diagnostics for
the WiredTiger Storage Engine
<https://docs.mongodb.com/manual/faq/diagnostics/#memory-diagnostics-for-the-wiredtiger-storage-engine>
as a starting point.

Best regards
Kevin
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/f0455bd9-4e46-4ea1-8676-18988f46466f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Nick Judson
2018-01-22 15:41:46 UTC
Permalink
Hi Kevin - thanks for replying.

You are correct that I am referring to the --wiredTigerCacheSizeGB (set to
1). I was sent a screenshot of task manager showing the mongo db process
using >3GB of memory, as well as the metrics files (which I cannot open). I
am aware of all of the settings and potential overages for MongoDB/WT, and
it is entirely possible that there is some edge-case workload going on here
which is causing MongoDB to bump up to 3x the wiredTigerCacheSizeGB
setting. This does seem unlikely though as I have never encountered it
before (in many other installations). In the past I would have posted this
on the MongoDB JIRA page and the contents of the metrics file would be
exploded into those nice graphs which show exactly what is occurring. I
suppose I'm asking if it's possible for someone to do the same via this
forum? I could PM the metrics files and then someone internal to MongoDB
could post the resulting graph/prognosis.

Thanks,
Nick.
Post by 'Kevin Adistambha' via mongodb-user
Hi Nick
Could you elaborate on how you determine that MongoDB is using >3 GB? Note
that by “cache size” I’m assuming you meant the WiredTiger cache size
setting.
Please note that WiredTiger cache size setting governs only the maximum
cache size of WiredTiger, and not the overall MongoDB memory use. Other
processes such as client connection, aggregation framework, queries, etc.
uses memory on top of WiredTiger cache usage. Therefore, a busy server
could use significantly more memory on top of the WiredTiger cache size
setting.
Having said that, as of MongoDB 3.6.2, there is currently no built-in
method to limit MongoDB memory usage, since MongoDB was designed to utilize
as much memory as possible to give you the best performance.
For diagnosing memory usage, you may be able to use Memory Diagnostics
for the WiredTiger Storage Engine
<https://docs.mongodb.com/manual/faq/diagnostics/#memory-diagnostics-for-the-wiredtiger-storage-engine>
as a starting point.
Best regards
Kevin
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/f60bc085-2885-4b62-8072-de29f6e23a13%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Anthony Maire
2018-01-22 17:06:12 UTC
Permalink
Hi Kevin and Nick

I'm currently experiencing the same kind of issue on linux with mongo 3.4.1
We use mongoDB for our internal monitoring system and we are currently
doing some experimentation about redesigning the schema to improve our
performance.
So I have 2 servers :

- The development server (mongo 3.6.1) which runs perfectly : it hosts a
dump of the production database (16GB), a copy of this dump using our new
schema (6GB) and a stress test database (300GB). It is started with a WT
cache size of 2GB, and even when doing massive insertions on the stress
test DB (on 50 connections), the resident set size (as reported by the
"top" command) stay only slightly above the cache size, somewhere between
2.3 and 2.5GB. This server runs on CentOS 6, without any
virtualization/containerization

- The production server (mongo 3.4.1) which experiences excessive memory
consumption and crashes : it host the production database. It is running in
an OpenVZ container (on top of CentOS 6, without cap on privvmpages so
there shouldn't be any issue like the one that occured with older versions
of OpenVZ). The whole container has 5GB of RAM available, and WT cache size
is 2GB too (confirmed by executing db.serverStatus().wiredTiger.cache on
the prod environment) . However, we always have a bigger resident set side
for the mongod process, often greater than 4GB. This lead to have the mongo
instance performs really poorly (since swap start to be used) and
eventually the kernel OOM killer kicks in when the container dedicated RAM
is full. This occurs several times per week

For the access patterns : our applications periodically push some new
documents in the DB, but there are very few query/aggregations running,
since they are very few (< 5) users that are getting data from the DB, and
they only do it occasionally, most of the time there is no query at all.

So I would like to know :
- Is there any major memory leak that have been fixed between 3.4 and 3.6
that can explain this ?
- How can I investigate on the cause of this memory consumption ?

Best Regards,
Anthony
Post by Nick Judson
Hi Kevin - thanks for replying.
You are correct that I am referring to the --wiredTigerCacheSizeGB (set to
1). I was sent a screenshot of task manager showing the mongo db process
using >3GB of memory, as well as the metrics files (which I cannot open). I
am aware of all of the settings and potential overages for MongoDB/WT, and
it is entirely possible that there is some edge-case workload going on here
which is causing MongoDB to bump up to 3x the wiredTigerCacheSizeGB
setting. This does seem unlikely though as I have never encountered it
before (in many other installations). In the past I would have posted this
on the MongoDB JIRA page and the contents of the metrics file would be
exploded into those nice graphs which show exactly what is occurring. I
suppose I'm asking if it's possible for someone to do the same via this
forum? I could PM the metrics files and then someone internal to MongoDB
could post the resulting graph/prognosis.
Thanks,
Nick.
Post by 'Kevin Adistambha' via mongodb-user
Hi Nick
Could you elaborate on how you determine that MongoDB is using >3 GB?
Note that by “cache size” I’m assuming you meant the WiredTiger cache size
setting.
Please note that WiredTiger cache size setting governs only the maximum
cache size of WiredTiger, and not the overall MongoDB memory use. Other
processes such as client connection, aggregation framework, queries, etc.
uses memory on top of WiredTiger cache usage. Therefore, a busy server
could use significantly more memory on top of the WiredTiger cache size
setting.
Having said that, as of MongoDB 3.6.2, there is currently no built-in
method to limit MongoDB memory usage, since MongoDB was designed to utilize
as much memory as possible to give you the best performance.
For diagnosing memory usage, you may be able to use Memory Diagnostics
for the WiredTiger Storage Engine
<https://docs.mongodb.com/manual/faq/diagnostics/#memory-diagnostics-for-the-wiredtiger-storage-engine>
as a starting point.
Best regards
Kevin
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/5f4506c0-2d9d-4972-8ab2-259adbae2356%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Kevin Adistambha' via mongodb-user
2018-01-25 02:20:32 UTC
Permalink
Hi Nick

Before digging into diagnostics, could you specify the environment you’re
running? Such as your deployment topology (standalone, replica set, etc.),
your OS version, and your hardware specification?

I was sent a screenshot of task manager showing the mongo db process using
Post by Nick Judson
3GB of memory
Just so we’re on the same page, are you seeing any performance issues due
to this memory usage, or is this a one-time event that you’re curious
about? Also, does the memory usage stabilize at some value after a while?

it is entirely possible that there is some edge-case workload going on here
which is causing MongoDB to bump up to 3x the wiredTigerCacheSizeGB setting

If finding an edge-case workload is the goal, it’s generally more precise
to identify the workload in question by looking at mongod logs. Things to
look for could include:

- Are there any slow queries, or anything that could point to what the
server was doing during the time of high memory usage? Using mloginfo
from the mtools package <https://github.com/rueckstiess/mtools> could
show you the slow queries during this period.
- Note that large spike in memory usage could be caused by incoming
connections into MongoDB. Each connection into MongoDB would take about 1
MB of RAM, so if you have 1000 active connections, mongod would allocate
an extra 1 GB of RAM to handle them.
- How does the memory usage change over time? Do you see this sudden
spike once, or multiple times? Is there any pattern that you can observe of
high memory usage vs. the server’s uptime?

Also since you’re using 3.2.10, could you try to upgrade to the latest in
the 3.2 series (currently 3.2.18) and see if the issue persists?

Best regards
Kevin
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/984bfaa2-d7b7-41fe-b9a3-1bad8249a07f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Nick Judson
2018-01-25 03:33:37 UTC
Permalink
Hi Anthony,

This issue was sent to my by a customer - I do not have access to the
machine itself. This is a standalone mongod (no replset) running on
w2k12r2. No idea on the hardware...could well be a VM. From what I can
gather, the memory increases slowly over the period of one week from 1GB to
3.5GB. I do not know if it stabilizes as the customer kills the process
regularly. I also cannot upgrade the customer since this version of MongoDB
is tied to the version of the product they have installed.

This particular workload has been a challenge for MongoDB in the past (it
produced the bugs for the bug hunt runner up for 3.0, 3.2, and 3.4) and
I've found the best way to accurately determine what is going on is to
provide MongoDB with the metrics files. In the past, symptoms like this
have proved to be highly technical and not something that is easily
diagnosed externally. I had a look through the JIRA list and didn't see
anything which might be causing this (although please correct me if you're
aware of something). I thought that diagnosing issues such as this was
exactly what the metric data was for?

I have manually looked at the log files and there are a few slower queries
(perhaps 5 seconds) as well as some very large queries (large 'in' array)
but I would expect those to require temporary spikes in memory only.

Cheers,
Nick.
Post by 'Kevin Adistambha' via mongodb-user
Hi Nick
Before digging into diagnostics, could you specify the environment you’re
running? Such as your deployment topology (standalone, replica set, etc.),
your OS version, and your hardware specification?
I was sent a screenshot of task manager showing the mongo db process using
Post by Nick Judson
3GB of memory
Just so we’re on the same page, are you seeing any performance issues due
to this memory usage, or is this a one-time event that you’re curious
about? Also, does the memory usage stabilize at some value after a while?
it is entirely possible that there is some edge-case workload going on
here which is causing MongoDB to bump up to 3x the wiredTigerCacheSizeGB
setting
If finding an edge-case workload is the goal, it’s generally more precise
to identify the workload in question by looking at mongod logs. Things to
- Are there any slow queries, or anything that could point to what the
server was doing during the time of high memory usage? Using mloginfo
from the mtools package <https://github.com/rueckstiess/mtools> could
show you the slow queries during this period.
- Note that large spike in memory usage could be caused by incoming
connections into MongoDB. Each connection into MongoDB would take about 1
MB of RAM, so if you have 1000 active connections, mongod would
allocate an extra 1 GB of RAM to handle them.
- How does the memory usage change over time? Do you see this sudden
spike once, or multiple times? Is there any pattern that you can observe of
high memory usage vs. the server’s uptime?
Also since you’re using 3.2.10, could you try to upgrade to the latest in
the 3.2 series (currently 3.2.18) and see if the issue persists?
Best regards
Kevin
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/5b950226-8583-4c5a-a614-f25e9826bd95%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Kevin Adistambha' via mongodb-user
2018-01-25 02:23:09 UTC
Permalink
Hi Anthony

I’m currently experiencing the same kind of issue on linux with mongo 3.4.1

Although the issue looks superficially similar, chances are it’s a
different issue.

Could you please open a new thread with your issue so that we’re not
talking about two different things in this thread?

Best regards
Kevin
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/fa3aa74d-6e24-4e39-bbd5-ddb736e89451%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Anthony Maire
2018-01-29 09:48:41 UTC
Permalink
Hi Kevin

Thank you for your answer.
I'm going to open another thread (and I have some further information to
add about my issue)

Regards,
Anthony
Post by 'Kevin Adistambha' via mongodb-user
Hi Anthony
I’m currently experiencing the same kind of issue on linux with mongo 3.4.1
Although the issue looks superficially similar, chances are it’s a
different issue.
Could you please open a new thread with your issue so that we’re not
talking about two different things in this thread?
Best regards
Kevin
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/9e0f711d-dcd0-4bc8-ac1d-8adf582e0904%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...