We have been playing with the WiredTiger cache settings: eviction tuning
and cache size in order to minimize the impact of the problem. However, we
haven't found a setup that allow us to fully avoid the disk flush stall.
Finally we are considering to scale the disk to one with more IOPS and MB/s.
We are running with MongoDB 3.4.9 (standalone) using WiredTiger Storage
- Window 2016 server in Azure
- 64GB of RAM
- SSD Premium (azure disk type name: P30)
- 5000 IOPS
- 200 MB/s
- 1 TB size
We plan to resize the disk to P40, this is 7500 IOPS, 250 MB/s and 2TB
size, this is the next azure disk available
Bfore performing this resize, we have tried to confirm that we are I/O
Bound. We have been monitoring the disk with PerfMon and this has been the
- IOPS max value is around 1000 IOPS (*DiskTransfersSec.png)*
- MB/s max value is around 12 MB/s (*DiskBytesSec.png)*
Those values are far behind the promised limit.
we have been stressing the disk and we reached the disk promised limit in
IOPS and Throughput.
Reviewing mongo, we found that our max dirty bytes in cache is around 500MB
right before the flush (I have attached the result of
db.serverStatus().wiredTiger right before the flush to disk
ServerStatusWiredTiger.txt). Finally in mongostat we can see that the flush
provokes drop in performance around 10 seconds
See attached image MongoStat.png (note: the used% is not at 80% in the
screenshot because we recently increased to the defult wiredTiger cache and
the memory is slowly growing until the 80%)
We are feeling we are missing some point, with the provided data do you
still think the problem is the Disk and that we are currently I/O bound? If
we are I/O bound why we do not see in PerfMon values around the disk limit?
Finally, do you know somebody that could help us with this? We are thinking
in 1 hour remote meeting that of course will be payed. In this case you
know contact me in private at francesco.rivola @ xepient.com.
Thank you in advanced,
Post by 'Kevin Adistambha' via mongodb-user
âreducing the WiredTiger cache is throttling our workflowâ. Does this
happen as the default 5% eviction_dirty_target is now reached, so the
eviction thread starts to write to disk reducing the amount of work that
need to be done in the checkpoint? Is it right?
Correct. The full explanation of the tunables are described in Cache and
eviction tuning page
under the heading Eviction tuning. By default (in MongoDB 3.4) this value
is set to 5% of the WiredTiger cache size (see
I believe the mechanism at work here by lowering the WiredTiger cache from
31GB to 1GB is that it allows the disk to keep up writing dirty data. 5% of
31GB is ~1.5GB, while 5% of 1GB is ~50MB, a much smaller amount of data to
write. In other words, when your cache size is 31GB, you write faster to
memory but also wait longer for the writes to be flushed to disk, leading
to âstallsâ. When this value is lowered to 1GB, the flushes are smaller but
more regular and faster, thus the âstallsâ are smoothed out over time. The
overall time it took to write the data should be the same, since the
workload appears to be disk-bound.
What could be the factor that limits the disk in the checkpoint scenario?
The 200MiB/s throughput or the 5000 IOPS? Or both.
I think itâs both, since what you described so far sounds like the disk is
struggling to fulfil the work required of it. Larger throughput + IOPS
should generally provide you with a better performance.
You may be able to find a suitable setting for the eviction_dirty_target
parameter that is optimal for your workload and your provisioned hardware.
However this is an advanced tuning that is best reserved when all other
options are exhausted. Please make sure that you have recent backups before
doing advanced maintenance on your deployment, and that you have tested the
new parameters on test deployments before implementing it in production.
You received this message because you are subscribed to the Google Groups "mongodb-user"
For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to email@example.com.
To post to this group, send email to firstname.lastname@example.org.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/479f5880-a1f8-4f25-a0fd-15b4be1516ea%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.