Discussion:
Help : YCSB Tests with MongoDB 3.0.6 throws up Client view of cluster state is {type=Unknown, servers=[{address=:27017 Error
(too old to reply)
prasad2019
2015-09-03 23:55:09 UTC
Permalink
YCSB tar file downloaded from https://github.com/brianfrankcooper/YCSB
/releases/download/0.3.0/ycsb-0.3.0.tar.gz works fine it conencts to
Mongo server from Client machine, though it shows the following warning
message

Cluster created with settings {hosts=[X.X.X.X:27017], mode=SINGLE,
requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms',
maxWaitQueueSize=500}
Updating cluster description to {type=UNKNOWN,
servers=[{address=X.X.X.X:27017, type=UNKNOWN, state=CONNECTING}]


It connects to MongoDB Server from YCSB Client and I am getting a good
performance with WiredTiger storage engine.

But I wanted to see better performance using YCSB code from
https://github.com/10gen-labs/YCSB (Due to Batchsize changed from 1 to
100)

It compiles with no issues but when I execute workloadA it throws up the
error message

"Timed out after 10000 ms while waiting for a server that matches
AnyServerSelector{}. Client view of cluster state is {type=Unknown,
servers=[{address=1:27017, type=Unknown, state=Connecting"


I even HardCoded MongoDB server IP address so that it can pick IP but
inspite of that I get the same error message.

mongodb/src/main/java/com/yahoo/ycsb/db/MongoDbClient.java

I was checking to see if you guys run into similar issues in the past and
how did you overcome. Any pointers will be greatly appreciated.


Thanks
Prasad
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/6aed493d-b684-49e0-ab57-5a6832c3680e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Rob Moore
2015-09-04 01:31:16 UTC
Permalink
Prasad -

I cannot help you with the 10gen-labs repo but the 0.3.0 version of YCSB
supports batched inserts via the mongodb.batchSize parameter.

You can see all of the supported options here:

https://github.com/brianfrankcooper/YCSB/tree/master/mongodb#mongodb-configuration-parameters

HTH,
Rob.
Post by prasad2019
YCSB tar file downloaded from https://github.com/brianfrankcooper/YCSB
/releases/download/0.3.0/ycsb-0.3.0.tar.gz works fine it conencts to
Mongo server from Client machine, though it shows the following warning
message
Cluster created with settings {hosts=[X.X.X.X:27017], mode=SINGLE,
requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms',
maxWaitQueueSize=500}
Updating cluster description to {type=UNKNOWN,
servers=[{address=X.X.X.X:27017, type=UNKNOWN, state=CONNECTING}]
It connects to MongoDB Server from YCSB Client and I am getting a good
performance with WiredTiger storage engine.
But I wanted to see better performance using YCSB code from
https://github.com/10gen-labs/YCSB (Due to Batchsize changed from 1 to
100)
It compiles with no issues but when I execute workloadA it throws up the
error message
"Timed out after 10000 ms while waiting for a server that matches
AnyServerSelector{}. Client view of cluster state is {type=Unknown,
servers=[{address=1:27017, type=Unknown, state=Connecting"
I even HardCoded MongoDB server IP address so that it can pick IP but
inspite of that I get the same error message.
mongodb/src/main/java/com/yahoo/ycsb/db/MongoDbClient.java
I was checking to see if you guys run into similar issues in the past and
how did you overcome. Any pointers will be greatly appreciated.
Thanks
Prasad
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/83ef9f60-8f6b-4a9f-96cd-0eb7bd78695a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
prasad2019
2015-09-04 22:53:28 UTC
Permalink
Hi Rob,

Thanks for the pointer, I did used BatchSize parameter for YCSB load
tests and results went down.


Documents Size Operations/Sec YCSB Threads AverageLatency(us)
95thPercentileLatency(ms) 99thPercentileLatency(ms) Remarks 30 Million
61004 64 1036 0 1 Batchsize=Default 30 Million 50119 64 1236 2 3
Batchsize=100

I also tried setting Read preferred to Primary
"db.getMongo().setReadPref('primaryPreferred')" and replaced the
MongoDB Driver file from 3.0.2 to latest 3.0.3

*I am following this Blog from Aysa and get my baseline right before I
start doing all the YCSB tests.*

https://www.mongodb.com/blog/post/performance-testing-mongodb-30-part-1-throughput-improvements-measured-ycsb

Since this is not clear this Blog has compression set to "Snappy" or None,
I tried changing the compression with the below command run into Mongo
startup issues

numactl --interleave=all mongod --storageEngine wiredTiger
--wiredTigerJournalCompressor none --journal --config /etc/mongod.conf


*Mongod.log* shows this error message



“ log_read: Compressed record with no configured compressor: WT_ERROR:
non-specific WiredTiger error”





Thanks
Prasad
Post by Rob Moore
Prasad -
I cannot help you with the 10gen-labs repo but the 0.3.0 version of YCSB
supports batched inserts via the mongodb.batchSize parameter.
https://github.com/brianfrankcooper/YCSB/tree/master/mongodb#mongodb-configuration-parameters
HTH,
Rob.
Post by prasad2019
YCSB tar file downloaded from https://github.com/brianfrankcooper/YCSB
/releases/download/0.3.0/ycsb-0.3.0.tar.gz works fine it conencts to
Mongo server from Client machine, though it shows the following warning
message
Cluster created with settings {hosts=[X.X.X.X:27017], mode=SINGLE,
requiredClusterType=UNKNOWN, serverSelectionTimeout='30000 ms',
maxWaitQueueSize=500}
Updating cluster description to {type=UNKNOWN,
servers=[{address=X.X.X.X:27017, type=UNKNOWN, state=CONNECTING}]
It connects to MongoDB Server from YCSB Client and I am getting a good
performance with WiredTiger storage engine.
But I wanted to see better performance using YCSB code from
https://github.com/10gen-labs/YCSB (Due to Batchsize changed from 1 to
100)
It compiles with no issues but when I execute workloadA it throws up the
error message
"Timed out after 10000 ms while waiting for a server that matches
AnyServerSelector{}. Client view of cluster state is {type=Unknown,
servers=[{address=1:27017, type=Unknown, state=Connecting"
I even HardCoded MongoDB server IP address so that it can pick IP but
inspite of that I get the same error message.
mongodb/src/main/java/com/yahoo/ycsb/db/MongoDbClient.java
I was checking to see if you guys run into similar issues in the past and
how did you overcome. Any pointers will be greatly appreciated.
Thanks
Prasad
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/425d9972-32c2-4426-ae83-16adf65f9aff%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Rob Moore
2015-09-05 15:34:26 UTC
Permalink
Post by prasad2019
Thanks for the pointer, I did used BatchSize parameter for YCSB load
tests and results went down.
Documents Size Operations/Sec YCSB Threads AverageLatency(us)
95thPercentileLatency(ms) 99thPercentileLatency(ms) Remarks 30 Million
61004 64 1036 0 1 Batchsize=Default 30 Million 50119 64 1236 2 3
Batchsize=100
Can you describe your setup a little bit better. What is the MongoDB
configuration? How many servers? Where is the client in relation to the
servers? What version of MongoDB?

On paper batching in a higher latency environment should improve throughput
but I never saw the data to prove it. Asya's blog does not talk about using
it and the configuration for the feature talks about improving the "load"
of the dataset not the run.

Also, to match Asya's performance you need to make sure you shrink the YCSB
document size from the default of ~1KB to 151 bytes. In the workload file
set/add:

fieldcount=1
fieldlength=100
Post by prasad2019
I also tried setting Read preferred to Primary
"db.getMongo().setReadPref('primaryPreferred')" and replaced the
MongoDB Driver file from 3.0.2 to latest 3.0.3
FYI - You can set the read preference via the MongoDb URI:

http://docs.mongodb.org/manual/reference/connection-string/#read-preference-options
Post by prasad2019
non-specific WiredTiger error”
I suspect that switching compression will require you to wipe of the data
files.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/50127ebc-730c-4f7c-b4e0-0d9f770f5dfa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
prasad2019
2015-09-05 22:58:27 UTC
Permalink
Rob,

I have 4 physical servers each with 28 Cores/256GB RAM and SSD Storage.
MongoDB 3.0.6 with one Primary and two replica set and YCSB and YCSB
client triggered from fourth server.

Another interesting point I observed with BatchSize=1 the datadisk
location around 400MB per second writes and Log around 30 to 40MB close to
12 to 15% CPU Utilization with consistently 60K ops/sec.

Whereas with BatchSize=1 the datadisk location writes around 50 to 450MB
per second and Log around 10 to 20 MB CPU 4 to 15% CPU and 15K to 72K
ops/sec variation. Bottomline BatchSize with 100 performance

variation quite lot.

Well regarding YCSB document size 1KB to 151 bytes that's good point I kind
of overlooked. Will try to change and check out.


Finally can you please point how to switch compression and didn't
understood wipe out data means drop the default database and change the
compression and load it again.



Thanks
Prasad
Post by Rob Moore
Post by prasad2019
Thanks for the pointer, I did used BatchSize parameter for YCSB load
tests and results went down.
Documents Size Operations/Sec YCSB Threads AverageLatency(us)
95thPercentileLatency(ms) 99thPercentileLatency(ms) Remarks 30 Million
61004 64 1036 0 1 Batchsize=Default 30 Million 50119 64 1236 2 3
Batchsize=100
Can you describe your setup a little bit better. What is the MongoDB
configuration? How many servers? Where is the client in relation to the
servers? What version of MongoDB?
On paper batching in a higher latency environment should improve
throughput but I never saw the data to prove it. Asya's blog does not talk
about using it and the configuration for the feature talks about improving
the "load" of the dataset not the run.
Also, to match Asya's performance you need to make sure you shrink the
YCSB document size from the default of ~1KB to 151 bytes. In the workload
fieldcount=1
fieldlength=100
Post by prasad2019
I also tried setting Read preferred to Primary
"db.getMongo().setReadPref('primaryPreferred')" and replaced the
MongoDB Driver file from 3.0.2 to latest 3.0.3
http://docs.mongodb.org/manual/reference/connection-string/#read-preference-options
Post by prasad2019
non-specific WiredTiger error”
I suspect that switching compression will require you to wipe of the data
files.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/85e254c9-0fd4-49a5-acd0-9c774ecb00ed%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-09-09 22:21:44 UTC
Permalink
Hi Prasad:

As Rob pointed out - you can do batch inserts on either branch - in fact,
the 10gen-labs branch was created because YCSB seemed to have gone very
dormant as a project, but recently they became very active again, merging
pull requests, removing old crufty stuff that hasn't worked, and in general
revitalizing the project. So since Rob has submitted a number of
improvements similar to what we created in the labs version, it's probably
better to use the master for regular YCSB testing. Our long term plan is
to use our own branch to add some functionality that is missing in YCSB -
ability to use data that's more realistic (not random byte streams), using
secondary indexes, and queries by secondary indexes, as well as more
flexibility in types of reads and writes.

When we originally added batch inserts it was to reduce the time it takes
to wipe and repopulate a cluster to then run some sort of mixed workload.
Depending on the configuration of the cluster, batching could gain anywhere
from 2x to 6x in my experience. But how much you gain heavily depends on
where the bottleneck is for the single document insert case. If you're
already writing as much as the server can write, then that's your limit -
but if the server is underloaded because the back-and-forth on the network
is slowing things down, batching can help hugely.

Hope that helps,
Asya
Post by prasad2019
Rob,
I have 4 physical servers each with 28 Cores/256GB RAM and SSD Storage.
MongoDB 3.0.6 with one Primary and two replica set and YCSB and YCSB
client triggered from fourth server.
Another interesting point I observed with BatchSize=1 the datadisk
location around 400MB per second writes and Log around 30 to 40MB close to
12 to 15% CPU Utilization with consistently 60K ops/sec.
Whereas with BatchSize=1 the datadisk location writes around 50 to 450MB
per second and Log around 10 to 20 MB CPU 4 to 15% CPU and 15K to 72K
ops/sec variation. Bottomline BatchSize with 100 performance
variation quite lot.
Well regarding YCSB document size 1KB to 151 bytes that's good point I
kind of overlooked. Will try to change and check out.
Finally can you please point how to switch compression and didn't
understood wipe out data means drop the default database and change the
compression and load it again.
Thanks
Prasad
Post by Rob Moore
Post by prasad2019
Thanks for the pointer, I did used BatchSize parameter for YCSB load
tests and results went down.
Documents Size Operations/Sec YCSB Threads AverageLatency(us)
95thPercentileLatency(ms) 99thPercentileLatency(ms) Remarks 30 Million
61004 64 1036 0 1 Batchsize=Default 30 Million 50119 64 1236 2 3
Batchsize=100
Can you describe your setup a little bit better. What is the MongoDB
configuration? How many servers? Where is the client in relation to the
servers? What version of MongoDB?
On paper batching in a higher latency environment should improve
throughput but I never saw the data to prove it. Asya's blog does not talk
about using it and the configuration for the feature talks about improving
the "load" of the dataset not the run.
Also, to match Asya's performance you need to make sure you shrink the
YCSB document size from the default of ~1KB to 151 bytes. In the workload
fieldcount=1
fieldlength=100
Post by prasad2019
I also tried setting Read preferred to Primary
"db.getMongo().setReadPref('primaryPreferred')" and replaced the
MongoDB Driver file from 3.0.2 to latest 3.0.3
http://docs.mongodb.org/manual/reference/connection-string/#read-preference-options
Post by prasad2019
non-specific WiredTiger error”
I suspect that switching compression will require you to wipe of the data
files.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/85e254c9-0fd4-49a5-acd0-9c774ecb00ed%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/85e254c9-0fd4-49a5-acd0-9c774ecb00ed%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJDaYs%2BTo5T3YQKiZ%3D%3DkSjBQxN9Rv4OPBTP7U6Z34h41Ug%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
prasad2019
2015-09-10 21:00:13 UTC
Permalink
Hi Aysa,

Thanks I realized that too after couple of tests system CPU was just 15%
so plenty of room, it was the IO which was getting maxed out (450MB /Sec
Writes) during Data load, To confirm this I am planning to add additional
disks to realize the benefit of Batchsize from default 1 to see if this
helps in increasing helps for data loading.

Well now I am encountering checksum error on one of the datafile, Beacuse
of this primary MongoDB instance crashes and two replicas are not catching
up with the primary. Even after primary restart hoping with Journaling on,
it will correct no need to run "Repair Database" command.

"2015-09-10T13:03:45.678-0600 E STORAGE [conn47] WiredTiger (0)
[1441911825:678595][11777:0x7f7a6aa06700],
file:collection-12--8249752617800779708.wt, cursor.next: read checksum
error for 32768B block at offset 24421470208: calculated block checksum of
179228071 doesn't match expected checksum of 2102742815"
"2015-09-10T13:03:45.678-0600 E STORAGE [conn47] WiredTiger (0)
[1441911825:678683][11777:0x7f7a6aa06700],
file:collection-12--8249752617800779708.wt, cursor.next:
collection-12--8249752617800779708.wt: encountered an illegal file format
or internal value"
"2015-09-10T13:03:45.678-0600 E STORAGE [conn47] WiredTiger (-31804)
[1441911825:678705][11777:0x7f7a6aa06700],
file:collection-12--8249752617800779708.wt, cursor.next: the process must
exit and restart: WT_PANIC: WiredTiger library panic"

Can you please help me how do I recover this block checksum error, is there
block repair command that I can use?


Thanks
Prasad
Post by Asya Kamsky
As Rob pointed out - you can do batch inserts on either branch - in fact,
the 10gen-labs branch was created because YCSB seemed to have gone very
dormant as a project, but recently they became very active again, merging
pull requests, removing old crufty stuff that hasn't worked, and in general
revitalizing the project. So since Rob has submitted a number of
improvements similar to what we created in the labs version, it's probably
better to use the master for regular YCSB testing. Our long term plan is
to use our own branch to add some functionality that is missing in YCSB -
ability to use data that's more realistic (not random byte streams), using
secondary indexes, and queries by secondary indexes, as well as more
flexibility in types of reads and writes.
When we originally added batch inserts it was to reduce the time it takes
to wipe and repopulate a cluster to then run some sort of mixed workload.
Depending on the configuration of the cluster, batching could gain anywhere
from 2x to 6x in my experience. But how much you gain heavily depends on
where the bottleneck is for the single document insert case. If you're
already writing as much as the server can write, then that's your limit -
but if the server is underloaded because the back-and-forth on the network
is slowing things down, batching can help hugely.
Hope that helps,
Asya
Post by prasad2019
Rob,
I have 4 physical servers each with 28 Cores/256GB RAM and SSD Storage.
MongoDB 3.0.6 with one Primary and two replica set and YCSB and YCSB
client triggered from fourth server.
Another interesting point I observed with BatchSize=1 the datadisk
location around 400MB per second writes and Log around 30 to 40MB close to
12 to 15% CPU Utilization with consistently 60K ops/sec.
Whereas with BatchSize=1 the datadisk location writes around 50 to 450MB
per second and Log around 10 to 20 MB CPU 4 to 15% CPU and 15K to 72K
ops/sec variation. Bottomline BatchSize with 100 performance
variation quite lot.
Well regarding YCSB document size 1KB to 151 bytes that's good point I
kind of overlooked. Will try to change and check out.
Finally can you please point how to switch compression and didn't
understood wipe out data means drop the default database and change the
compression and load it again.
Thanks
Prasad
Post by Rob Moore
Post by prasad2019
Thanks for the pointer, I did used BatchSize parameter for YCSB load
tests and results went down.
Documents Size Operations/Sec YCSB Threads AverageLatency(us)
95thPercentileLatency(ms) 99thPercentileLatency(ms) Remarks 30
Million 61004 64 1036 0 1 Batchsize=Default 30 Million 50119 64 1236
2 3 Batchsize=100
Can you describe your setup a little bit better. What is the MongoDB
configuration? How many servers? Where is the client in relation to the
servers? What version of MongoDB?
On paper batching in a higher latency environment should improve
throughput but I never saw the data to prove it. Asya's blog does not talk
about using it and the configuration for the feature talks about improving
the "load" of the dataset not the run.
Also, to match Asya's performance you need to make sure you shrink the
YCSB document size from the default of ~1KB to 151 bytes. In the workload
fieldcount=1
fieldlength=100
Post by prasad2019
I also tried setting Read preferred to Primary
"db.getMongo().setReadPref('primaryPreferred')" and replaced the
MongoDB Driver file from 3.0.2 to latest 3.0.3
http://docs.mongodb.org/manual/reference/connection-string/#read-preference-options
Post by prasad2019
non-specific WiredTiger error”
I suspect that switching compression will require you to wipe of the
data files.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/85e254c9-0fd4-49a5-acd0-9c774ecb00ed%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/85e254c9-0fd4-49a5-acd0-9c774ecb00ed%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/51aa6800-068d-4292-9b73-eb5cf17bf326%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-09-11 07:41:17 UTC
Permalink
This strongly suggests an issue with underlying storage layer - if the file
on disk gets corrupt there isn't any way journaling can help.

Since this is YCSB data you can just wipe out the directory and repopulate
- that would be my suggestion but I would verify the file system first.

Asya
Post by prasad2019
Hi Aysa,
Thanks I realized that too after couple of tests system CPU was just
15% so plenty of room, it was the IO which was getting maxed out (450MB
/Sec Writes) during Data load, To confirm this I am planning to add
additional disks to realize the benefit of Batchsize from default 1 to see
if this helps in increasing helps for data loading.
Well now I am encountering checksum error on one of the datafile, Beacuse
of this primary MongoDB instance crashes and two replicas are not catching
up with the primary. Even after primary restart hoping with Journaling on,
it will correct no need to run "Repair Database" command.
"2015-09-10T13:03:45.678-0600 E STORAGE [conn47] WiredTiger (0)
[1441911825:678595][11777:0x7f7a6aa06700],
file:collection-12--8249752617800779708.wt, cursor.next: read checksum
error for 32768B block at offset 24421470208: calculated block checksum of
179228071 doesn't match expected checksum of 2102742815"
"2015-09-10T13:03:45.678-0600 E STORAGE [conn47] WiredTiger (0)
[1441911825:678683][11777:0x7f7a6aa06700],
collection-12--8249752617800779708.wt: encountered an illegal file format
or internal value"
"2015-09-10T13:03:45.678-0600 E STORAGE [conn47] WiredTiger (-31804)
[1441911825:678705][11777:0x7f7a6aa06700],
file:collection-12--8249752617800779708.wt, cursor.next: the process must
exit and restart: WT_PANIC: WiredTiger library panic"
Can you please help me how do I recover this block checksum error, is
there block repair command that I can use?
Thanks
Prasad
Post by Asya Kamsky
As Rob pointed out - you can do batch inserts on either branch - in fact,
the 10gen-labs branch was created because YCSB seemed to have gone very
dormant as a project, but recently they became very active again, merging
pull requests, removing old crufty stuff that hasn't worked, and in general
revitalizing the project. So since Rob has submitted a number of
improvements similar to what we created in the labs version, it's probably
better to use the master for regular YCSB testing. Our long term plan is
to use our own branch to add some functionality that is missing in YCSB -
ability to use data that's more realistic (not random byte streams), using
secondary indexes, and queries by secondary indexes, as well as more
flexibility in types of reads and writes.
When we originally added batch inserts it was to reduce the time it takes
to wipe and repopulate a cluster to then run some sort of mixed workload.
Depending on the configuration of the cluster, batching could gain anywhere
from 2x to 6x in my experience. But how much you gain heavily depends on
where the bottleneck is for the single document insert case. If you're
already writing as much as the server can write, then that's your limit -
but if the server is underloaded because the back-and-forth on the network
is slowing things down, batching can help hugely.
Hope that helps,
Asya
Post by prasad2019
Rob,
I have 4 physical servers each with 28 Cores/256GB RAM and SSD Storage.
MongoDB 3.0.6 with one Primary and two replica set and YCSB and YCSB
client triggered from fourth server.
Another interesting point I observed with BatchSize=1 the datadisk
location around 400MB per second writes and Log around 30 to 40MB close to
12 to 15% CPU Utilization with consistently 60K ops/sec.
Whereas with BatchSize=1 the datadisk location writes around 50 to
450MB per second and Log around 10 to 20 MB CPU 4 to 15% CPU and 15K to 72K
ops/sec variation. Bottomline BatchSize with 100 performance
variation quite lot.
Well regarding YCSB document size 1KB to 151 bytes that's good point I
kind of overlooked. Will try to change and check out.
Finally can you please point how to switch compression and didn't
understood wipe out data means drop the default database and change the
compression and load it again.
Thanks
Prasad
Post by Rob Moore
Post by prasad2019
Thanks for the pointer, I did used BatchSize parameter for YCSB load
tests and results went down.
Documents Size Operations/Sec YCSB Threads AverageLatency(us)
95thPercentileLatency(ms) 99thPercentileLatency(ms) Remarks 30
Million 61004 64 1036 0 1 Batchsize=Default 30 Million 50119 64 1236
2 3 Batchsize=100
Can you describe your setup a little bit better. What is the MongoDB
configuration? How many servers? Where is the client in relation to the
servers? What version of MongoDB?
On paper batching in a higher latency environment should improve
throughput but I never saw the data to prove it. Asya's blog does not talk
about using it and the configuration for the feature talks about improving
the "load" of the dataset not the run.
Also, to match Asya's performance you need to make sure you shrink the
YCSB document size from the default of ~1KB to 151 bytes. In the workload
fieldcount=1
fieldlength=100
Post by prasad2019
I also tried setting Read preferred to Primary
"db.getMongo().setReadPref('primaryPreferred')" and replaced the
MongoDB Driver file from 3.0.2 to latest 3.0.3
http://docs.mongodb.org/manual/reference/connection-string/#read-preference-options
Post by prasad2019
non-specific WiredTiger error”
I suspect that switching compression will require you to wipe of the
data files.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/85e254c9-0fd4-49a5-acd0-9c774ecb00ed%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/85e254c9-0fd4-49a5-acd0-9c774ecb00ed%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/51aa6800-068d-4292-9b73-eb5cf17bf326%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/51aa6800-068d-4292-9b73-eb5cf17bf326%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJBa7Oni1U%2B9wcrGMWvWUCyOx_Syn8eev-zN09pcoUp%3DMw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
prasad2019
2015-09-14 23:41:16 UTC
Permalink
Hi Aysa,

Thanks it was issue with XFS file system on RHEL 7.1, yes now I have
repopulated with increased dataset from 30 Million to 300 Million
documents. Data loading went pretty fine but realized later the replica
sets were not able to catch up to the data loading

rate ( Writing 180 MB/s of data (this time it was Hard Disk with RAID 10 on
it and With SSD it was 420 MB/s) with the default oplogSize=1024 MB ). To
sync up I brought down the mongo instances and on replicaset servers
dropped

the datapath directories. OplogSize=5GB changed on both primary and
replicas with this change replicaset caught up with Primary (Including
Index build) for a dataset size of 358GB size in 1 Hour 51 Minutes
(Against 3 Hours with OPLOGSIZE=1024 MB even with

SSD).

So having a larger Oplogsize made lot of difference, starting liking these
internal operations that explains why MongoDB engine ranking is going up
rapidly (http://db-engines.com/en/ranking)


BTW I wanted to validate these YCSB test results with MongoDB how do I do
it, any pointers.



Thanks
Prasad




Thanks
Prasad



Thanks
Prasad
Post by Asya Kamsky
This strongly suggests an issue with underlying storage layer - if the
file on disk gets corrupt there isn't any way journaling can help.
Since this is YCSB data you can just wipe out the directory and repopulate
- that would be my suggestion but I would verify the file system first.
Asya
Post by prasad2019
Hi Aysa,
Thanks I realized that too after couple of tests system CPU was just
15% so plenty of room, it was the IO which was getting maxed out (450MB
/Sec Writes) during Data load, To confirm this I am planning to add
additional disks to realize the benefit of Batchsize from default 1 to see
if this helps in increasing helps for data loading.
Well now I am encountering checksum error on one of the datafile, Beacuse
of this primary MongoDB instance crashes and two replicas are not catching
up with the primary. Even after primary restart hoping with Journaling on,
it will correct no need to run "Repair Database" command.
"2015-09-10T13:03:45.678-0600 E STORAGE [conn47] WiredTiger (0)
[1441911825:678595][11777:0x7f7a6aa06700],
file:collection-12--8249752617800779708.wt, cursor.next: read checksum
error for 32768B block at offset 24421470208: calculated block checksum of
179228071 doesn't match expected checksum of 2102742815"
"2015-09-10T13:03:45.678-0600 E STORAGE [conn47] WiredTiger (0)
[1441911825:678683][11777:0x7f7a6aa06700],
collection-12--8249752617800779708.wt: encountered an illegal file format
or internal value"
"2015-09-10T13:03:45.678-0600 E STORAGE [conn47] WiredTiger (-31804)
[1441911825:678705][11777:0x7f7a6aa06700],
file:collection-12--8249752617800779708.wt, cursor.next: the process must
exit and restart: WT_PANIC: WiredTiger library panic"
Can you please help me how do I recover this block checksum error, is
there block repair command that I can use?
Thanks
Prasad
Post by Asya Kamsky
As Rob pointed out - you can do batch inserts on either branch - in
fact, the 10gen-labs branch was created because YCSB seemed to have gone
very dormant as a project, but recently they became very active again,
merging pull requests, removing old crufty stuff that hasn't worked, and in
general revitalizing the project. So since Rob has submitted a number of
improvements similar to what we created in the labs version, it's probably
better to use the master for regular YCSB testing. Our long term plan is
to use our own branch to add some functionality that is missing in YCSB -
ability to use data that's more realistic (not random byte streams), using
secondary indexes, and queries by secondary indexes, as well as more
flexibility in types of reads and writes.
When we originally added batch inserts it was to reduce the time it
takes to wipe and repopulate a cluster to then run some sort of mixed
workload. Depending on the configuration of the cluster, batching could
gain anywhere from 2x to 6x in my experience. But how much you gain
heavily depends on where the bottleneck is for the single document insert
case. If you're already writing as much as the server can write, then
that's your limit - but if the server is underloaded because the
back-and-forth on the network is slowing things down, batching can help
hugely.
Hope that helps,
Asya
Post by prasad2019
Rob,
I have 4 physical servers each with 28 Cores/256GB RAM and SSD
Storage. MongoDB 3.0.6 with one Primary and two replica set and YCSB and
YCSB client triggered from fourth server.
Another interesting point I observed with BatchSize=1 the datadisk
location around 400MB per second writes and Log around 30 to 40MB close to
12 to 15% CPU Utilization with consistently 60K ops/sec.
Whereas with BatchSize=1 the datadisk location writes around 50 to
450MB per second and Log around 10 to 20 MB CPU 4 to 15% CPU and 15K to 72K
ops/sec variation. Bottomline BatchSize with 100 performance
variation quite lot.
Well regarding YCSB document size 1KB to 151 bytes that's good point I
kind of overlooked. Will try to change and check out.
Finally can you please point how to switch compression and didn't
understood wipe out data means drop the default database and change the
compression and load it again.
Thanks
Prasad
Post by Rob Moore
Post by prasad2019
Thanks for the pointer, I did used BatchSize parameter for YCSB
load tests and results went down.
Documents Size Operations/Sec YCSB Threads AverageLatency(us)
95thPercentileLatency(ms) 99thPercentileLatency(ms) Remarks 30
Million 61004 64 1036 0 1 Batchsize=Default 30 Million 50119 64
1236 2 3 Batchsize=100
Can you describe your setup a little bit better. What is the MongoDB
configuration? How many servers? Where is the client in relation to the
servers? What version of MongoDB?
On paper batching in a higher latency environment should improve
throughput but I never saw the data to prove it. Asya's blog does not talk
about using it and the configuration for the feature talks about improving
the "load" of the dataset not the run.
Also, to match Asya's performance you need to make sure you shrink the
YCSB document size from the default of ~1KB to 151 bytes. In the workload
fieldcount=1
fieldlength=100
Post by prasad2019
I also tried setting Read preferred to Primary
"db.getMongo().setReadPref('primaryPreferred')" and replaced the
MongoDB Driver file from 3.0.2 to latest 3.0.3
http://docs.mongodb.org/manual/reference/connection-string/#read-preference-options
Post by prasad2019
WT_ERROR: non-specific WiredTiger error”
I suspect that switching compression will require you to wipe of the
data files.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/85e254c9-0fd4-49a5-acd0-9c774ecb00ed%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/85e254c9-0fd4-49a5-acd0-9c774ecb00ed%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/51aa6800-068d-4292-9b73-eb5cf17bf326%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/51aa6800-068d-4292-9b73-eb5cf17bf326%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/d050bcc5-8942-4e92-b5e9-82c95f266246%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-09-15 16:23:35 UTC
Permalink
Post by prasad2019
BTW I wanted to validate these YCSB test results with MongoDB how do I do
it, any pointers.

One of the problems with YCSB is the data is random and arbitrary - so by
validate if you mean checking which/how many documents are
updated/correctness of updates - it's not very easy to do that.

This is partly why I recommend creating your own benchmark similar to your
actual application requirements - that makes it easier to validate the
results...

If you meant something else by validate, please clarify with more details.

Asya
Post by prasad2019
Hi Aysa,
Thanks it was issue with XFS file system on RHEL 7.1, yes now I have
repopulated with increased dataset from 30 Million to 300 Million
documents. Data loading went pretty fine but realized later the replica
sets were not able to catch up to the data loading
rate ( Writing 180 MB/s of data (this time it was Hard Disk with RAID 10
on it and With SSD it was 420 MB/s) with the default oplogSize=1024 MB ).
To sync up I brought down the mongo instances and on replicaset servers
dropped
the datapath directories. OplogSize=5GB changed on both primary and
replicas with this change replicaset caught up with Primary (Including
Index build) for a dataset size of 358GB size in 1 Hour 51 Minutes
(Against 3 Hours with OPLOGSIZE=1024 MB even with
SSD).
So having a larger Oplogsize made lot of difference, starting liking
these internal operations that explains why MongoDB engine ranking is going
up rapidly (http://db-engines.com/en/ranking)
BTW I wanted to validate these YCSB test results with MongoDB how do I do
it, any pointers.
Thanks
Prasad
Thanks
Prasad
Thanks
Prasad
Post by Asya Kamsky
This strongly suggests an issue with underlying storage layer - if the
file on disk gets corrupt there isn't any way journaling can help.
Since this is YCSB data you can just wipe out the directory and
repopulate - that would be my suggestion but I would verify the file system
first.
Asya
Post by prasad2019
Hi Aysa,
Thanks I realized that too after couple of tests system CPU was just
15% so plenty of room, it was the IO which was getting maxed out (450MB
/Sec Writes) during Data load, To confirm this I am planning to add
additional disks to realize the benefit of Batchsize from default 1 to see
if this helps in increasing helps for data loading.
Well now I am encountering checksum error on one of the datafile,
Beacuse of this primary MongoDB instance crashes and two replicas are not
catching up with the primary. Even after primary restart hoping with
Journaling on, it will correct no need to run "Repair Database" command.
"2015-09-10T13:03:45.678-0600 E STORAGE [conn47] WiredTiger (0)
[1441911825:678595][11777:0x7f7a6aa06700],
file:collection-12--8249752617800779708.wt, cursor.next: read checksum
error for 32768B block at offset 24421470208: calculated block checksum of
179228071 doesn't match expected checksum of 2102742815"
"2015-09-10T13:03:45.678-0600 E STORAGE [conn47] WiredTiger (0)
[1441911825:678683][11777:0x7f7a6aa06700],
collection-12--8249752617800779708.wt: encountered an illegal file format
or internal value"
"2015-09-10T13:03:45.678-0600 E STORAGE [conn47] WiredTiger (-31804)
[1441911825:678705][11777:0x7f7a6aa06700],
file:collection-12--8249752617800779708.wt, cursor.next: the process must
exit and restart: WT_PANIC: WiredTiger library panic"
Can you please help me how do I recover this block checksum error, is
there block repair command that I can use?
Thanks
Prasad
Post by Asya Kamsky
As Rob pointed out - you can do batch inserts on either branch - in
fact, the 10gen-labs branch was created because YCSB seemed to have gone
very dormant as a project, but recently they became very active again,
merging pull requests, removing old crufty stuff that hasn't worked, and in
general revitalizing the project. So since Rob has submitted a number of
improvements similar to what we created in the labs version, it's probably
better to use the master for regular YCSB testing. Our long term plan is
to use our own branch to add some functionality that is missing in YCSB -
ability to use data that's more realistic (not random byte streams), using
secondary indexes, and queries by secondary indexes, as well as more
flexibility in types of reads and writes.
When we originally added batch inserts it was to reduce the time it
takes to wipe and repopulate a cluster to then run some sort of mixed
workload. Depending on the configuration of the cluster, batching could
gain anywhere from 2x to 6x in my experience. But how much you gain
heavily depends on where the bottleneck is for the single document insert
case. If you're already writing as much as the server can write, then
that's your limit - but if the server is underloaded because the
back-and-forth on the network is slowing things down, batching can help
hugely.
Hope that helps,
Asya
Post by prasad2019
Rob,
I have 4 physical servers each with 28 Cores/256GB RAM and SSD
Storage. MongoDB 3.0.6 with one Primary and two replica set and YCSB and
YCSB client triggered from fourth server.
Another interesting point I observed with BatchSize=1 the datadisk
location around 400MB per second writes and Log around 30 to 40MB close to
12 to 15% CPU Utilization with consistently 60K ops/sec.
Whereas with BatchSize=1 the datadisk location writes around 50 to
450MB per second and Log around 10 to 20 MB CPU 4 to 15% CPU and 15K to 72K
ops/sec variation. Bottomline BatchSize with 100 performance
variation quite lot.
Well regarding YCSB document size 1KB to 151 bytes that's good point I
kind of overlooked. Will try to change and check out.
Finally can you please point how to switch compression and didn't
understood wipe out data means drop the default database and change the
compression and load it again.
Thanks
Prasad
Post by Rob Moore
Post by prasad2019
Thanks for the pointer, I did used BatchSize parameter for YCSB
load tests and results went down.
Documents Size Operations/Sec YCSB Threads AverageLatency(us)
95thPercentileLatency(ms) 99thPercentileLatency(ms) Remarks 30
Million 61004 64 1036 0 1 Batchsize=Default 30 Million 50119 64
1236 2 3 Batchsize=100
Can you describe your setup a little bit better. What is the MongoDB
configuration? How many servers? Where is the client in relation to the
servers? What version of MongoDB?
On paper batching in a higher latency environment should improve
throughput but I never saw the data to prove it. Asya's blog does not talk
about using it and the configuration for the feature talks about improving
the "load" of the dataset not the run.
Also, to match Asya's performance you need to make sure you shrink
the YCSB document size from the default of ~1KB to 151 bytes. In the
fieldcount=1
fieldlength=100
Post by prasad2019
I also tried setting Read preferred to Primary
"db.getMongo().setReadPref('primaryPreferred')" and replaced the
MongoDB Driver file from 3.0.2 to latest 3.0.3
http://docs.mongodb.org/manual/reference/connection-string/#read-preference-options
Post by prasad2019
WT_ERROR: non-specific WiredTiger error”
I suspect that switching compression will require you to wipe of the
data files.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/85e254c9-0fd4-49a5-acd0-9c774ecb00ed%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/85e254c9-0fd4-49a5-acd0-9c774ecb00ed%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/51aa6800-068d-4292-9b73-eb5cf17bf326%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/51aa6800-068d-4292-9b73-eb5cf17bf326%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/d050bcc5-8942-4e92-b5e9-82c95f266246%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/d050bcc5-8942-4e92-b5e9-82c95f266246%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJDfM90GjsJE1fXW472Y3KwuQoa8BUqO-1RDbQki45zy0A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
prasad2019
2015-09-16 21:05:20 UTC
Permalink
Asya,

I meant folks from MongoDB looking at YCSB test results for validation.
This helps to make sure we have accurate YSCB test results with best
practices of MongoDB implemented.

Let me know how to proceed on this.



Thanks
Prasad
Post by prasad2019
BTW I wanted to validate these YCSB test results with MongoDB how do I
do it, any pointers.
One of the problems with YCSB is the data is random and arbitrary - so by
validate if you mean checking which/how many documents are
updated/correctness of updates - it's not very easy to do that.
This is partly why I recommend creating your own benchmark similar to your
actual application requirements - that makes it easier to validate the
results...
If you meant something else by validate, please clarify with more details.
Asya
Post by prasad2019
Hi Aysa,
Thanks it was issue with XFS file system on RHEL 7.1, yes now I have
repopulated with increased dataset from 30 Million to 300 Million
documents. Data loading went pretty fine but realized later the replica
sets were not able to catch up to the data loading
rate ( Writing 180 MB/s of data (this time it was Hard Disk with RAID 10
on it and With SSD it was 420 MB/s) with the default oplogSize=1024 MB ).
To sync up I brought down the mongo instances and on replicaset servers
dropped
the datapath directories. OplogSize=5GB changed on both primary and
replicas with this change replicaset caught up with Primary (Including
Index build) for a dataset size of 358GB size in 1 Hour 51 Minutes
(Against 3 Hours with OPLOGSIZE=1024 MB even with
SSD).
So having a larger Oplogsize made lot of difference, starting liking
these internal operations that explains why MongoDB engine ranking is going
up rapidly (http://db-engines.com/en/ranking)
BTW I wanted to validate these YCSB test results with MongoDB how do I do
it, any pointers.
Thanks
Prasad
Thanks
Prasad
Thanks
Prasad
Post by Asya Kamsky
This strongly suggests an issue with underlying storage layer - if the
file on disk gets corrupt there isn't any way journaling can help.
Since this is YCSB data you can just wipe out the directory and
repopulate - that would be my suggestion but I would verify the file system
first.
Asya
Post by prasad2019
Hi Aysa,
Thanks I realized that too after couple of tests system CPU was just
15% so plenty of room, it was the IO which was getting maxed out (450MB
/Sec Writes) during Data load, To confirm this I am planning to add
additional disks to realize the benefit of Batchsize from default 1 to see
if this helps in increasing helps for data loading.
Well now I am encountering checksum error on one of the datafile,
Beacuse of this primary MongoDB instance crashes and two replicas are not
catching up with the primary. Even after primary restart hoping with
Journaling on, it will correct no need to run "Repair Database" command.
"2015-09-10T13:03:45.678-0600 E STORAGE [conn47] WiredTiger (0)
[1441911825:678595][11777:0x7f7a6aa06700],
file:collection-12--8249752617800779708.wt, cursor.next: read checksum
error for 32768B block at offset 24421470208: calculated block checksum of
179228071 doesn't match expected checksum of 2102742815"
"2015-09-10T13:03:45.678-0600 E STORAGE [conn47] WiredTiger (0)
[1441911825:678683][11777:0x7f7a6aa06700],
collection-12--8249752617800779708.wt: encountered an illegal file format
or internal value"
"2015-09-10T13:03:45.678-0600 E STORAGE [conn47] WiredTiger (-31804)
[1441911825:678705][11777:0x7f7a6aa06700],
file:collection-12--8249752617800779708.wt, cursor.next: the process must
exit and restart: WT_PANIC: WiredTiger library panic"
Can you please help me how do I recover this block checksum error, is
there block repair command that I can use?
Thanks
Prasad
Post by Asya Kamsky
As Rob pointed out - you can do batch inserts on either branch - in
fact, the 10gen-labs branch was created because YCSB seemed to have gone
very dormant as a project, but recently they became very active again,
merging pull requests, removing old crufty stuff that hasn't worked, and in
general revitalizing the project. So since Rob has submitted a number of
improvements similar to what we created in the labs version, it's probably
better to use the master for regular YCSB testing. Our long term plan is
to use our own branch to add some functionality that is missing in YCSB -
ability to use data that's more realistic (not random byte streams), using
secondary indexes, and queries by secondary indexes, as well as more
flexibility in types of reads and writes.
When we originally added batch inserts it was to reduce the time it
takes to wipe and repopulate a cluster to then run some sort of mixed
workload. Depending on the configuration of the cluster, batching could
gain anywhere from 2x to 6x in my experience. But how much you gain
heavily depends on where the bottleneck is for the single document insert
case. If you're already writing as much as the server can write, then
that's your limit - but if the server is underloaded because the
back-and-forth on the network is slowing things down, batching can help
hugely.
Hope that helps,
Asya
Post by prasad2019
Rob,
I have 4 physical servers each with 28 Cores/256GB RAM and SSD
Storage. MongoDB 3.0.6 with one Primary and two replica set and YCSB and
YCSB client triggered from fourth server.
Another interesting point I observed with BatchSize=1 the datadisk
location around 400MB per second writes and Log around 30 to 40MB close to
12 to 15% CPU Utilization with consistently 60K ops/sec.
Whereas with BatchSize=1 the datadisk location writes around 50 to
450MB per second and Log around 10 to 20 MB CPU 4 to 15% CPU and 15K to 72K
ops/sec variation. Bottomline BatchSize with 100 performance
variation quite lot.
Well regarding YCSB document size 1KB to 151 bytes that's good point
I kind of overlooked. Will try to change and check out.
Finally can you please point how to switch compression and didn't
understood wipe out data means drop the default database and change the
compression and load it again.
Thanks
Prasad
Post by Rob Moore
Post by prasad2019
Thanks for the pointer, I did used BatchSize parameter for YCSB
load tests and results went down.
Documents Size Operations/Sec YCSB Threads AverageLatency(us)
95thPercentileLatency(ms) 99thPercentileLatency(ms) Remarks 30
Million 61004 64 1036 0 1 Batchsize=Default 30 Million 50119 64
1236 2 3 Batchsize=100
Can you describe your setup a little bit better. What is the MongoDB
configuration? How many servers? Where is the client in relation to the
servers? What version of MongoDB?
On paper batching in a higher latency environment should improve
throughput but I never saw the data to prove it. Asya's blog does not talk
about using it and the configuration for the feature talks about improving
the "load" of the dataset not the run.
Also, to match Asya's performance you need to make sure you shrink
the YCSB document size from the default of ~1KB to 151 bytes. In the
fieldcount=1
fieldlength=100
Post by prasad2019
I also tried setting Read preferred to Primary
"db.getMongo().setReadPref('primaryPreferred')" and replaced the
MongoDB Driver file from 3.0.2 to latest 3.0.3
http://docs.mongodb.org/manual/reference/connection-string/#read-preference-options
Post by prasad2019
WT_ERROR: non-specific WiredTiger error”
I suspect that switching compression will require you to wipe of the
data files.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/85e254c9-0fd4-49a5-acd0-9c774ecb00ed%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/85e254c9-0fd4-49a5-acd0-9c774ecb00ed%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/51aa6800-068d-4292-9b73-eb5cf17bf326%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/51aa6800-068d-4292-9b73-eb5cf17bf326%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/d050bcc5-8942-4e92-b5e9-82c95f266246%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/d050bcc5-8942-4e92-b5e9-82c95f266246%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/f3140045-5230-4002-8436-e56ebcc4ec2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...