Discussion:
Tag Aware Sharding not distributing the data to appropriate shards.
(too old to reply)
Naveen
2015-11-26 00:33:52 UTC
Permalink
Trying to a multi regional sharded replicaset cluster with Mongodb 3.0.3.

I've a shard key called "sourceKey" which is of type "String".
Created 2 shards with "sourceKey" as shardKey between 100 to 199 in "ShardA"
and where as between 200 to 299 to "ShardB".

Inserted the documents containing shard keys to fall between both the
shards.
sourceKeys as "109" and "214".

But I still see the documents being written to "ShardA" even with the shard
key as "214".
It is supposed to go in to "ShardB".

May I know what I'm doing wrong here. Shard key type is String here.

Any suggestions.
Thanks for the help.


#Enable sharding on collection
sh.shardCollection("TST.AUDIT",{"sourceKey":1})

#Tag aware sharding rangessh.addTagRange("TST.AUDIT",{"sourceKey":"100"},{"sourceKey":"199"},
"ShardA")sh.addTagRange("TST.AUDIT",{"sourceKey":"200"},{"sourceKey":"299"},
"ShardB")


db.AUDIT.getShardDistribution()

Shard ShardA at rs1/iphost2:27020,iphost1:27017
data : 89kb docs : 101 chunks : 2
estimated data per chunk : 44kb
estimated docs per chunk : 50


Shard ShardB at rs2/iphost2:27017,iphost1:27020
data : 0b docs : 0 chunks : 1
estimated data per chunk : 0b
estimated docs per chunk : 0


Totals
data : 89kb docs : 101 chunks : 3
Shard ShardA contains 100% data, 100% docs in cluster, avg obj size on
shard : 911b
Shard ShardB contains 0% data, 0% docs in cluster, avg obj size on shard :
NaNGb
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/df29a8f4-615d-4c33-98ae-33a611514a98%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Chris De Bruyne
2015-11-26 12:12:29 UTC
Permalink
Off the top of my head I would say that you need to insert more data to
force the balancer to kick in.
You can lower the chunksize
(https://docs.mongodb.org/manual/reference/configuration-options/#sharding.chunkSize)
so this will happen faster, for testing purpose only of course .

Give it a try, and let us now

Kind regards
Chris
Post by Naveen
Trying to a multi regional sharded replicaset cluster with Mongodb 3.0.3.
I've a shard key called "sourceKey" which is of type "String".
Created 2 shards with "sourceKey" as shardKey between 100 to 199 in "ShardA"
and where as between 200 to 299 to "ShardB".
Inserted the documents containing shard keys to fall between both the
shards.
sourceKeys as "109" and "214".
But I still see the documents being written to "ShardA" even with the
shard key as "214".
It is supposed to go in to "ShardB".
May I know what I'm doing wrong here. Shard key type is String here.
Any suggestions.
Thanks for the help.
#Enable sharding on collection
sh.shardCollection("TST.AUDIT",{"sourceKey":1})
#Tag aware sharding rangessh.addTagRange("TST.AUDIT",{"sourceKey":"100"},{"sourceKey":"199"},
"ShardA")sh.addTagRange("TST.AUDIT",{"sourceKey":"200"},{"sourceKey":"299"},
"ShardB")
db.AUDIT.getShardDistribution()
Shard ShardA at rs1/iphost2:27020,iphost1:27017
data : 89kb docs : 101 chunks : 2
estimated data per chunk : 44kb
estimated docs per chunk : 50
Shard ShardB at rs2/iphost2:27017,iphost1:27020
data : 0b docs : 0 chunks : 1
estimated data per chunk : 0b
estimated docs per chunk : 0
Totals
data : 89kb docs : 101 chunks : 3
Shard ShardA contains 100% data, 100% docs in cluster, avg obj size on
shard : 911b
Shard ShardB contains 0% data, 0% docs in cluster, avg obj size on shard
: NaNGb
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/36e3229b-1818-4b21-8f9a-6a8f948c6f0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Naveen
2015-11-26 17:39:18 UTC
Permalink
Thanks Chris.

Had a hunch on this, but I thought that router will honor the tagging
rules, if I'm starting with new collections.
Will try it.
Post by Chris De Bruyne
Off the top of my head I would say that you need to insert more data to
force the balancer to kick in.
You can lower the chunksize (
https://docs.mongodb.org/manual/reference/configuration-options/#sharding.chunkSize)
so this will happen faster, for testing purpose only of course .
Give it a try, and let us now
Kind regards
Chris
Post by Naveen
Trying to a multi regional sharded replicaset cluster with Mongodb 3.0.3.
I've a shard key called "sourceKey" which is of type "String".
Created 2 shards with "sourceKey" as shardKey between 100 to 199 in "ShardA"
and where as between 200 to 299 to "ShardB".
Inserted the documents containing shard keys to fall between both the
shards.
sourceKeys as "109" and "214".
But I still see the documents being written to "ShardA" even with the
shard key as "214".
It is supposed to go in to "ShardB".
May I know what I'm doing wrong here. Shard key type is String here.
Any suggestions.
Thanks for the help.
#Enable sharding on collection
sh.shardCollection("TST.AUDIT",{"sourceKey":1})
#Tag aware sharding rangessh.addTagRange("TST.AUDIT",{"sourceKey":"100"},{"sourceKey":"199"},
"ShardA")sh.addTagRange("TST.AUDIT",{"sourceKey":"200"},{"sourceKey":"299"},
"ShardB")
db.AUDIT.getShardDistribution()
Shard ShardA at rs1/iphost2:27020,iphost1:27017
data : 89kb docs : 101 chunks : 2
estimated data per chunk : 44kb
estimated docs per chunk : 50
Shard ShardB at rs2/iphost2:27017,iphost1:27020
data : 0b docs : 0 chunks : 1
estimated data per chunk : 0b
estimated docs per chunk : 0
Totals
data : 89kb docs : 101 chunks : 3
Shard ShardA contains 100% data, 100% docs in cluster, avg obj size on
shard : 911b
Shard ShardB contains 0% data, 0% docs in cluster, avg obj size on
shard : NaNGb
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/94469783-04fe-434c-b7a9-d152b6021235%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-11-28 06:06:56 UTC
Permalink
The router will honor tagging rules and you don't need to change chunk size
or anything else. However how did you sent the tags on shards?

In general, it's helpful to show output to sh.status() so we can see all
relevant info like shard tags, whether balancer is enabled, etc.
Post by Naveen
Thanks Chris.
Had a hunch on this, but I thought that router will honor the tagging
rules, if I'm starting with new collections.
Will try it.
Post by Chris De Bruyne
Off the top of my head I would say that you need to insert more data to
force the balancer to kick in.
You can lower the chunksize (
https://docs.mongodb.org/manual/reference/configuration-options/#sharding.chunkSize)
so this will happen faster, for testing purpose only of course .
Give it a try, and let us now
Kind regards
Chris
Post by Naveen
Trying to a multi regional sharded replicaset cluster with Mongodb 3.0.3.
I've a shard key called "sourceKey" which is of type "String".
Created 2 shards with "sourceKey" as shardKey between 100 to 199 in "ShardA"
and where as between 200 to 299 to "ShardB".
Inserted the documents containing shard keys to fall between both the
shards.
sourceKeys as "109" and "214".
But I still see the documents being written to "ShardA" even with the
shard key as "214".
It is supposed to go in to "ShardB".
May I know what I'm doing wrong here. Shard key type is String here.
Any suggestions.
Thanks for the help.
#Enable sharding on collection
sh.shardCollection("TST.AUDIT",{"sourceKey":1})
#Tag aware sharding rangessh.addTagRange("TST.AUDIT",{"sourceKey":"100"},{"sourceKey":"199"},
"ShardA")sh.addTagRange("TST.AUDIT",{"sourceKey":"200"},{"sourceKey":"299"},
"ShardB")
db.AUDIT.getShardDistribution()
Shard ShardA at rs1/iphost2:27020,iphost1:27017
data : 89kb docs : 101 chunks : 2
estimated data per chunk : 44kb
estimated docs per chunk : 50
Shard ShardB at rs2/iphost2:27017,iphost1:27020
data : 0b docs : 0 chunks : 1
estimated data per chunk : 0b
estimated docs per chunk : 0
Totals
data : 89kb docs : 101 chunks : 3
Shard ShardA contains 100% data, 100% docs in cluster, avg obj size on
shard : 911b
Shard ShardB contains 0% data, 0% docs in cluster, avg obj size on
shard : NaNGb
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/94469783-04fe-434c-b7a9-d152b6021235%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/94469783-04fe-434c-b7a9-d152b6021235%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJD1%3Dj%2BTRxKnPm-VDY6PWqMuiUvrOyCsSrgLBVZzHsMWrA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Naveen
2015-11-28 19:07:04 UTC
Permalink
Thanks Asya.
Here are the status, shard distribution and the query results to show that
keys fall in both tag ranges.
Chunk difference between shards is just 1.
Post by Asya Kamsky
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5657dabcaf958ce7a4072901")
}
shards:
{ "_id" : "ShardA", "host" : "rs1/host1:27020,host2:27017", "tags" : [
"ShardA" ] }
{ "_id" : "ShardB", "host" : "rs2/host1:27017,host2:27020", "tags" : [
"ShardB" ] }
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "TST", "partitioned" : true, "primary" : "ShardB" }
TST.AUDIT
shard key: { "sourceKey" : 1 }
chunks:
ShardB 1
ShardA 2
{ "sourceKey" : { $minKey : 1 } } -->> { "sourceKey" : "100" } on : ShardB
Timestamp(2000, 1)
{ "sourceKey" : "100" } -->> { "sourceKey" : "200" } on : ShardA Timestamp(
2000, 2)
{ "sourceKey" : "200" } -->> { "sourceKey" : { $maxKey : 1 } } on : ShardA
Timestamp(2000, 3)
tag: ShardA { "sourceKey" : "100" } -->> { "sourceKey" : "199" }
tag: ShardB { "sourceKey" : "200" } -->> { "sourceKey" : "299" }




# Collection data
Post by Asya Kamsky
db.AUDIT.distinct("sourceKey")
{
"0" : "120",
"1" : "215"
}
Post by Asya Kamsky
db.AUDIT.getShardDistribution()
Shard ShardA at rs1/host1:27020,host2:27017
data : 199.31Mb docs : 51224 chunks : 2
estimated data per chunk : 99.65Mb
estimated docs per chunk : 25612


Shard ShardB at rs2/host1:27017,host2:27020
data : 0b docs : 0 chunks : 1
estimated data per chunk : 0b
estimated docs per chunk : 0


Totals
data : 199.31Mb docs : 51224 chunks : 3
Shard ShardA contains 100% data, 100% docs in cluster, avg obj size on
shard : 3kb
Shard ShardB contains 0% data, 0% docs in cluster, avg obj size on shard :
NaNGb
Post by Asya Kamsky
The router will honor tagging rules and you don't need to change chunk
size or anything else. However how did you sent the tags on shards?
In general, it's helpful to show output to sh.status() so we can see all
relevant info like shard tags, whether balancer is enabled, etc.
Post by Naveen
Thanks Chris.
Had a hunch on this, but I thought that router will honor the tagging
rules, if I'm starting with new collections.
Will try it.
Post by Chris De Bruyne
Off the top of my head I would say that you need to insert more data to
force the balancer to kick in.
You can lower the chunksize (
https://docs.mongodb.org/manual/reference/configuration-options/#sharding.chunkSize)
so this will happen faster, for testing purpose only of course .
Give it a try, and let us now
Kind regards
Chris
Post by Naveen
Trying to a multi regional sharded replicaset cluster with Mongodb 3.0.3.
I've a shard key called "sourceKey" which is of type "String".
Created 2 shards with "sourceKey" as shardKey between 100 to 199 in "ShardA"
and where as between 200 to 299 to "ShardB".
Inserted the documents containing shard keys to fall between both the
shards.
sourceKeys as "109" and "214".
But I still see the documents being written to "ShardA" even with the
shard key as "214".
It is supposed to go in to "ShardB".
May I know what I'm doing wrong here. Shard key type is String here.
Any suggestions.
Thanks for the help.
#Enable sharding on collection
sh.shardCollection("TST.AUDIT",{"sourceKey":1})
#Tag aware sharding rangessh.addTagRange("TST.AUDIT",{"sourceKey":"100"},{"sourceKey":"199"},
"ShardA")sh.addTagRange("TST.AUDIT",{"sourceKey":"200"},{"sourceKey":"299"},
"ShardB")
db.AUDIT.getShardDistribution()
Shard ShardA at rs1/iphost2:27020,iphost1:27017
data : 89kb docs : 101 chunks : 2
estimated data per chunk : 44kb
estimated docs per chunk : 50
Shard ShardB at rs2/iphost2:27017,iphost1:27020
data : 0b docs : 0 chunks : 1
estimated data per chunk : 0b
estimated docs per chunk : 0
Totals
data : 89kb docs : 101 chunks : 3
Shard ShardA contains 100% data, 100% docs in cluster, avg obj size
on shard : 911b
Shard ShardB contains 0% data, 0% docs in cluster, avg obj size on
shard : NaNGb
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/94469783-04fe-434c-b7a9-d152b6021235%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/94469783-04fe-434c-b7a9-d152b6021235%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/cd420518-09f9-4d06-b75f-288e17b772e9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-11-28 23:54:36 UTC
Permalink
What version is this? In the latest, sh.status() outputs whether the
balancer is running, when it last balanced successfully, etc.

I see none of that information. Could you provide exact server version for
your shards/mongos/configservers?

Note that your chunks are correctly split at the tag boundary:
100->200 (not-inclusive)
and 200->maxkey are separate chunks, but 200+ chunk has not been moved to
ShardB and I suspect it's because the balancer either isn't running or
isn't able to successfully move the chunk to the other shard.

Asya
Post by Naveen
Thanks Asya.
Here are the status, shard distribution and the query results to show that
keys fall in both tag ranges.
Chunk difference between shards is just 1.
Post by Asya Kamsky
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5657dabcaf958ce7a4072901")
}
{ "_id" : "ShardA", "host" : "rs1/host1:27020,host2:27017", "tags" : [
"ShardA" ] }
{ "_id" : "ShardB", "host" : "rs2/host1:27017,host2:27020", "tags" : [
"ShardB" ] }
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "TST", "partitioned" : true, "primary" : "ShardB" }
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardB 1
ShardA 2
ShardB Timestamp(2000, 1)
{ "sourceKey" : "100" } -->> { "sourceKey" : "200" } on : ShardA
Timestamp(2000, 2)
ShardA Timestamp(2000, 3)
tag: ShardA { "sourceKey" : "100" } -->> { "sourceKey" : "199" }
tag: ShardB { "sourceKey" : "200" } -->> { "sourceKey" : "299" }
# Collection data
Post by Asya Kamsky
db.AUDIT.distinct("sourceKey")
{
"0" : "120",
"1" : "215"
}
Post by Asya Kamsky
db.AUDIT.getShardDistribution()
Shard ShardA at rs1/host1:27020,host2:27017
data : 199.31Mb docs : 51224 chunks : 2
estimated data per chunk : 99.65Mb
estimated docs per chunk : 25612
Shard ShardB at rs2/host1:27017,host2:27020
data : 0b docs : 0 chunks : 1
estimated data per chunk : 0b
estimated docs per chunk : 0
Totals
data : 199.31Mb docs : 51224 chunks : 3
Shard ShardA contains 100% data, 100% docs in cluster, avg obj size on
shard : 3kb
Shard ShardB contains 0% data, 0% docs in cluster, avg obj size on shard
: NaNGb
Post by Asya Kamsky
The router will honor tagging rules and you don't need to change chunk
size or anything else. However how did you sent the tags on shards?
In general, it's helpful to show output to sh.status() so we can see all
relevant info like shard tags, whether balancer is enabled, etc.
Post by Naveen
Thanks Chris.
Had a hunch on this, but I thought that router will honor the tagging
rules, if I'm starting with new collections.
Will try it.
Post by Chris De Bruyne
Off the top of my head I would say that you need to insert more data to
force the balancer to kick in.
You can lower the chunksize (
https://docs.mongodb.org/manual/reference/configuration-options/#sharding.chunkSize)
so this will happen faster, for testing purpose only of course .
Give it a try, and let us now
Kind regards
Chris
Post by Naveen
Trying to a multi regional sharded replicaset cluster with Mongodb 3.0.3.
I've a shard key called "sourceKey" which is of type "String".
Created 2 shards with "sourceKey" as shardKey between 100 to 199 in "ShardA"
and where as between 200 to 299 to "ShardB".
Inserted the documents containing shard keys to fall between both the
shards.
sourceKeys as "109" and "214".
But I still see the documents being written to "ShardA" even with the
shard key as "214".
It is supposed to go in to "ShardB".
May I know what I'm doing wrong here. Shard key type is String here.
Any suggestions.
Thanks for the help.
#Enable sharding on collection
sh.shardCollection("TST.AUDIT",{"sourceKey":1})
#Tag aware sharding rangessh.addTagRange("TST.AUDIT",{"sourceKey":"100"},{"sourceKey":"199"},
"ShardA")sh.addTagRange("TST.AUDIT",{"sourceKey":"200"},{"sourceKey":"299"},
"ShardB")
db.AUDIT.getShardDistribution()
Shard ShardA at rs1/iphost2:27020,iphost1:27017
data : 89kb docs : 101 chunks : 2
estimated data per chunk : 44kb
estimated docs per chunk : 50
Shard ShardB at rs2/iphost2:27017,iphost1:27020
data : 0b docs : 0 chunks : 1
estimated data per chunk : 0b
estimated docs per chunk : 0
Totals
data : 89kb docs : 101 chunks : 3
Shard ShardA contains 100% data, 100% docs in cluster, avg obj size
on shard : 911b
Shard ShardB contains 0% data, 0% docs in cluster, avg obj size on
shard : NaNGb
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/94469783-04fe-434c-b7a9-d152b6021235%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/94469783-04fe-434c-b7a9-d152b6021235%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/cd420518-09f9-4d06-b75f-288e17b772e9%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/cd420518-09f9-4d06-b75f-288e17b772e9%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJCi-CBaZqt2dV5kttGb1W50ef4iNHDKaDCLtjswqQvFjA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-11-29 00:09:33 UTC
Permalink
Actually I think I see the answer to my question (you did say Mongodb 3.0.3
so I'm not sure why the balancer information is not there).

But I can tell you that distribution tells me that there is definitely a
problem with your metadata - there should have been chunks splits happening
to keep chunks below 64Mbs:

estimated data per chunk : 99.65Mb

I think you should check your logs and see if all config servers are up and
if the metadata can successfully be changed. Output of balancer state
would definitely help out here.

Asya
Post by Asya Kamsky
What version is this? In the latest, sh.status() outputs whether the
balancer is running, when it last balanced successfully, etc.
I see none of that information. Could you provide exact server version
for your shards/mongos/configservers?
100->200 (not-inclusive)
and 200->maxkey are separate chunks, but 200+ chunk has not been moved to
ShardB and I suspect it's because the balancer either isn't running or
isn't able to successfully move the chunk to the other shard.
Asya
Post by Naveen
Thanks Asya.
Here are the status, shard distribution and the query results to show
that keys fall in both tag ranges.
Chunk difference between shards is just 1.
Post by Asya Kamsky
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5657dabcaf958ce7a4072901")
}
[ "ShardA" ] }
[ "ShardB" ] }
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "TST", "partitioned" : true, "primary" : "ShardB" }
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardB 1
ShardA 2
ShardB Timestamp(2000, 1)
{ "sourceKey" : "100" } -->> { "sourceKey" : "200" } on : ShardA
Timestamp(2000, 2)
ShardA Timestamp(2000, 3)
tag: ShardA { "sourceKey" : "100" } -->> { "sourceKey" : "199" }
tag: ShardB { "sourceKey" : "200" } -->> { "sourceKey" : "299" }
# Collection data
Post by Asya Kamsky
db.AUDIT.distinct("sourceKey")
{
"0" : "120",
"1" : "215"
}
Post by Asya Kamsky
db.AUDIT.getShardDistribution()
Shard ShardA at rs1/host1:27020,host2:27017
data : 199.31Mb docs : 51224 chunks : 2
estimated data per chunk : 99.65Mb
estimated docs per chunk : 25612
Shard ShardB at rs2/host1:27017,host2:27020
data : 0b docs : 0 chunks : 1
estimated data per chunk : 0b
estimated docs per chunk : 0
Totals
data : 199.31Mb docs : 51224 chunks : 3
Shard ShardA contains 100% data, 100% docs in cluster, avg obj size on
shard : 3kb
Shard ShardB contains 0% data, 0% docs in cluster, avg obj size on
shard : NaNGb
Post by Asya Kamsky
The router will honor tagging rules and you don't need to change chunk
size or anything else. However how did you sent the tags on shards?
In general, it's helpful to show output to sh.status() so we can see all
relevant info like shard tags, whether balancer is enabled, etc.
Post by Naveen
Thanks Chris.
Had a hunch on this, but I thought that router will honor the tagging
rules, if I'm starting with new collections.
Will try it.
Post by Chris De Bruyne
Off the top of my head I would say that you need to insert more data
to force the balancer to kick in.
You can lower the chunksize (
https://docs.mongodb.org/manual/reference/configuration-options/#sharding.chunkSize)
so this will happen faster, for testing purpose only of course .
Give it a try, and let us now
Kind regards
Chris
Post by Naveen
Trying to a multi regional sharded replicaset cluster with Mongodb 3.0.3.
I've a shard key called "sourceKey" which is of type "String".
Created 2 shards with "sourceKey" as shardKey between 100 to 199 in "ShardA"
and where as between 200 to 299 to "ShardB".
Inserted the documents containing shard keys to fall between both
the shards.
sourceKeys as "109" and "214".
But I still see the documents being written to "ShardA" even with
the shard key as "214".
It is supposed to go in to "ShardB".
May I know what I'm doing wrong here. Shard key type is String here.
Any suggestions.
Thanks for the help.
#Enable sharding on collection
sh.shardCollection("TST.AUDIT",{"sourceKey":1})
#Tag aware sharding rangessh.addTagRange("TST.AUDIT",{"sourceKey":"100"},{"sourceKey":"199"},
"ShardA")sh.addTagRange("TST.AUDIT",{"sourceKey":"200"},{"sourceKey":"299"},
"ShardB")
db.AUDIT.getShardDistribution()
Shard ShardA at rs1/iphost2:27020,iphost1:27017
data : 89kb docs : 101 chunks : 2
estimated data per chunk : 44kb
estimated docs per chunk : 50
Shard ShardB at rs2/iphost2:27017,iphost1:27020
data : 0b docs : 0 chunks : 1
estimated data per chunk : 0b
estimated docs per chunk : 0
Totals
data : 89kb docs : 101 chunks : 3
Shard ShardA contains 100% data, 100% docs in cluster, avg obj size
on shard : 911b
Shard ShardB contains 0% data, 0% docs in cluster, avg obj size on
shard : NaNGb
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/94469783-04fe-434c-b7a9-d152b6021235%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/94469783-04fe-434c-b7a9-d152b6021235%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/cd420518-09f9-4d06-b75f-288e17b772e9%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/cd420518-09f9-4d06-b75f-288e17b772e9%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJBmJ7JXs8m415qthawuakUE-nSdmEO0ffwZ0i2nyLoT_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Naveen
2015-11-29 02:41:35 UTC
Permalink
sh.getBalancerState()
true

I've also inserted new record and started balancer manually to see the
impact, with no useful impact on shard distribution.

Noticed that chunk splits are not honoring my tag ranges.
Though my tag range for ShardB is between "200" and "299", actual chunk
split is showing '1" to "100" for ShardB.
Highlighted them in the below section. Just to inform you my shard key is
of String type.



TST.AUDIT
shard key: { "sourceKey" : 1 }
chunks:
ShardB 1
ShardA 2
{ "sourceKey" : { $minKey : 1 } } -->> { "sourceKey" : "100" } on : ShardB
Timestamp(2000, 1)
{ "sourceKey" : "100" } -->> { "sourceKey" : "200" } on : ShardA Timestamp(
2000, 2)
{ "sourceKey" : "200" } -->> { "sourceKey" : { $maxKey : 1 } } on : ShardA
Timestamp(2000, 3)
tag: ShardA { "sourceKey" : "100" } -->> { "sourceKey" : "199" }
tag:* ShardB { "sourceKey" : "200" } -->> { "sourceKey" : "299" }*


Thanks
Actually I think I see the answer to my question (you did say Mongodb
3.0.3 so I'm not sure why the balancer information is not there).
But I can tell you that distribution tells me that there is definitely a
problem with your metadata - there should have been chunks splits happening
estimated data per chunk : 99.65Mb
I think you should check your logs and see if all config servers are up
and if the metadata can successfully be changed. Output of balancer state
would definitely help out here.
Asya
Post by Asya Kamsky
What version is this? In the latest, sh.status() outputs whether the
balancer is running, when it last balanced successfully, etc.
I see none of that information. Could you provide exact server version
for your shards/mongos/configservers?
100->200 (not-inclusive)
and 200->maxkey are separate chunks, but 200+ chunk has not been moved to
ShardB and I suspect it's because the balancer either isn't running or
isn't able to successfully move the chunk to the other shard.
Asya
Post by Naveen
Thanks Asya.
Here are the status, shard distribution and the query results to show
that keys fall in both tag ranges.
Chunk difference between shards is just 1.
Post by Asya Kamsky
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5657dabcaf958ce7a4072901")
}
[ "ShardA" ] }
[ "ShardB" ] }
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "TST", "partitioned" : true, "primary" : "ShardB" }
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardB 1
ShardA 2
ShardB Timestamp(2000, 1)
{ "sourceKey" : "100" } -->> { "sourceKey" : "200" } on : ShardA
Timestamp(2000, 2)
ShardA Timestamp(2000, 3)
tag: ShardA { "sourceKey" : "100" } -->> { "sourceKey" : "199" }
tag: ShardB { "sourceKey" : "200" } -->> { "sourceKey" : "299" }
# Collection data
Post by Asya Kamsky
db.AUDIT.distinct("sourceKey")
{
"0" : "120",
"1" : "215"
}
Post by Asya Kamsky
db.AUDIT.getShardDistribution()
Shard ShardA at rs1/host1:27020,host2:27017
data : 199.31Mb docs : 51224 chunks : 2
estimated data per chunk : 99.65Mb
estimated docs per chunk : 25612
Shard ShardB at rs2/host1:27017,host2:27020
data : 0b docs : 0 chunks : 1
estimated data per chunk : 0b
estimated docs per chunk : 0
Totals
data : 199.31Mb docs : 51224 chunks : 3
Shard ShardA contains 100% data, 100% docs in cluster, avg obj size on
shard : 3kb
Shard ShardB contains 0% data, 0% docs in cluster, avg obj size on
shard : NaNGb
Post by Asya Kamsky
The router will honor tagging rules and you don't need to change chunk
size or anything else. However how did you sent the tags on shards?
In general, it's helpful to show output to sh.status() so we can see
all relevant info like shard tags, whether balancer is enabled, etc.
Post by Naveen
Thanks Chris.
Had a hunch on this, but I thought that router will honor the tagging
rules, if I'm starting with new collections.
Will try it.
Post by Chris De Bruyne
Off the top of my head I would say that you need to insert more data
to force the balancer to kick in.
You can lower the chunksize (
https://docs.mongodb.org/manual/reference/configuration-options/#sharding.chunkSize)
so this will happen faster, for testing purpose only of course .
Give it a try, and let us now
Kind regards
Chris
Post by Naveen
Trying to a multi regional sharded replicaset cluster with Mongodb 3.0.3.
I've a shard key called "sourceKey" which is of type "String".
Created 2 shards with "sourceKey" as shardKey between 100 to 199 in "ShardA"
and where as between 200 to 299 to "ShardB".
Inserted the documents containing shard keys to fall between both
the shards.
sourceKeys as "109" and "214".
But I still see the documents being written to "ShardA" even with
the shard key as "214".
It is supposed to go in to "ShardB".
May I know what I'm doing wrong here. Shard key type is String here.
Any suggestions.
Thanks for the help.
#Enable sharding on collection
sh.shardCollection("TST.AUDIT",{"sourceKey":1})
#Tag aware sharding rangessh.addTagRange("TST.AUDIT",{"sourceKey":"100"},{"sourceKey":"199"},
"ShardA")sh.addTagRange("TST.AUDIT",{"sourceKey":"200"},{"sourceKey":"299"},
"ShardB")
db.AUDIT.getShardDistribution()
Shard ShardA at rs1/iphost2:27020,iphost1:27017
data : 89kb docs : 101 chunks : 2
estimated data per chunk : 44kb
estimated docs per chunk : 50
Shard ShardB at rs2/iphost2:27017,iphost1:27020
data : 0b docs : 0 chunks : 1
estimated data per chunk : 0b
estimated docs per chunk : 0
Totals
data : 89kb docs : 101 chunks : 3
Shard ShardA contains 100% data, 100% docs in cluster, avg obj
size on shard : 911b
Shard ShardB contains 0% data, 0% docs in cluster, avg obj size on
shard : NaNGb
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/94469783-04fe-434c-b7a9-d152b6021235%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/94469783-04fe-434c-b7a9-d152b6021235%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
<javascript:>.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/cd420518-09f9-4d06-b75f-288e17b772e9%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/cd420518-09f9-4d06-b75f-288e17b772e9%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/e19aee0c-2054-4b9f-a12c-35d81056660c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-11-29 21:49:01 UTC
Permalink
The splits ARE honoring your tagging - the reason the split exists on value
"200" is because it's where your tagged ranges split.

Is there a reason you don't want to look at the results of the balancer's
attempts to balance? It's clear to me that they must be failing. Without
seeing the error, I can't tell you *why* they are failing.

sh.status() shows the last five balancer attempt results.
Logs will have errors for all balancer attempts.

Asya
Post by Naveen
sh.getBalancerState()
true
I've also inserted new record and started balancer manually to see the
impact, with no useful impact on shard distribution.
Noticed that chunk splits are not honoring my tag ranges.
Though my tag range for ShardB is between "200" and "299", actual chunk
split is showing '1" to "100" for ShardB.
Highlighted them in the below section. Just to inform you my shard key is
of String type.
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardB 1
ShardA 2
ShardB Timestamp(2000, 1)
{ "sourceKey" : "100" } -->> { "sourceKey" : "200" } on : ShardA
Timestamp(2000, 2)
ShardA Timestamp(2000, 3)
tag: ShardA { "sourceKey" : "100" } -->> { "sourceKey" : "199" }
tag:* ShardB { "sourceKey" : "200" } -->> { "sourceKey" : "299" }*
Thanks
Actually I think I see the answer to my question (you did say Mongodb
3.0.3 so I'm not sure why the balancer information is not there).
But I can tell you that distribution tells me that there is definitely a
problem with your metadata - there should have been chunks splits happening
estimated data per chunk : 99.65Mb
I think you should check your logs and see if all config servers are up
and if the metadata can successfully be changed. Output of balancer state
would definitely help out here.
Asya
Post by Asya Kamsky
What version is this? In the latest, sh.status() outputs whether the
balancer is running, when it last balanced successfully, etc.
I see none of that information. Could you provide exact server version
for your shards/mongos/configservers?
100->200 (not-inclusive)
and 200->maxkey are separate chunks, but 200+ chunk has not been moved
to ShardB and I suspect it's because the balancer either isn't running or
isn't able to successfully move the chunk to the other shard.
Asya
Post by Naveen
Thanks Asya.
Here are the status, shard distribution and the query results to show
that keys fall in both tag ranges.
Chunk difference between shards is just 1.
Post by Asya Kamsky
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5657dabcaf958ce7a4072901")
}
{ "_id" : "ShardA", "host" : "rs1/host1:27020,host2:27017", "tags"
: [ "ShardA" ] }
{ "_id" : "ShardB", "host" : "rs2/host1:27017,host2:27020", "tags"
: [ "ShardB" ] }
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "TST", "partitioned" : true, "primary" : "ShardB" }
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardB 1
ShardA 2
ShardB Timestamp(2000, 1)
{ "sourceKey" : "100" } -->> { "sourceKey" : "200" } on : ShardA
Timestamp(2000, 2)
ShardA Timestamp(2000, 3)
tag: ShardA { "sourceKey" : "100" } -->> { "sourceKey" : "199" }
tag: ShardB { "sourceKey" : "200" } -->> { "sourceKey" : "299" }
# Collection data
Post by Asya Kamsky
db.AUDIT.distinct("sourceKey")
{
"0" : "120",
"1" : "215"
}
Post by Asya Kamsky
db.AUDIT.getShardDistribution()
Shard ShardA at rs1/host1:27020,host2:27017
data : 199.31Mb docs : 51224 chunks : 2
estimated data per chunk : 99.65Mb
estimated docs per chunk : 25612
Shard ShardB at rs2/host1:27017,host2:27020
data : 0b docs : 0 chunks : 1
estimated data per chunk : 0b
estimated docs per chunk : 0
Totals
data : 199.31Mb docs : 51224 chunks : 3
Shard ShardA contains 100% data, 100% docs in cluster, avg obj size
on shard : 3kb
Shard ShardB contains 0% data, 0% docs in cluster, avg obj size on
shard : NaNGb
Post by Asya Kamsky
The router will honor tagging rules and you don't need to change chunk
size or anything else. However how did you sent the tags on shards?
In general, it's helpful to show output to sh.status() so we can see
all relevant info like shard tags, whether balancer is enabled, etc.
Post by Naveen
Thanks Chris.
Had a hunch on this, but I thought that router will honor the tagging
rules, if I'm starting with new collections.
Will try it.
Post by Chris De Bruyne
Off the top of my head I would say that you need to insert more data
to force the balancer to kick in.
You can lower the chunksize (
https://docs.mongodb.org/manual/reference/configuration-options/#sharding.chunkSize)
so this will happen faster, for testing purpose only of course .
Give it a try, and let us now
Kind regards
Chris
Post by Naveen
Trying to a multi regional sharded replicaset cluster with Mongodb 3.0.3.
I've a shard key called "sourceKey" which is of type "String".
Created 2 shards with "sourceKey" as shardKey between 100 to 199
in "ShardA" and where as between 200 to 299 to "ShardB".
Inserted the documents containing shard keys to fall between both
the shards.
sourceKeys as "109" and "214".
But I still see the documents being written to "ShardA" even with
the shard key as "214".
It is supposed to go in to "ShardB".
May I know what I'm doing wrong here. Shard key type is String here.
Any suggestions.
Thanks for the help.
#Enable sharding on collection
sh.shardCollection("TST.AUDIT",{"sourceKey":1})
#Tag aware sharding rangessh.addTagRange("TST.AUDIT",{"sourceKey":"100"},{"sourceKey":"199"},
"ShardA")sh.addTagRange("TST.AUDIT",{"sourceKey":"200"},{"sourceKey":"299"},
"ShardB")
db.AUDIT.getShardDistribution()
Shard ShardA at rs1/iphost2:27020,iphost1:27017
data : 89kb docs : 101 chunks : 2
estimated data per chunk : 44kb
estimated docs per chunk : 50
Shard ShardB at rs2/iphost2:27017,iphost1:27020
data : 0b docs : 0 chunks : 1
estimated data per chunk : 0b
estimated docs per chunk : 0
Totals
data : 89kb docs : 101 chunks : 3
Shard ShardA contains 100% data, 100% docs in cluster, avg obj
size on shard : 911b
Shard ShardB contains 0% data, 0% docs in cluster, avg obj size
on shard : NaNGb
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/94469783-04fe-434c-b7a9-d152b6021235%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/94469783-04fe-434c-b7a9-d152b6021235%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/cd420518-09f9-4d06-b75f-288e17b772e9%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/cd420518-09f9-4d06-b75f-288e17b772e9%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/e19aee0c-2054-4b9f-a12c-35d81056660c%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/e19aee0c-2054-4b9f-a12c-35d81056660c%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJCYs6jbi3PVwW5_pirHmVqbJ4jd_hbyq%2BOwY%3DHbCX2h%2Bw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Naveen
2015-11-30 08:36:33 UTC
Permalink
One of my other collection is having nested attribute and it's failing.

sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5657dabcaf958ce7a4072901")
}
shards:
{ "_id" : "ShardA", "host" : "rs1/host1:27020,host2:27017",
"tags" : [ "ShardA" ] }
{ "_id" : "ShardB", "host" : "rs2/host1:27017,host2:27020",
"tags" : [ "ShardB" ] }
balancer:
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 5
Last reported error: field names of bound { operation.key: "100" }
do not match those of keyPattern { operation.key: 1.0 }
Time of Reported error: Mon Nov 30 2015 10:12:50 GMT+0530 (India
Standard Time)
Migration Results for the last 24 hours:
No recent migrations
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "TST", "partitioned" : true, "primary" : "ShardB" }
TST.AUDIT
shard key: { "sourceKey" : 1 }
chunks:
ShardA 2
ShardB 1
{ "sourceKey" : { "$minKey" : 1 } } -->> {
"sourceKey" : "100" } on : ShardB Timestamp(2, 1)
{ "sourceKey" : "100" } -->> { "sourceKey" : "200"}
on : ShardA Timestamp(2, 2)
{ "sourceKey" : "200" } -->> { "sourceKey" : {
"$maxKey" : 1 } } on : ShardA Timestamp(2, 3)
tag: ShardA { "sourceKey" : "100" } -->> {
"sourceKey" : "199" }
tag: ShardB { "sourceKey" : "200" } -->> {
"sourceKey" : "299" }

{ "_id" : "RAW_DATA", "partitioned" : true, "primary" : "ShardB"
}
RAW_DATA.OPERATIONAL_METRICS
shard key: { "operation.key" : 1 }
chunks:
ShardA 1
ShardB 3
{ "operation.key" : { "$minKey" : 1 } } -->> {
"operation.key" : "105" } on : ShardB Timestamp(2, 1)
{ "operation.key" : "105" } -->> { "operation.key"
: "205" } on : ShardB Timestamp(1, 2)
{ "operation.key" : "205" } -->> { "operation.key"
: "215" } on : ShardB Timestamp(1, 3)
{ "operation.key" : "215" } -->> { "operation.key"
: { "$maxKey" : 1 } } on : ShardA Timestamp(2, 0)
tag: ShardA { "operation.key" : "100" } -->> {
"operation.key" : "199" }
tag: ShardB { "operation.key" : "200" } -->> {
"operation.key" : "299" }


Thanks
Post by Asya Kamsky
The splits ARE honoring your tagging - the reason the split exists on
value "200" is because it's where your tagged ranges split.
Is there a reason you don't want to look at the results of the balancer's
attempts to balance? It's clear to me that they must be failing. Without
seeing the error, I can't tell you *why* they are failing.
sh.status() shows the last five balancer attempt results.
Logs will have errors for all balancer attempts.
Asya
sh.getBalancerState()
true
I've also inserted new record and started balancer manually to see the
impact, with no useful impact on shard distribution.
Noticed that chunk splits are not honoring my tag ranges.
Though my tag range for ShardB is between "200" and "299", actual chunk
split is showing '1" to "100" for ShardB.
Highlighted them in the below section. Just to inform you my shard key is
of String type.
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardB 1
ShardA 2
ShardB Timestamp(2000, 1)
{ "sourceKey" : "100" } -->> { "sourceKey" : "200" } on : ShardA
Timestamp(2000, 2)
ShardA Timestamp(2000, 3)
tag: ShardA { "sourceKey" : "100" } -->> { "sourceKey" : "199" }
tag:* ShardB { "sourceKey" : "200" } -->> { "sourceKey" : "299" }*
Thanks
Actually I think I see the answer to my question (you did say Mongodb
3.0.3 so I'm not sure why the balancer information is not there).
But I can tell you that distribution tells me that there is definitely a
problem with your metadata - there should have been chunks splits happening
estimated data per chunk : 99.65Mb
I think you should check your logs and see if all config servers are up
and if the metadata can successfully be changed. Output of balancer state
would definitely help out here.
Asya
What version is this? In the latest, sh.status() outputs whether the
balancer is running, when it last balanced successfully, etc.
I see none of that information. Could you provide exact server version
for your shards/mongos/configservers?
100->200 (not-inclusive)
and 200->maxkey are separate chunks, but 200+ chunk has not been moved to
ShardB and I suspect it's because the balancer either isn't running or
isn't able to successfully move the chunk to the other shard.
Asya
Thanks Asya.
Here are the status, shard distribution and the query results to show that
keys fall in both tag ranges.
Chunk difference between shards is just 1.
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5657dabcaf958ce7a4072901")
}
{ "_id" : "ShardA", "host" : "rs1/host1:27020,host2:27017", "tags" : [
"ShardA" ] }
{ "_id" : "ShardB", "host" : "rs2/host1:27017,host2:27020", "tags" : [
"ShardB" ] }
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "TST", "partitioned" : true, "primary" : "ShardB" }
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardB 1
ShardA 2
ShardB Timestamp(2000, 1)
{ "sourceKey" : "100" } -->> { "sourceKey" : "200" } on : ShardA
Timestamp(2000, 2)
ShardA Timestamp(2000, 3)
tag: ShardA { "sourceKey" : "100" } -->> { "sourceKey" : "199" }
tag: ShardB { "sourceKey" : "200" } -->> { "sourceKey" : "299" }
# Collection data
db.AUDIT.distinct("sourceKey")
{
"0" : "120",
"1" : "215"
}
db.AUDIT.getShardDistribution()
Shard ShardA at rs1/host1:27020,host2:27017
data : 199.31Mb docs : 51224 chunks : 2
estimated data per chunk : 99.65Mb
estimated docs per chunk : 25612
Shard ShardB at rs2/host1:27017,host2:27020
data : 0b docs : 0 chunks : 1
estimated data per chunk : 0b
estimated docs per chunk : 0
Totals
data : 199.31Mb docs : 51224 chunks : 3
Shard ShardA contains 100% data, 100% docs in cluster, avg obj size on
shard : 3kb
Shard ShardB contains 0% data, 0% docs in cluster, avg obj size on shard
: NaNGb
The router will honor tagging rules and you don't need to change chunk
size or anything else. However how did you sent the tags on shards?
In general, it's helpful to show output to sh.status() so we can see all
relevant info like shard tags, whether balancer is enabled, etc.
Thanks Chris.
Had a hunch on this, but I thought that router will honor the tagging
rules, if I'm starting with new collections.
Will try it.
Off the top of my head I would say that you need to insert more data to
force the balancer to kick in.
You can lower the chunksize (
https://docs.mongodb.org/manual/reference/configuration-options/#sharding.chunkSize)
so this will happen faster, for testing purpose only of course .
Give it a try, and let us now
Kind regards
Chris
Trying to a multi regional sharded replicaset cluster with Mongodb 3.0.3.
I've a shard key called "sourceKey" which is of type "String".
Created 2 shards with "sourceKey" as shardKey between 100 to 199 in "ShardA"
and where as between 200 to 299 to "ShardB".
Inserted the documents containing shard keys to fall between both the
shards.
sourceKeys as "109" and "214".
But I still see the documents being written to "ShardA" even with the
shard key as "214".
It is supposed to go in to "ShardB".
May I know what I'm doing wrong here. Shard key type is String here.
Any suggestions.
Thanks for the help.
#Enable sharding on collection
sh.shardCollection("TST.AUDIT",{"sourceKey":1})
#Tag aware sharding rangessh.addTagRange("TST.AUDIT",{"sourceKey":"100"},{"sourceKey":"199"},
"ShardA")sh.addTagRange("TST.AUDIT",{"sourceKey":"200"},{"sourceKey":"299"},
"ShardB")
db.AUDIT.getShardDistribution()
Shard ShardA at rs1/iphost2:27020,iphost1:27017
data : 89kb docs : 101 chunks : 2
estimated data per chunk : 44kb
estimated docs per chunk : 50
Shard ShardB at rs2/iphost2:27017,iphost1:27020
...
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/599fa4e0-7786-42a2-a80b-f4e4bb0f648a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Naveen
2015-11-30 16:40:56 UTC
Permalink
Asya,

Tried with latest mongodb version 3.0.7
I've observed the below error while the balancer was actually trying to
move the chunks, it is failing on the other collection which has embedded
attribute
"operation.key".
field names of bound { operation.key: "100" } do not match those of
keyPattern { operation.key: 1.0 }

Problem could be that RAW_DATA.OPERATIONAL_METRICS collection's shard key
was created in the following manner, as it was not working with the "dot",
we have used "\uff0e" while adding tag ranges.

When we used embedded attribute we see the following error. To avoid this
we have used "\uff0e", that seems the problem.

mongos> sh.addTagRange("RAW_DATA.OPERATIONAL_METRICS",{"operation.Key":"100"
},{"operation.key":"200"}, "ShardA")

2015-11-30T08:38:36.084-0800 E QUERY Error: can't have . in field names
[operation.Key]
at Error (<anonymous>)
at DBCollection._validateForStorage
(src/mongo/shell/collection.js:157:19)
at DBCollection._validateForStorage
(src/mongo/shell/collection.js:165:18)
at DBCollection._validateForStorage
(src/mongo/shell/collection.js:165:18)
at DBCollection._validateUpdateDoc
(src/mongo/shell/collection.js:388:14)
at Object.findOperations.updateOne (src/mongo/shell/bulk_api.js:675:20)
at DBCollection.update (src/mongo/shell/collection.js:455:22)
at Function.sh.addTagRange (src/mongo/shell/utils_sh.js:365:17)
at (shell):1:4 at src/mongo/shell/collection.js:157


Are embedded attributes supported?
And also we have other mapreduce collection which has shardkey defined on
"_id.x.y" in the similar pattern by using "\uff0e".


sh.addTagRange("RAW_DATA.OPERATIONAL_METRICS",{"operation\uff0ekey":"100"},{
"operation\uff0ekey":"199"}, "ShardA")

sh.addTagRange("RAW_DATA.OPERATIONAL_METRICS",{"operation\uff0ekey":"200"},{
"operation\uff0ekey":"299"}, "ShardB")

2015-11-30T18:29:31.306+0530 I SHARDING [Balancer] chunk { _id:
"TST.AUDIT-sourceKey_"100"", ns: "TST.AUDIT", min: { sourceKey: "100" },
max: { sourceKey: "200" }, version: Timestamp 1000|3, versionEpoch:
ObjectId('565c04d5bb810e444ecf4a9b'), lastmod: Timestamp 1000|3,
lastmodEpoch: ObjectId('565c04d5bb810e444ecf4a9b'), shard: "ShardB" } is
not on a shard with the right tag: ShardA
2015-11-30T18:29:31.306+0530 I SHARDING [Balancer] going to move to: ShardA
2015-11-30T18:29:31.308+0530 I - [Balancer] Assertion: 16634:field
names of bound { operation.key: "100" } do not match those of keyPattern {
operation.key: 1.0 }
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\stacktrace_win.cpp(175)
mongo::printStackTrace+0x43
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\log.cpp(134)
mongo::logContext+0x97
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\assert_util.cpp(219)
mongo::msgasserted+0xd7
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\assert_util.cpp(211)
mongo::msgasserted+0x13
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\db\keypattern.cpp(73)
mongo::KeyPattern::extendRangeBound+0x2f7
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\s\balance.cpp(473)
mongo::Balancer::_doBalanceRound+0x12eb
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\s\balance.cpp(648)
mongo::Balancer::run+0xd2c
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\background.cpp(153)
mongo::BackgroundJob::jobBody+0x148
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185)
boost::`anonymous namespace'::thread_start_function+0x21
2015-11-30T18:29:31.576+0530 I CONTROL [Balancer] MSVCR120.dll
beginthreadex+0x107
2015-11-30T18:29:31.576+0530 I CONTROL [Balancer] MSVCR120.dll
endthreadex+0x192
2015-11-30T18:29:31.576+0530 I CONTROL [Balancer] kernel32.dll

BaseThreadInitThunk+0xd

sh.addTagRange("RAW_DATA.OPERATIONAL_METRICS",{"operation\uff0ekey":"100"},{
"operation\uff0ekey":"199"}, "ShardA")

sh.addTagRange("RAW_DATA.OPERATIONAL_METRICS",{"operation\uff0ekey":"200"},{
"operation\uff0ekey":"299"}, "ShardB")
Post by Naveen
One of my other collection is having nested attribute and it's failing.
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5657dabcaf958ce7a4072901")
}
{ "_id" : "ShardA", "host" : "rs1/host1:27020,host2:27017",
"tags" : [ "ShardA" ] }
{ "_id" : "ShardB", "host" : "rs2/host1:27017,host2:27020",
"tags" : [ "ShardB" ] }
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 5
Last reported error: field names of bound { operation.key: "100"
} do not match those of keyPattern { operation.key: 1.0 }
Time of Reported error: Mon Nov 30 2015 10:12:50 GMT+0530 (India
Standard Time)
No recent migrations
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "TST", "partitioned" : true, "primary" : "ShardB" }
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardA 2
ShardB 1
{ "sourceKey" : { "$minKey" : 1 } } -->> {
"sourceKey" : "100" } on : ShardB Timestamp(2, 1)
"200"} on : ShardA Timestamp(2, 2)
{ "sourceKey" : "200" } -->> { "sourceKey" : {
"$maxKey" : 1 } } on : ShardA Timestamp(2, 3)
tag: ShardA { "sourceKey" : "100" } -->> { "sourceKey" : "199" }
tag: ShardB { "sourceKey" : "200" } -->> { "sourceKey" : "299" }
"ShardB" }
RAW_DATA.OPERATIONAL_METRICS
shard key: { "operation.key" : 1 }
ShardA 1
ShardB 3
{ "operation.key" : { "$minKey" : 1 } } -->> {
"operation.key" : "105" } on : ShardB Timestamp(2, 1)
{ "operation.key" : "105" } -->> { "operation.key"
: "205" } on : ShardB Timestamp(1, 2)
{ "operation.key" : "205" } -->> { "operation.key"
: "215" } on : ShardB Timestamp(1, 3)
{ "operation.key" : "215" } -->> { "operation.key"
: { "$maxKey" : 1 } } on : ShardA Timestamp(2, 0)
tag: ShardA { "operation.key" : "100" } -->> {
"operation.key" : "199" }
tag: ShardB { "operation.key" : "200" } -->> {
"operation.key" : "299" }
Thanks
The splits ARE honoring your tagging - the reason the split exists on
value "200" is because it's where your tagged ranges split.
Is there a reason you don't want to look at the results of the balancer's
attempts to balance? It's clear to me that they must be failing. Without
seeing the error, I can't tell you *why* they are failing.
sh.status() shows the last five balancer attempt results.
Logs will have errors for all balancer attempts.
Asya
sh.getBalancerState()
true
I've also inserted new record and started balancer manually to see the
impact, with no useful impact on shard distribution.
Noticed that chunk splits are not honoring my tag ranges.
Though my tag range for ShardB is between "200" and "299", actual chunk
split is showing '1" to "100" for ShardB.
Highlighted them in the below section. Just to inform you my shard key is
of String type.
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardB 1
ShardA 2
ShardB Timestamp(2000, 1)
{ "sourceKey" : "100" } -->> { "sourceKey" : "200" } on : ShardA
Timestamp(2000, 2)
ShardA Timestamp(2000, 3)
tag: ShardA { "sourceKey" : "100" } -->> { "sourceKey" : "199" }
tag:* ShardB { "sourceKey" : "200" } -->> { "sourceKey" : "299" }*
Thanks
Actually I think I see the answer to my question (you did say Mongodb
3.0.3 so I'm not sure why the balancer information is not there).
But I can tell you that distribution tells me that there is definitely a
problem with your metadata - there should have been chunks splits happening
estimated data per chunk : 99.65Mb
I think you should check your logs and see if all config servers are up
and if the metadata can successfully be changed. Output of balancer state
would definitely help out here.
Asya
What version is this? In the latest, sh.status() outputs whether the
balancer is running, when it last balanced successfully, etc.
I see none of that information. Could you provide exact server version
for your shards/mongos/configservers?
100->200 (not-inclusive)
and 200->maxkey are separate chunks, but 200+ chunk has not been moved to
ShardB and I suspect it's because the balancer either isn't running or
isn't able to successfully move the chunk to the other shard.
Asya
Thanks Asya.
Here are the status, shard distribution and the query results to show that
keys fall in both tag ranges.
Chunk difference between shards is just 1.
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5657dabcaf958ce7a4072901")
}
{ "_id" : "ShardA", "host" : "rs1/host1:27020,host2:27017", "tags" : [
"ShardA" ] }
{ "_id" : "ShardB", "host" : "rs2/host1:27017,host2:27020", "tags" : [
"ShardB" ] }
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "TST", "partitioned" : true, "primary" : "ShardB" }
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardB 1
ShardA 2
ShardB Timestamp(2000, 1)
{ "sourceKey" : "100" } -->> { "sourceKey" : "200" } on : ShardA
Timestamp(2000, 2)
ShardA Timestamp(2000, 3)
tag: ShardA { "sourceKey" : "100" } -->> { "sourceKey" : "199" }
tag: ShardB { "sourceKey" : "200" } -->> { "sourceKey" : "299" }
# Collection data
db.AUDIT.distinct("sourceKey")
{
"0" : "120",
"1" : "215"
}
db.AUDIT.getShardDistribution()
Shard ShardA at rs1/host1:27020,host2:27017
data : 199.31Mb docs : 51224 chunks : 2
estimated data per chunk : 99.65Mb
estimated docs per chunk : 25612
Shard ShardB at rs2/host1:27017,host2:27020
data : 0b docs : 0 chunks : 1
estimated data per chunk : 0b
estimated docs per chunk : 0
Totals
data : 199.31Mb docs : 51224 chunks : 3
Shard ShardA contains 100% data, 100% docs in cluster, avg obj size on
shard : 3kb
Shard ShardB contains 0% data, 0% docs in cluster, avg obj size on shard
: NaNGb
The router will honor tagging rules and you don't need to change chunk
size or anything else. However how did you sent the tags on shards?
In general, it's helpful to show output to sh.status() so we can see all
relevant info like shard tags, whether balancer is enabled, etc.
Thanks Chris.
Had a hunch on this, but I thought that router will honor the tagging
rules, if I'm starting with new collections.
Will try it.
Off the top of my head I would say that you need to insert more data to
force the balancer to kick in.
You can lower the chunksize (
https://docs.mongodb.org/manual/reference/configuration-options/#sharding.chunkSize)
so this will happen faster, for testing purpose only of course .
Give it a try, and let us now
Kind regards
Chris
Trying to a multi regional sharded replicaset cluster with Mongodb 3.0.3.
I've a shard key called "sourceKey" which is of type "String".
...
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/4b122473-e124-467d-a0c0-b24deddc40c0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Naveen
2015-11-30 17:32:10 UTC
Permalink
I see the following existing JIRA's.

Embedded "dot-notation" fields broken for tag based balancing
<https://jira.mongodb.org/browse/SERVER-6999>

Can't add tag ranges when sharding on subdocument fields
<https://jira.mongodb.org/browse/SERVER-13690>
Post by Naveen
Asya,
Tried with latest mongodb version 3.0.7
I've observed the below error while the balancer was actually trying to
move the chunks, it is failing on the other collection which has embedded
attribute
"operation.key".
field names of bound { operation.key: "100" } do not match those of
keyPattern { operation.key: 1.0 }
Problem could be that RAW_DATA.OPERATIONAL_METRICS collection's shard key
was created in the following manner, as it was not working with the "dot",
we have used "\uff0e" while adding tag ranges.
When we used embedded attribute we see the following error. To avoid this
we have used "\uff0e", that seems the problem.
"100"},{"operation.key":"200"}, "ShardA")
2015-11-30T08:38:36.084-0800 E QUERY Error: can't have . in field
names [operation.Key]
at Error (<anonymous>)
at DBCollection._validateForStorage
(src/mongo/shell/collection.js:157:19)
at DBCollection._validateForStorage
(src/mongo/shell/collection.js:165:18)
at DBCollection._validateForStorage
(src/mongo/shell/collection.js:165:18)
at DBCollection._validateUpdateDoc
(src/mongo/shell/collection.js:388:14)
at Object.findOperations.updateOne (src/mongo/shell/bulk_api.js:675:20)
at DBCollection.update (src/mongo/shell/collection.js:455:22)
at Function.sh.addTagRange (src/mongo/shell/utils_sh.js:365:17)
at (shell):1:4 at src/mongo/shell/collection.js:157
Are embedded attributes supported?
And also we have other mapreduce collection which has shardkey defined on
"_id.x.y" in the similar pattern by using "\uff0e".
sh.addTagRange("RAW_DATA.OPERATIONAL_METRICS",{"operation\uff0ekey":"100"
},{"operation\uff0ekey":"199"}, "ShardA")
sh.addTagRange("RAW_DATA.OPERATIONAL_METRICS",{"operation\uff0ekey":"200"
},{"operation\uff0ekey":"299"}, "ShardB")
"TST.AUDIT-sourceKey_"100"", ns: "TST.AUDIT", min: { sourceKey: "100" },
ObjectId('565c04d5bb810e444ecf4a9b'), lastmod: Timestamp 1000|3,
lastmodEpoch: ObjectId('565c04d5bb810e444ecf4a9b'), shard: "ShardB" } is
not on a shard with the right tag: ShardA
2015-11-30T18:29:31.306+0530 I SHARDING [Balancer] going to move to: ShardA
2015-11-30T18:29:31.308+0530 I - [Balancer] Assertion: 16634:field
names of bound { operation.key: "100" } do not match those of keyPattern {
operation.key: 1.0 }
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\stacktrace_win.cpp(175)
mongo::printStackTrace+0x43
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\log.cpp(134)
mongo::logContext+0x97
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\assert_util.cpp(219)
mongo::msgasserted+0xd7
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\assert_util.cpp(211)
mongo::msgasserted+0x13
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\db\keypattern.cpp(73)
mongo::KeyPattern::extendRangeBound+0x2f7
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\s\balance.cpp(473)
mongo::Balancer::_doBalanceRound+0x12eb
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\s\balance.cpp(648)
mongo::Balancer::run+0xd2c
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\background.cpp(153)
mongo::BackgroundJob::jobBody+0x148
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185)
boost::`anonymous namespace'::thread_start_function+0x21
2015-11-30T18:29:31.576+0530 I CONTROL [Balancer] MSVCR120.dll
beginthreadex+0x107
2015-11-30T18:29:31.576+0530 I CONTROL [Balancer] MSVCR120.dll
endthreadex+0x192
2015-11-30T18:29:31.576+0530 I CONTROL [Balancer] kernel32.dll
BaseThreadInitThunk+0xd
sh.addTagRange("RAW_DATA.OPERATIONAL_METRICS",{"operation\uff0ekey":"100"
},{"operation\uff0ekey":"199"}, "ShardA")
sh.addTagRange("RAW_DATA.OPERATIONAL_METRICS",{"operation\uff0ekey":"200"
},{"operation\uff0ekey":"299"}, "ShardB")
One of my other collection is having nested attribute and it's failing.
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5657dabcaf958ce7a4072901")
}
{ "_id" : "ShardA", "host" : "rs1/host1:27020,host2:27017",
"tags" : [ "ShardA" ] }
{ "_id" : "ShardB", "host" : "rs2/host1:27017,host2:27020",
"tags" : [ "ShardB" ] }
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 5
Last reported error: field names of bound { operation.key: "100"
} do not match those of keyPattern { operation.key: 1.0 }
Time of Reported error: Mon Nov 30 2015 10:12:50 GMT+0530 (India
Standard Time)
No recent migrations
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "TST", "partitioned" : true, "primary" : "ShardB" }
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardA 2
ShardB 1
{ "sourceKey" : { "$minKey" : 1 } } -->> {
"sourceKey" : "100" } on : ShardB Timestamp(2, 1)
"200"} on : ShardA Timestamp(2, 2)
{ "sourceKey" : "200" } -->> { "sourceKey" : {
"$maxKey" : 1 } } on : ShardA Timestamp(2, 3)
tag: ShardA { "sourceKey" : "100" } -->> { "sourceKey" : "199" }
tag: ShardB { "sourceKey" : "200" } -->> { "sourceKey" : "299" }
"ShardB" }
RAW_DATA.OPERATIONAL_METRICS
shard key: { "operation.key" : 1 }
ShardA 1
ShardB 3
{ "operation.key" : { "$minKey" : 1 } } -->> {
"operation.key" : "105" } on : ShardB Timestamp(2, 1)
{ "operation.key" : "105" } -->> { "operation.key"
: "205" } on : ShardB Timestamp(1, 2)
{ "operation.key" : "205" } -->> { "operation.key"
: "215" } on : ShardB Timestamp(1, 3)
{ "operation.key" : "215" } -->> { "operation.key"
: { "$maxKey" : 1 } } on : ShardA Timestamp(2, 0)
tag: ShardA { "operation.key" : "100" } -->> {
"operation.key" : "199" }
tag: ShardB { "operation.key" : "200" } -->> {
"operation.key" : "299" }
Thanks
The splits ARE honoring your tagging - the reason the split exists on
value "200" is because it's where your tagged ranges split.
Is there a reason you don't want to look at the results of the balancer's
attempts to balance? It's clear to me that they must be failing. Without
seeing the error, I can't tell you *why* they are failing.
sh.status() shows the last five balancer attempt results.
Logs will have errors for all balancer attempts.
Asya
sh.getBalancerState()
true
I've also inserted new record and started balancer manually to see the
impact, with no useful impact on shard distribution.
Noticed that chunk splits are not honoring my tag ranges.
Though my tag range for ShardB is between "200" and "299", actual chunk
split is showing '1" to "100" for ShardB.
Highlighted them in the below section. Just to inform you my shard key is
of String type.
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardB 1
ShardA 2
ShardB Timestamp(2000, 1)
{ "sourceKey" : "100" } -->> { "sourceKey" : "200" } on : ShardA
Timestamp(2000, 2)
ShardA Timestamp(2000, 3)
tag: ShardA { "sourceKey" : "100" } -->> { "sourceKey" : "199" }
tag:* ShardB { "sourceKey" : "200" } -->> { "sourceKey" : "299" }*
Thanks
Actually I think I see the answer to my question (you did say Mongodb
3.0.3 so I'm not sure why the balancer information is not there).
But I can tell you that distribution tells me that there is definitely a
problem with your metadata - there should have been chunks splits happening
estimated data per chunk : 99.65Mb
I think you should check your logs and see if all config servers are up
and if the metadata can successfully be changed. Output of balancer state
would definitely help out here.
Asya
What version is this? In the latest, sh.status() outputs whether the
balancer is running, when it last balanced successfully, etc.
I see none of that information. Could you provide exact server version
for your shards/mongos/configservers?
100->200 (not-inclusive)
and 200->maxkey are separate chunks, but 200+ chunk has not been moved to
ShardB and I suspect it's because the balancer either isn't running or
isn't able to successfully move the chunk to the other shard.
Asya
Thanks Asya.
Here are the status, shard distribution and the query results to show that
keys fall in both tag ranges.
Chunk difference between shards is just 1.
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5657dabcaf958ce7a4072901")
}
{ "_id" : "ShardA", "host" : "rs1/host1:27020,host2:27017", "tags" : [
"ShardA" ] }
{ "_id" : "ShardB", "host" : "rs2/host1:27017,host2:27020", "tags" : [
"ShardB" ] }
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "TST", "partitioned" : true, "primary" : "ShardB" }
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardB 1
ShardA 2
{ "sourceKey" : { $minKey : 1 } } --
...
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/48039ff9-06d8-4364-bc39-c163be659b14%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-11-30 18:49:48 UTC
Permalink
And note that 6999 documents a workaround how to specify tag ranges.

Naveen: you might want to backtrack and remove your collection that's
messed up.

Asya
Post by Naveen
I see the following existing JIRA's.
Embedded "dot-notation" fields broken for tag based balancing
<https://jira.mongodb.org/browse/SERVER-6999>
Can't add tag ranges when sharding on subdocument fields
<https://jira.mongodb.org/browse/SERVER-13690>
Post by Naveen
Asya,
Tried with latest mongodb version 3.0.7
I've observed the below error while the balancer was actually trying to
move the chunks, it is failing on the other collection which has embedded
attribute
"operation.key".
field names of bound { operation.key: "100" } do not match those of
keyPattern { operation.key: 1.0 }
Problem could be that RAW_DATA.OPERATIONAL_METRICS collection's shard
key was created in the following manner, as it was not working with the "dot",
we have used "\uff0e" while adding tag ranges.
When we used embedded attribute we see the following error. To avoid this
we have used "\uff0e", that seems the problem.
"100"},{"operation.key":"200"}, "ShardA")
2015-11-30T08:38:36.084-0800 E QUERY Error: can't have . in field
names [operation.Key]
at Error (<anonymous>)
at DBCollection._validateForStorage
(src/mongo/shell/collection.js:157:19)
at DBCollection._validateForStorage
(src/mongo/shell/collection.js:165:18)
at DBCollection._validateForStorage
(src/mongo/shell/collection.js:165:18)
at DBCollection._validateUpdateDoc
(src/mongo/shell/collection.js:388:14)
at Object.findOperations.updateOne
(src/mongo/shell/bulk_api.js:675:20)
at DBCollection.update (src/mongo/shell/collection.js:455:22)
at Function.sh.addTagRange (src/mongo/shell/utils_sh.js:365:17)
at (shell):1:4 at src/mongo/shell/collection.js:157
Are embedded attributes supported?
And also we have other mapreduce collection which has shardkey defined on
"_id.x.y" in the similar pattern by using "\uff0e".
sh.addTagRange("RAW_DATA.OPERATIONAL_METRICS",{"operation\uff0ekey":"100"
},{"operation\uff0ekey":"199"}, "ShardA")
sh.addTagRange("RAW_DATA.OPERATIONAL_METRICS",{"operation\uff0ekey":"200"
},{"operation\uff0ekey":"299"}, "ShardB")
"TST.AUDIT-sourceKey_"100"", ns: "TST.AUDIT", min: { sourceKey: "100" },
ObjectId('565c04d5bb810e444ecf4a9b'), lastmod: Timestamp 1000|3,
lastmodEpoch: ObjectId('565c04d5bb810e444ecf4a9b'), shard: "ShardB" } is
not on a shard with the right tag: ShardA
2015-11-30T18:29:31.306+0530 I SHARDING [Balancer] going to move to: ShardA
2015-11-30T18:29:31.308+0530 I - [Balancer] Assertion: 16634:field
names of bound { operation.key: "100" } do not match those of keyPattern {
operation.key: 1.0 }
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\stacktrace_win.cpp(175)
mongo::printStackTrace+0x43
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\log.cpp(134)
mongo::logContext+0x97
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\assert_util.cpp(219)
mongo::msgasserted+0xd7
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\assert_util.cpp(211)
mongo::msgasserted+0x13
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\db\keypattern.cpp(73)
mongo::KeyPattern::extendRangeBound+0x2f7
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\s\balance.cpp(473)
mongo::Balancer::_doBalanceRound+0x12eb
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\s\balance.cpp(648)
mongo::Balancer::run+0xd2c
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\background.cpp(153)
mongo::BackgroundJob::jobBody+0x148
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185)
boost::`anonymous namespace'::thread_start_function+0x21
2015-11-30T18:29:31.576+0530 I CONTROL [Balancer] MSVCR120.dll
beginthreadex+0x107
2015-11-30T18:29:31.576+0530 I CONTROL [Balancer] MSVCR120.dll
endthreadex+0x192
2015-11-30T18:29:31.576+0530 I CONTROL [Balancer] kernel32.dll
BaseThreadInitThunk+0xd
sh.addTagRange("RAW_DATA.OPERATIONAL_METRICS",{"operation\uff0ekey":"100"
},{"operation\uff0ekey":"199"}, "ShardA")
sh.addTagRange("RAW_DATA.OPERATIONAL_METRICS",{"operation\uff0ekey":"200"
},{"operation\uff0ekey":"299"}, "ShardB")
One of my other collection is having nested attribute and it's failing.
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5657dabcaf958ce7a4072901")
}
{ "_id" : "ShardA", "host" : "rs1/host1:27020,host2:27017",
"tags" : [ "ShardA" ] }
{ "_id" : "ShardB", "host" : "rs2/host1:27017,host2:27020",
"tags" : [ "ShardB" ] }
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 5
Last reported error: field names of bound { operation.key: "100"
} do not match those of keyPattern { operation.key: 1.0 }
Time of Reported error: Mon Nov 30 2015 10:12:50 GMT+0530 (India
Standard Time)
No recent migrations
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "TST", "partitioned" : true, "primary" : "ShardB" }
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardA 2
ShardB 1
{ "sourceKey" : { "$minKey" : 1 } } -->> {
"sourceKey" : "100" } on : ShardB Timestamp(2, 1)
"200"} on : ShardA Timestamp(2, 2)
{ "sourceKey" : "200" } -->> { "sourceKey" : {
"$maxKey" : 1 } } on : ShardA Timestamp(2, 3)
tag: ShardA { "sourceKey" : "100" } -->> {
"sourceKey" : "199" }
tag: ShardB { "sourceKey" : "200" } -->> {
"sourceKey" : "299" }
"ShardB" }
RAW_DATA.OPERATIONAL_METRICS
shard key: { "operation.key" : 1 }
ShardA 1
ShardB 3
{ "operation.key" : { "$minKey" : 1 } } -->> {
"operation.key" : "105" } on : ShardB Timestamp(2, 1)
{ "operation.key" : "105" } -->> {
"operation.key" : "205" } on : ShardB Timestamp(1, 2)
{ "operation.key" : "205" } -->> {
"operation.key" : "215" } on : ShardB Timestamp(1, 3)
{ "operation.key" : "215" } -->> {
"operation.key" : { "$maxKey" : 1 } } on : ShardA Timestamp(2, 0)
tag: ShardA { "operation.key" : "100" } -->> {
"operation.key" : "199" }
tag: ShardB { "operation.key" : "200" } -->> {
"operation.key" : "299" }
Thanks
The splits ARE honoring your tagging - the reason the split exists on
value "200" is because it's where your tagged ranges split.
Is there a reason you don't want to look at the results of the balancer's
attempts to balance? It's clear to me that they must be failing. Without
seeing the error, I can't tell you *why* they are failing.
sh.status() shows the last five balancer attempt results.
Logs will have errors for all balancer attempts.
Asya
sh.getBalancerState()
true
I've also inserted new record and started balancer manually to see the
impact, with no useful impact on shard distribution.
Noticed that chunk splits are not honoring my tag ranges.
Though my tag range for ShardB is between "200" and "299", actual chunk
split is showing '1" to "100" for ShardB.
Highlighted them in the below section. Just to inform you my shard key is
of String type.
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardB 1
ShardA 2
ShardB Timestamp(2000, 1)
{ "sourceKey" : "100" } -->> { "sourceKey" : "200" } on : ShardA
Timestamp(2000, 2)
ShardA Timestamp(2000, 3)
tag: ShardA { "sourceKey" : "100" } -->> { "sourceKey" : "199" }
tag:* ShardB { "sourceKey" : "200" } -->> { "sourceKey" : "299" }*
Thanks
Actually I think I see the answer to my question (you did say Mongodb
3.0.3 so I'm not sure why the balancer information is not there).
But I can tell you that distribution tells me that there is definitely a
problem with your metadata - there should have been chunks splits happening
estimated data per chunk : 99.65Mb
I think you should check your logs and see if all config servers are up
and if the metadata can successfully be changed. Output of balancer state
would definitely help out here.
Asya
What version is this? In the latest, sh.status() outputs whether the
balancer is running, when it last balanced successfully, etc.
I see none of that information. Could you provide exact server version
for your shards/mongos/configservers?
100->200 (not-inclusive)
and 200->maxkey are separate chunks, but 200+ chunk has not been moved to
ShardB and I suspect it's because the balancer either isn't running or
isn't able to successfully move the chunk to the other shard.
Asya
Thanks Asya.
Here are the status, shard distribution and the query results to show
that keys fall in both tag ranges.
Chunk difference between shards is just 1.
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5657dabcaf958ce7a4072901")
}
[ "ShardA" ] }
[ "ShardB" ] }
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "TST", "partitioned" : true, "primary" : "ShardB" }
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardB 1
ShardA 2
{ "sourceKey" : { $minKey : 1 } } --
...
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/48039ff9-06d8-4364-bc39-c163be659b14%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/48039ff9-06d8-4364-bc39-c163be659b14%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJDpLJTokmqf5TxQEnuLOysVbKC2ySRQU4-x-vuhtDLDwA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Naveen
2015-12-01 01:27:24 UTC
Permalink
Thanks Asya.

Already moved out nested attributes as shard keys to top level as that
wasn't working and we need to release the product.
However I need to use nested attributes as shard keys in mapreduce output
collection called "MR_OUT".

Shard distribution happens properly on input collection "MR_IN" as shard
keys are top level, however when MR jobs write the output they don't
distribute data to appropriate shards. Chunks are not getting split
properly. Is it supported, if not what is the suggested way.


# MapReduce Input collection

sh.enableSharding("MR_IN")
sh.shardCollection("MR_IN.TESTSCHEDULEA",{"containerkey":1})
sh.addTagRange("MR_IN.TESTSCHEDULEA",{"_id":{"containerkey":"Container1key"
}}, {"_id":{"containerkey":"Container1key"}}, "ShardA")
sh.addTagRange("MR_IN.TESTSCHEDULEA",{"_id":{"containerkey":"Container2key"
}}, {"_id":{"containerkey":"Container3key"}}, "ShardB")



# MapReduce Output collection
# Shard distribution fails and all the data goes into one shard

sh.enableSharding("MR_OUT")
sh.shardCollection("MR_OUT.TESTSCHEDULEA",{"_id.containerkey":1})
sh.addTagRange("MR_OUT.TESTSCHEDULEA",{"_id.containerkey":"Container1key"},
{"_id.containerkey":"Container2key"}, "ShardA")
sh.addTagRange("MR_OUT.TESTSCHEDULEA",{"_id.containerkey":"Container2key"},
{"_id.containerkey":"Container3key"}, "ShardB")


--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("565bca4978756d108db519c1")
}
shards:
{ "_id" : "ShardA", "host" : "rs1/localhost:11001,localhost:11002",
"tags" : [ "ShardA" ] }
{ "_id" : "ShardB", "host" : "rs2/localhost:11004,localhost:11005",
"tags" : [ "ShardB" ] }
databases:
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "MR_OUT", "partitioned" : true, "primary" : "ShardB" }
MR_OUT.TESTSCHEDULEA
shard key: { "_id.containerkey" : 1 }
chunks:
ShardB 1
{ "_id.containerkey" : { $minKey : 1 } } -->> { "_id.containerkey" : {
$maxKey : 1 } } on : ShardB Timestamp(1000, 0)
tag: ShardA { "_id" : { "containerkey" : "Container1key" } } -->> { "_id"
: { "containerkey" : "Container2key" } }
tag: ShardB { "_id" : { "containerkey" : "Container2key" } } -->> { "_id"
: { "containerkey" : "Container3key" } }

{ "_id" : "MR_IN", "partitioned" : true, "primary" : "ShardB" }

MR_IN.TESTSCHEDULEA
shard key: { "containerkey" : 1 }
chunks:
ShardB 2
ShardA 1
{ "containerkey" : { $minKey : 1 } } -->> { "containerkey" :
"Container1key" } on : ShardB Timestamp(2000, 1)
{ "containerkey" : "Container1key" } -->> { "containerkey" :
"Container2key" } on : ShardA Timestamp(2000, 0)
{ "containerkey" : "Container2key" } -->> { "containerkey" : { $maxKey : 1
} } on : ShardB Timestamp(1000, 4)
tag: ShardA { "containerkey" : "Container1key" } -->> { "containerkey" :
"Container2key" }
tag: ShardB { "containerkey" : "Container2key" } -->> { "containerkey" :
"Container3key" }
Post by Asya Kamsky
And note that 6999 documents a workaround how to specify tag ranges.
Naveen: you might want to backtrack and remove your collection that's
messed up.
Asya
Post by Naveen
I see the following existing JIRA's.
Embedded "dot-notation" fields broken for tag based balancing
<https://jira.mongodb.org/browse/SERVER-6999>
Can't add tag ranges when sharding on subdocument fields
<https://jira.mongodb.org/browse/SERVER-13690>
Post by Naveen
Asya,
Tried with latest mongodb version 3.0.7
I've observed the below error while the balancer was actually trying to
move the chunks, it is failing on the other collection which has embedded
attribute
"operation.key".
field names of bound { operation.key: "100" } do not match those of
keyPattern { operation.key: 1.0 }
Problem could be that RAW_DATA.OPERATIONAL_METRICS collection's shard
key was created in the following manner, as it was not working with the "dot",
we have used "\uff0e" while adding tag ranges.
When we used embedded attribute we see the following error. To avoid
this we have used "\uff0e", that seems the problem.
"100"},{"operation.key":"200"}, "ShardA")
2015-11-30T08:38:36.084-0800 E QUERY Error: can't have . in field
names [operation.Key]
at Error (<anonymous>)
at DBCollection._validateForStorage
(src/mongo/shell/collection.js:157:19)
at DBCollection._validateForStorage
(src/mongo/shell/collection.js:165:18)
at DBCollection._validateForStorage
(src/mongo/shell/collection.js:165:18)
at DBCollection._validateUpdateDoc
(src/mongo/shell/collection.js:388:14)
at Object.findOperations.updateOne
(src/mongo/shell/bulk_api.js:675:20)
at DBCollection.update (src/mongo/shell/collection.js:455:22)
at Function.sh.addTagRange (src/mongo/shell/utils_sh.js:365:17)
at (shell):1:4 at src/mongo/shell/collection.js:157
Are embedded attributes supported?
And also we have other mapreduce collection which has shardkey defined
on "_id.x.y" in the similar pattern by using "\uff0e".
"100"},{"operation\uff0ekey":"199"}, "ShardA")
"200"},{"operation\uff0ekey":"299"}, "ShardB")
"TST.AUDIT-sourceKey_"100"", ns: "TST.AUDIT", min: { sourceKey: "100" },
ObjectId('565c04d5bb810e444ecf4a9b'), lastmod: Timestamp 1000|3,
lastmodEpoch: ObjectId('565c04d5bb810e444ecf4a9b'), shard: "ShardB" } is
not on a shard with the right tag: ShardA
2015-11-30T18:29:31.306+0530 I SHARDING [Balancer] going to move to: ShardA
16634:field names of bound { operation.key: "100" } do not match those of
keyPattern { operation.key: 1.0 }
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\stacktrace_win.cpp(175)
mongo::printStackTrace+0x43
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\log.cpp(134)
mongo::logContext+0x97
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\assert_util.cpp(219)
mongo::msgasserted+0xd7
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\assert_util.cpp(211)
mongo::msgasserted+0x13
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\db\keypattern.cpp(73)
mongo::KeyPattern::extendRangeBound+0x2f7
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\s\balance.cpp(473)
mongo::Balancer::_doBalanceRound+0x12eb
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\s\balance.cpp(648)
mongo::Balancer::run+0xd2c
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\mongo\util\background.cpp(153)
mongo::BackgroundJob::jobBody+0x148
2015-11-30T18:29:31.575+0530 I CONTROL [Balancer] mongos.exe
...\src\third_party\boost\libs\thread\src\win32\thread.cpp(185)
boost::`anonymous namespace'::thread_start_function+0x21
2015-11-30T18:29:31.576+0530 I CONTROL [Balancer] MSVCR120.dll
beginthreadex+0x107
2015-11-30T18:29:31.576+0530 I CONTROL [Balancer] MSVCR120.dll
endthreadex+0x192
2015-11-30T18:29:31.576+0530 I CONTROL [Balancer] kernel32.dll
BaseThreadInitThunk+0xd
"100"},{"operation\uff0ekey":"199"}, "ShardA")
"200"},{"operation\uff0ekey":"299"}, "ShardB")
One of my other collection is having nested attribute and it's failing.
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5657dabcaf958ce7a4072901")
}
{ "_id" : "ShardA", "host" : "rs1/host1:27020,host2:27017",
"tags" : [ "ShardA" ] }
{ "_id" : "ShardB", "host" : "rs2/host1:27017,host2:27020",
"tags" : [ "ShardB" ] }
Currently enabled: yes
Currently running: no
Failed balancer rounds in last 5 attempts: 5
"100" } do not match those of keyPattern { operation.key: 1.0 }
Time of Reported error: Mon Nov 30 2015 10:12:50 GMT+0530
(India Standard Time)
No recent migrations
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "TST", "partitioned" : true, "primary" : "ShardB" }
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardA 2
ShardB 1
{ "sourceKey" : { "$minKey" : 1 } } -->> {
"sourceKey" : "100" } on : ShardB Timestamp(2, 1)
"200"} on : ShardA Timestamp(2, 2)
{ "sourceKey" : "200" } -->> { "sourceKey" : {
"$maxKey" : 1 } } on : ShardA Timestamp(2, 3)
tag: ShardA { "sourceKey" : "100" } -->> {
"sourceKey" : "199" }
tag: ShardB { "sourceKey" : "200" } -->> {
"sourceKey" : "299" }
"ShardB" }
RAW_DATA.OPERATIONAL_METRICS
shard key: { "operation.key" : 1 }
ShardA 1
ShardB 3
{ "operation.key" : { "$minKey" : 1 } } -->> {
"operation.key" : "105" } on : ShardB Timestamp(2, 1)
{ "operation.key" : "105" } -->> {
"operation.key" : "205" } on : ShardB Timestamp(1, 2)
{ "operation.key" : "205" } -->> {
"operation.key" : "215" } on : ShardB Timestamp(1, 3)
{ "operation.key" : "215" } -->> {
"operation.key" : { "$maxKey" : 1 } } on : ShardA Timestamp(2, 0)
tag: ShardA { "operation.key" : "100" } -->> {
"operation.key" : "199" }
tag: ShardB { "operation.key" : "200" } -->> {
"operation.key" : "299" }
Thanks
The splits ARE honoring your tagging - the reason the split exists on
value "200" is because it's where your tagged ranges split.
Is there a reason you don't want to look at the results of the
balancer's attempts to balance? It's clear to me that they must be
failing. Without seeing the error, I can't tell you *why* they are failing.
sh.status() shows the last five balancer attempt results.
Logs will have errors for all balancer attempts.
Asya
sh.getBalancerState()
true
I've also inserted new record and started balancer manually to see the
impact, with no useful impact on shard distribution.
Noticed that chunk splits are not honoring my tag ranges.
Though my tag range for ShardB is between "200" and "299", actual chunk
split is showing '1" to "100" for ShardB.
Highlighted them in the below section. Just to inform you my shard key
is of String type.
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardB 1
ShardA 2
ShardB Timestamp(2000, 1)
{ "sourceKey" : "100" } -->> { "sourceKey" : "200" } on : ShardA
Timestamp(2000, 2)
ShardA Timestamp(2000, 3)
tag: ShardA { "sourceKey" : "100" } -->> { "sourceKey" : "199" }
tag:* ShardB { "sourceKey" : "200" } -->> { "sourceKey" : "299" }*
Thanks
Actually I think I see the answer to my question (you did say Mongodb
3.0.3 so I'm not sure why the balancer information is not there).
But I can tell you that distribution tells me that there is definitely a
problem with your metadata - there should have been chunks splits happening
estimated data per chunk : 99.65Mb
I think you should check your logs and see if all config servers are up
and if the metadata can successfully be changed. Output of balancer state
would definitely help out here.
Asya
What version is this? In the latest, sh.status() outputs whether the
balancer is running, when it last balanced successfully, etc.
I see none of that information. Could you provide exact server version
for your shards/mongos/configservers?
100->200 (not-inclusive)
and 200->maxkey are separate chunks, but 200+ chunk has not been moved
to ShardB and I suspect it's because the balancer either isn't running or
isn't able to successfully move the chunk to the other shard.
Asya
Thanks Asya.
Here are the status, shard distribution and the query results to show
that keys fall in both tag ranges.
Chunk difference between shards is just 1.
sh.status()
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("5657dabcaf958ce7a4072901")
}
[ "ShardA" ] }
[ "ShardB" ] }
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "TST", "partitioned" : true, "primary" : "ShardB" }
TST.AUDIT
shard key: { "sourceKey" : 1 }
ShardB 1
ShardA 2
{ "sourceKey" : { $minKey : 1 } } --
...
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/48039ff9-06d8-4364-bc39-c163be659b14%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/48039ff9-06d8-4364-bc39-c163be659b14%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/4b6a4e46-f2bb-4acd-bdcf-78f6697cb362%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Naveen
2015-12-01 06:56:21 UTC
Permalink
Asya,

I've fixed mapreduce output collection's shard key issue in the following
way, and it was able to the proper chunk distribution
along with appropriate sharding of the data.

sh.enableSharding("MR_OUT")
sh.shardCollection("MR_OUT.TESTSCHEDULEA",{"_id":1})
sh.addTagRange("MR_OUT.TESTSCHEDULEA",{"_id":{ "containerkey":
"Container1key"}}, {"_id":{ "containerkey":"Container2key"}}, "ShardA")
sh.addTagRange("MR_OUT.TESTSCHEDULEA",{"_id":{ "containerkey":
"Container2key"}}, {"_id":{ "containerkey":"Container3key"}}, "ShardB")
Post by Naveen
Thanks Asya.
Already moved out nested attributes as shard keys to top level as that
wasn't working and we need to release the product.
However I need to use nested attributes as shard keys in mapreduce output
collection called "MR_OUT".
Shard distribution happens properly on input collection "MR_IN" as shard
keys are top level, however when MR jobs write the output they don't
distribute data to appropriate shards. Chunks are not getting split
properly. Is it supported, if not what is the suggested way.
# MapReduce Input collection
sh.enableSharding("MR_IN")
sh.shardCollection("MR_IN.TESTSCHEDULEA",{"containerkey":1})
"Container1key"}}, {"_id":{"containerkey":"Container1key"}}, "ShardA")
"Container2key"}}, {"_id":{"containerkey":"Container3key"}}, "ShardB")
# MapReduce Output collection
# Shard distribution fails and all the data goes into one shard
sh.enableSharding("MR_OUT")
sh.shardCollection("MR_OUT.TESTSCHEDULEA",{"_id.containerkey":1})
sh.addTagRange("MR_OUT.TESTSCHEDULEA",{"_id.containerkey":"Container1key"
}, {"_id.containerkey":"Container2key"}, "ShardA")
sh.addTagRange("MR_OUT.TESTSCHEDULEA",{"_id.containerkey":"Container2key"
}, {"_id.containerkey":"Container3key"}, "ShardB")
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("565bca4978756d108db519c1")
}
{ "_id" : "ShardA", "host" : "rs1/localhost:11001,localhost:11002",
"tags" : [ "ShardA" ] }
{ "_id" : "ShardB", "host" : "rs2/localhost:11004,localhost:11005",
"tags" : [ "ShardB" ] }
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "MR_OUT", "partitioned" : true, "primary" : "ShardB" }
MR_OUT.TESTSCHEDULEA
shard key: { "_id.containerkey" : 1 }
ShardB 1
{ "_id.containerkey" : { $minKey : 1 } } -->> { "_id.containerkey" : {
$maxKey : 1 } } on : ShardB Timestamp(1000, 0)
tag: ShardA { "_id" : { "containerkey" : "Container1key" } } -->> {
"_id" : { "containerkey" : "Container2key" } }
tag: ShardB { "_id" : { "containerkey" : "Container2key" } } -->> {
"_id" : { "containerkey" : "Container3key" } }
{ "_id" : "MR_IN", "partitioned" : true, "primary" : "ShardB" }
MR_IN.TESTSCHEDULEA
shard key: { "containerkey" : 1 }
ShardB 2
ShardA 1
"Container1key" } on : ShardB Timestamp(2000, 1)
"Container2key" } on : ShardA Timestamp(2000, 0)
...
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/fab9035e-41f3-4bfd-8011-fde9401ffb93%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-12-01 14:23:13 UTC
Permalink
Sorry, these shard keys don't seem to make any sense to me. Are they
actual meaningful values in your documents you query by? What's a
containerkey?
Post by Naveen
Asya,
I've fixed mapreduce output collection's shard key issue in the following
way, and it was able to the proper chunk distribution
along with appropriate sharding of the data.
sh.enableSharding("MR_OUT")
sh.shardCollection("MR_OUT.TESTSCHEDULEA",{"_id":1})
"Container1key"}}, {"_id":{ "containerkey":"Container2key"}}, "ShardA")
"Container2key"}}, {"_id":{ "containerkey":"Container3key"}}, "ShardB")
Post by Naveen
Thanks Asya.
Already moved out nested attributes as shard keys to top level as that
wasn't working and we need to release the product.
However I need to use nested attributes as shard keys in mapreduce output
collection called "MR_OUT".
Shard distribution happens properly on input collection "MR_IN" as shard
keys are top level, however when MR jobs write the output they don't
distribute data to appropriate shards. Chunks are not getting split
properly. Is it supported, if not what is the suggested way.
# MapReduce Input collection
sh.enableSharding("MR_IN")
sh.shardCollection("MR_IN.TESTSCHEDULEA",{"containerkey":1})
"Container1key"}}, {"_id":{"containerkey":"Container1key"}}, "ShardA")
"Container2key"}}, {"_id":{"containerkey":"Container3key"}}, "ShardB")
# MapReduce Output collection
# Shard distribution fails and all the data goes into one shard
sh.enableSharding("MR_OUT")
sh.shardCollection("MR_OUT.TESTSCHEDULEA",{"_id.containerkey":1})
sh.addTagRange("MR_OUT.TESTSCHEDULEA",{"_id.containerkey":"Container1key"
}, {"_id.containerkey":"Container2key"}, "ShardA")
sh.addTagRange("MR_OUT.TESTSCHEDULEA",{"_id.containerkey":"Container2key"
}, {"_id.containerkey":"Container3key"}, "ShardB")
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("565bca4978756d108db519c1")
}
{ "_id" : "ShardA", "host" : "rs1/localhost:11001,localhost:11002",
"tags" : [ "ShardA" ] }
{ "_id" : "ShardB", "host" : "rs2/localhost:11004,localhost:11005",
"tags" : [ "ShardB" ] }
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "MR_OUT", "partitioned" : true, "primary" : "ShardB" }
MR_OUT.TESTSCHEDULEA
shard key: { "_id.containerkey" : 1 }
ShardB 1
{ "_id.containerkey" : { $minKey : 1 } } -->> { "_id.containerkey" : {
$maxKey : 1 } } on : ShardB Timestamp(1000, 0)
tag: ShardA { "_id" : { "containerkey" : "Container1key" } } -->> {
"_id" : { "containerkey" : "Container2key" } }
tag: ShardB { "_id" : { "containerkey" : "Container2key" } } -->> {
"_id" : { "containerkey" : "Container3key" } }
{ "_id" : "MR_IN", "partitioned" : true, "primary" : "ShardB" }
MR_IN.TESTSCHEDULEA
shard key: { "containerkey" : 1 }
ShardB 2
ShardA 1
"Container1key" } on : ShardB Timestamp(2000, 1)
"Container2key" } on : ShardA Timestamp(2000, 0)
...
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/fab9035e-41f3-4bfd-8011-fde9401ffb93%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/fab9035e-41f3-4bfd-8011-fde9401ffb93%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJA2WDpHpasno%3DMvHNKWi3wZAbLbWE_F%2BeCj_PgKK5eLBQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Naveen
2015-12-02 00:55:52 UTC
Permalink
These are just test collections and keys.
Post by Asya Kamsky
Sorry, these shard keys don't seem to make any sense to me. Are they
actual meaningful values in your documents you query by? What's a
containerkey?
Post by Naveen
Asya,
I've fixed mapreduce output collection's shard key issue in the following
way, and it was able to the proper chunk distribution
along with appropriate sharding of the data.
sh.enableSharding("MR_OUT")
sh.shardCollection("MR_OUT.TESTSCHEDULEA",{"_id":1})
"Container1key"}}, {"_id":{ "containerkey":"Container2key"}}, "ShardA")
"Container2key"}}, {"_id":{ "containerkey":"Container3key"}}, "ShardB")
Post by Naveen
Thanks Asya.
Already moved out nested attributes as shard keys to top level as that
wasn't working and we need to release the product.
However I need to use nested attributes as shard keys in mapreduce
output collection called "MR_OUT".
Shard distribution happens properly on input collection "MR_IN" as shard
keys are top level, however when MR jobs write the output they don't
distribute data to appropriate shards. Chunks are not getting split
properly. Is it supported, if not what is the suggested way.
# MapReduce Input collection
sh.enableSharding("MR_IN")
sh.shardCollection("MR_IN.TESTSCHEDULEA",{"containerkey":1})
"Container1key"}}, {"_id":{"containerkey":"Container1key"}}, "ShardA")
"Container2key"}}, {"_id":{"containerkey":"Container3key"}}, "ShardB")
# MapReduce Output collection
# Shard distribution fails and all the data goes into one shard
sh.enableSharding("MR_OUT")
sh.shardCollection("MR_OUT.TESTSCHEDULEA",{"_id.containerkey":1})
"Container1key"}, {"_id.containerkey":"Container2key"}, "ShardA")
"Container2key"}, {"_id.containerkey":"Container3key"}, "ShardB")
--- Sharding Status ---
sharding version: {
"_id" : 1,
"minCompatibleVersion" : 5,
"currentVersion" : 6,
"clusterId" : ObjectId("565bca4978756d108db519c1")
}
{ "_id" : "ShardA", "host" : "rs1/localhost:11001,localhost:11002",
"tags" : [ "ShardA" ] }
{ "_id" : "ShardB", "host" : "rs2/localhost:11004,localhost:11005",
"tags" : [ "ShardB" ] }
{ "_id" : "admin", "partitioned" : false, "primary" : "config" }
{ "_id" : "MR_OUT", "partitioned" : true, "primary" : "ShardB" }
MR_OUT.TESTSCHEDULEA
shard key: { "_id.containerkey" : 1 }
ShardB 1
{ "_id.containerkey" : { $minKey : 1 } } -->> { "_id.containerkey" : {
$maxKey : 1 } } on : ShardB Timestamp(1000, 0)
tag: ShardA { "_id" : { "containerkey" : "Container1key" } } -->> {
"_id" : { "containerkey" : "Container2key" } }
tag: ShardB { "_id" : { "containerkey" : "Container2key" } } -->> {
"_id" : { "containerkey" : "Container3key" } }
{ "_id" : "MR_IN", "partitioned" : true, "primary" : "ShardB" }
MR_IN.TESTSCHEDULEA
shard key: { "containerkey" : 1 }
ShardB 2
ShardA 1
"Container1key" } on : ShardB Timestamp(2000, 1)
"Container2key" } on : ShardA Timestamp(2000, 0)
...
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/fab9035e-41f3-4bfd-8011-fde9401ffb93%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/fab9035e-41f3-4bfd-8011-fde9401ffb93%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
Free MongoDB Monitoring - cloud.mongodb.com
Free Online Education - university.mongodb.com
Get Involved - mongodb.org/community
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/b952d7bb-a334-48b4-be3d-5e1936316c5a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Continue reading on narkive:
Loading...