Discussion:
warning in log file
(too old to reply)
tetlika
2014-12-24 11:07:05 UTC
Permalink
suddenly we started having a lot of messages on log file of master on one
of shards:

Wed Dec 24 11:05:11.477 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.497 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.517 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.537 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.558 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.578 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.598 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.618 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.638 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.658 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.678 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.699 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.719 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.739 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.759 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.779 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.799 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.819 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.840 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.860 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.880 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.900 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.920 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.940 [migrateThread] warning: secondaries having hard
time keeping up with migrate

it continues to storm log file with those messages, but things look working
ok

what is that? how to fix that?


thanks
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/6ed25be2-e9de-4d4c-846a-08a553de6dd9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2014-12-29 12:47:00 UTC
Permalink
When you say things look okay, did you check secondary state? In
particular their replication lag?
Post by tetlika
suddenly we started having a lot of messages on log file of master on one
Wed Dec 24 11:05:11.477 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.497 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.517 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.537 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.558 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.578 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.598 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.618 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.638 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.658 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.678 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.699 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.719 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.739 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.759 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.779 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.799 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.819 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.840 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.860 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.880 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.900 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.920 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.940 [migrateThread] warning: secondaries having hard
time keeping up with migrate
it continues to storm log file with those messages, but things look
working ok
what is that? how to fix that?
thanks
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/6ed25be2-e9de-4d4c-846a-08a553de6dd9%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/6ed25be2-e9de-4d4c-846a-08a553de6dd9%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJC7gcdB-pvgPcc3NZm52ranCeMaJMubBu61io1DdE8cnA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
tetlika
2014-12-29 14:10:31 UTC
Permalink
it was because of addition of hidden secondaries with priority 0 , and they
were doing initial sync
Post by Asya Kamsky
When you say things look okay, did you check secondary state? In
particular their replication lag?
Post by tetlika
suddenly we started having a lot of messages on log file of master on one
Wed Dec 24 11:05:11.477 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.497 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.517 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.537 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.558 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.578 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.598 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.618 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.638 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.658 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.678 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.699 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.719 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.739 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.759 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.779 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.799 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.819 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.840 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.860 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.880 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.900 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.920 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.940 [migrateThread] warning: secondaries having hard
time keeping up with migrate
it continues to storm log file with those messages, but things look
working ok
what is that? how to fix that?
thanks
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/6ed25be2-e9de-4d4c-846a-08a553de6dd9%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/6ed25be2-e9de-4d4c-846a-08a553de6dd9%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/9296a15d-d5d6-4bab-8f4d-165c82c9deac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2014-12-29 18:57:27 UTC
Permalink
Ah, so now you increased the majority of the cluster, yes?

If you had 3 nodes before, then 2 are majority for voting and for
writeConcern majority. But if you added a 4th node your writeConcern
majority is now three. If you make sure you set the votes for hidden
secondaries to 0 then voting majority at least will go back to 2,
however, for writeConcern it will still be 3. Since migrations
always wait for replication to majority of the cluster before
committing the move, this can manifest as those messages you see (they
should stop when the nodes are all caught up).

Asya
Post by tetlika
it was because of addition of hidden secondaries with priority 0 , and they
were doing initial sync
Post by Asya Kamsky
When you say things look okay, did you check secondary state? In
particular their replication lag?
Post by tetlika
suddenly we started having a lot of messages on log file of master on one
Wed Dec 24 11:05:11.477 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.497 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.517 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.537 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.558 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.578 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.598 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.618 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.638 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.658 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.678 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.699 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.719 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.739 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.759 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.779 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.799 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.819 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.840 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.860 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.880 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.900 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.920 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.940 [migrateThread] warning: secondaries having hard
time keeping up with migrate
it continues to storm log file with those messages, but things look
working ok
what is that? how to fix that?
thanks
--
You received this message because you are subscribed to the Google Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/6ed25be2-e9de-4d4c-846a-08a553de6dd9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/9296a15d-d5d6-4bab-8f4d-165c82c9deac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJAJQidN0X0wnSORnjuimKW8KUN9B23N1c%2BmtwR4DqHo%3DQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
tetlika
2014-12-29 19:15:58 UTC
Permalink
thanks for reply Asya,

the picture was so:

1) we have 3 replicas in sharded cluster
2) each replica is 3 members replica set: 1 primary, 1 secondary and 1
arbiter
3) to each replica set it was added hidden member with zero priority
4) after 3 - I started seeing those errors messages

Finally I assumed that this happened because of adding new members so I
removed them with rs.remove("...") but the warning still continued to storm
the log file

just after I did stepdown on the replica set on which it was happening
things became ok and chunks migrated successfully
Post by Asya Kamsky
Ah, so now you increased the majority of the cluster, yes?
If you had 3 nodes before, then 2 are majority for voting and for
writeConcern majority. But if you added a 4th node your writeConcern
majority is now three. If you make sure you set the votes for hidden
secondaries to 0 then voting majority at least will go back to 2,
however, for writeConcern it will still be 3. Since migrations
always wait for replication to majority of the cluster before
committing the move, this can manifest as those messages you see (they
should stop when the nodes are all caught up).
Asya
Post by tetlika
it was because of addition of hidden secondaries with priority 0 , and
they
Post by tetlika
were doing initial sync
ППМеЎілПк, 29 груЎМя 2014 р. 14:47:32 UTC+2 кПрОстувач Asya Kamsky
Post by Asya Kamsky
When you say things look okay, did you check secondary state? In
particular their replication lag?
Post by tetlika
suddenly we started having a lot of messages on log file of master on
one
Post by tetlika
Post by Asya Kamsky
Post by tetlika
Wed Dec 24 11:05:11.477 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.497 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.517 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.537 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.558 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.578 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.598 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.618 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.638 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.658 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.678 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.699 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.719 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.739 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.759 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.779 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.799 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.819 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.840 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.860 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.880 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.900 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.920 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
Wed Dec 24 11:05:11.940 [migrateThread] warning: secondaries having
hard
Post by tetlika
Post by Asya Kamsky
Post by tetlika
time keeping up with migrate
it continues to storm log file with those messages, but things look
working ok
what is that? how to fix that?
thanks
--
You received this message because you are subscribed to the Google
Groups
Post by tetlika
Post by Asya Kamsky
Post by tetlika
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups
Post by tetlika
Post by Asya Kamsky
Post by tetlika
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
an
Post by tetlika
Post by Asya Kamsky
Post by tetlika
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/6ed25be2-e9de-4d4c-846a-08a553de6dd9%40googlegroups.com.
Post by tetlika
Post by Asya Kamsky
Post by tetlika
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups
Post by tetlika
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups
Post by tetlika
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
an
<javascript:>.
Post by tetlika
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/9296a15d-d5d6-4bab-8f4d-165c82c9deac%40googlegroups.com.
Post by tetlika
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/ad621ad7-89b5-4e6e-b791-e21b7e0ee6c0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2014-12-29 20:16:47 UTC
Permalink
In the future, consider doing this when you want to add a third data
holding (hidden) member:

seed that member with data files from a current secondary (follow the
process here: http://docs.mongodb.org/manual/tutorial/resync-replica-set-member/#sync-by-copying-data-files-from-another-member)

When you add this member, at the same time *remove the arbiter*.
That way you preserve three members in this replica set instead of
making it have four members.

Asya
Post by tetlika
thanks for reply Asya,
1) we have 3 replicas in sharded cluster
2) each replica is 3 members replica set: 1 primary, 1 secondary and 1
arbiter
3) to each replica set it was added hidden member with zero priority
4) after 3 - I started seeing those errors messages
Finally I assumed that this happened because of adding new members so I
removed them with rs.remove("...") but the warning still continued to storm
the log file
just after I did stepdown on the replica set on which it was happening
things became ok and chunks migrated successfully
Post by Asya Kamsky
Ah, so now you increased the majority of the cluster, yes?
If you had 3 nodes before, then 2 are majority for voting and for
writeConcern majority. But if you added a 4th node your writeConcern
majority is now three. If you make sure you set the votes for hidden
secondaries to 0 then voting majority at least will go back to 2,
however, for writeConcern it will still be 3. Since migrations
always wait for replication to majority of the cluster before
committing the move, this can manifest as those messages you see (they
should stop when the nodes are all caught up).
Asya
Post by tetlika
it was because of addition of hidden secondaries with priority 0 , and they
were doing initial sync
Понеділок, 29 грудня 2014 р. 14:47:32 UTC+2 користувач Asya Kamsky
Post by Asya Kamsky
When you say things look okay, did you check secondary state? In
particular their replication lag?
Post by tetlika
suddenly we started having a lot of messages on log file of master on one
Wed Dec 24 11:05:11.477 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.497 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.517 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.537 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.558 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.578 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.598 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.618 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.638 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.658 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.678 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.699 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.719 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.739 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.759 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.779 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.799 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.819 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.840 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.860 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.880 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.900 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.920 [migrateThread] warning: secondaries having hard
time keeping up with migrate
Wed Dec 24 11:05:11.940 [migrateThread] warning: secondaries having hard
time keeping up with migrate
it continues to storm log file with those messages, but things look
working ok
what is that? how to fix that?
thanks
--
You received this message because you are subscribed to the Google Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/6ed25be2-e9de-4d4c-846a-08a553de6dd9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/9296a15d-d5d6-4bab-8f4d-165c82c9deac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/ad621ad7-89b5-4e6e-b791-e21b7e0ee6c0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJDLxPC-1_-63ksFZkqFgN23HQz%3D2Gfp-%3Das_zBRDGjFVw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Continue reading on narkive:
Loading...