Discussion:
hit possible bug SERVER-12209 on 2.2.6?
(too old to reply)
r***@gmail.com
2015-03-14 12:13:51 UTC
Permalink
i have a mongo cluster with 3 nodes. suddenly all the mongo access stopped.
I found the following exception in the log.



Fri Mar 13 16:19:54 [repl writer worker 1] ERROR: writer worker caught
exception: invalid operator: $substr on: { ts: Timestamp 1426260605000|14,
h: 1590251066536574937, v: 2, op: "i", ns:
"mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [ undefined,
0.0, 10.0 ] }, value:
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATE..."
} }

Fri Mar 13 16:19:54 [repl writer worker 1] Fatal Assertion 16360

0xb10fc1 0xad61c3 0x9adba6 0xae44dd 0xb56d39 0x3421607851 0x34212e767d

mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb10fc1]

mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xad61c3]

mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x156)
[0x9adba6]

mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xae44dd]
mongod() [0xb56d39]
#####################################

This looks similar to a post in this group session. Is it possible that it
hit the bug SERVER-12206? Im running mongodb 2.2.6

Thanks
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/291c29f6-160b-4b5e-b46d-6acd1b51a58b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-03-14 20:41:02 UTC
Permalink
It appears that somehow on the primary you were able to insert the document:

{ _id: { $substr: [ undefined, 0.0, 10.0 ] },
value: "CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATE..."
}
}

Looks like it might have been output from mapReduce or aggregation-gone-wrong.

Since that's not a legal document, the secondary is unable to
continue. Since 2.2.6 there has been a lot of validation added to
ensure that invalid documents don't get written on the primary to
begin with.

The fix was back-ported to 2.4 branch but 2.2 has been end-of-life for
a while and it wasn't backported to that version.

Would upgrading be an option for you?

Asya
Post by r***@gmail.com
i have a mongo cluster with 3 nodes. suddenly all the mongo access stopped.
I found the following exception in the log.
Fri Mar 13 16:19:54 [repl writer worker 1] ERROR: writer worker caught
exception: invalid operator: $substr on: { ts: Timestamp 1426260605000|14,
"mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [ undefined,
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATE..."
} }
Fri Mar 13 16:19:54 [repl writer worker 1] Fatal Assertion 16360
0xb10fc1 0xad61c3 0x9adba6 0xae44dd 0xb56d39 0x3421607851 0x34212e767d
mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb10fc1]
mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xad61c3]
mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x156)
[0x9adba6]
mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xae44dd]
mongod() [0xb56d39]
#####################################
This looks similar to a post in this group session. Is it possible that it
hit the bug SERVER-12206? Im running mongodb 2.2.6
Thanks
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/291c29f6-160b-4b5e-b46d-6acd1b51a58b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJBHuX7Fr2YV19siD1Rpce-2FdoSfegOqKtu2PzOBhu22w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
r***@gmail.com
2015-03-15 02:55:13 UTC
Permalink
Thanks for your quick response.

I have restarted one of the notes without replication so we can keep the
application running.
Are there any options other than the upgrade?
Post by r***@gmail.com
i have a mongo cluster with 3 nodes. suddenly all the mongo access stopped.
I found the following exception in the log.
Fri Mar 13 16:19:54 [repl writer worker 1] ERROR: writer worker caught
exception: invalid operator: $substr on: { ts: Timestamp 1426260605000|14,
"mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [ undefined,
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATE..."
} }
Fri Mar 13 16:19:54 [repl writer worker 1] Fatal Assertion 16360
0xb10fc1 0xad61c3 0x9adba6 0xae44dd 0xb56d39 0x3421607851 0x34212e767d
mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb10fc1]
mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xad61c3]
mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x156)
[0x9adba6]
mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xae44dd]
mongod() [0xb56d39]
#####################################
This looks similar to a post in this group session. Is it possible that it
hit the bug SERVER-12206? Im running mongodb 2.2.6
Thanks
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/c560caf1-364d-4d85-84e1-50aa8a472e78%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-03-15 18:49:21 UTC
Permalink
The problem is even if you clean out the "bad" document manually,
there is something that's inserted this document. Unless you track
it down (and fix it or upgrade *it*) it will happen again.

I guess if you can find and remove/fix the bad document you can then
restart all your mongod with objcheck option on (it became default in
2.4) but I think the core of SERVER-12209 is that the checking was
only done at the top level and the illegal field name is embedded one
level below that.

Asya
Post by r***@gmail.com
Thanks for your quick response.
I have restarted one of the notes without replication so we can keep the
application running.
Are there any options other than the upgrade?
Post by r***@gmail.com
i have a mongo cluster with 3 nodes. suddenly all the mongo access stopped.
I found the following exception in the log.
Fri Mar 13 16:19:54 [repl writer worker 1] ERROR: writer worker caught
exception: invalid operator: $substr on: { ts: Timestamp 1426260605000|14,
"mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [ undefined,
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATE..."
} }
Fri Mar 13 16:19:54 [repl writer worker 1] Fatal Assertion 16360
0xb10fc1 0xad61c3 0x9adba6 0xae44dd 0xb56d39 0x3421607851 0x34212e767d
mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb10fc1]
mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xad61c3]
mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x156)
[0x9adba6]
mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xae44dd]
mongod() [0xb56d39]
#####################################
This looks similar to a post in this group session. Is it possible that it
hit the bug SERVER-12206? Im running mongodb 2.2.6
Thanks
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/c560caf1-364d-4d85-84e1-50aa8a472e78%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJDFzKyPHoM64Q6K9gbEdPGAKEFuypVgVbmeJuwrcCLDSw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
r***@gmail.com
2015-03-15 22:15:30 UTC
Permalink
I believe someone was running a map reduce function manually and inserted
the document so it was no really an app.
I was able to find the "bad" document in the oplog.rs in the primary but I
am not allow to delete it from the oplog.
There was also a tmp collection that had the bad document and was able to
drop the collection.
Is there any other way to delete it from oplog or thru some other way?
Post by Asya Kamsky
The problem is even if you clean out the "bad" document manually,
there is something that's inserted this document. Unless you track
it down (and fix it or upgrade *it*) it will happen again.
I guess if you can find and remove/fix the bad document you can then
restart all your mongod with objcheck option on (it became default in
2.4) but I think the core of SERVER-12209 is that the checking was
only done at the top level and the illegal field name is embedded one
level below that.
Asya
Post by r***@gmail.com
Thanks for your quick response.
I have restarted one of the notes without replication so we can keep
the
Post by r***@gmail.com
application running.
Are there any options other than the upgrade?
Post by r***@gmail.com
i have a mongo cluster with 3 nodes. suddenly all the mongo access stopped.
I found the following exception in the log.
Fri Mar 13 16:19:54 [repl writer worker 1] ERROR: writer worker caught
exception: invalid operator: $substr on: { ts: Timestamp
1426260605000|14,
Post by r***@gmail.com
Post by r***@gmail.com
"mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [
undefined,
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATE..."
Post by r***@gmail.com
Post by r***@gmail.com
} }
Fri Mar 13 16:19:54 [repl writer worker 1] Fatal Assertion 16360
0xb10fc1 0xad61c3 0x9adba6 0xae44dd 0xb56d39 0x3421607851 0x34212e767d
mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb10fc1]
mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xad61c3]
mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x156)
Post by r***@gmail.com
Post by r***@gmail.com
[0x9adba6]
mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xae44dd]
mongod() [0xb56d39]
#####################################
This looks similar to a post in this group session. Is it possible that
it
Post by r***@gmail.com
Post by r***@gmail.com
hit the bug SERVER-12206? Im running mongodb 2.2.6
Thanks
--
You received this message because you are subscribed to the Google
Groups
Post by r***@gmail.com
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups
Post by r***@gmail.com
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
an
<javascript:>.
Post by r***@gmail.com
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/c560caf1-364d-4d85-84e1-50aa8a472e78%40googlegroups.com.
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/8bfd4671-ad15-4220-abdd-2c7933f6528b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-03-16 00:04:28 UTC
Permalink
For the oplog you can update the entry changing the "op" field from
"i" to "n", that should do it to skip it.

Asya
Post by r***@gmail.com
I believe someone was running a map reduce function manually and inserted
the document so it was no really an app.
I was able to find the "bad" document in the oplog.rs in the primary but I
am not allow to delete it from the oplog.
There was also a tmp collection that had the bad document and was able to
drop the collection.
Is there any other way to delete it from oplog or thru some other way?
Post by Asya Kamsky
The problem is even if you clean out the "bad" document manually,
there is something that's inserted this document. Unless you track
it down (and fix it or upgrade *it*) it will happen again.
I guess if you can find and remove/fix the bad document you can then
restart all your mongod with objcheck option on (it became default in
2.4) but I think the core of SERVER-12209 is that the checking was
only done at the top level and the illegal field name is embedded one
level below that.
Asya
Post by r***@gmail.com
Thanks for your quick response.
I have restarted one of the notes without replication so we can keep the
application running.
Are there any options other than the upgrade?
Post by r***@gmail.com
i have a mongo cluster with 3 nodes. suddenly all the mongo access stopped.
I found the following exception in the log.
Fri Mar 13 16:19:54 [repl writer worker 1] ERROR: writer worker caught
exception: invalid operator: $substr on: { ts: Timestamp
1426260605000|14,
"mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [ undefined,
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATE..."
} }
Fri Mar 13 16:19:54 [repl writer worker 1] Fatal Assertion 16360
0xb10fc1 0xad61c3 0x9adba6 0xae44dd 0xb56d39 0x3421607851 0x34212e767d
mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb10fc1]
mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xad61c3]
mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x156)
[0x9adba6]
mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xae44dd]
mongod() [0xb56d39]
#####################################
This looks similar to a post in this group session. Is it possible that it
hit the bug SERVER-12206? Im running mongodb 2.2.6
Thanks
--
You received this message because you are subscribed to the Google Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/c560caf1-364d-4d85-84e1-50aa8a472e78%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/8bfd4671-ad15-4220-abdd-2c7933f6528b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJC7WVtiHSG%2BZGg%2BLEiG8vo%3DaJMoFNE1SovGZ25TL1RK0w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
r***@gmail.com
2015-03-16 23:43:55 UTC
Permalink
Okay. Sounds good. Will give it a try.
Thanks for your help.
Post by Asya Kamsky
For the oplog you can update the entry changing the "op" field from
"i" to "n", that should do it to skip it.
Asya
Post by r***@gmail.com
I believe someone was running a map reduce function manually and
inserted
Post by r***@gmail.com
the document so it was no really an app.
I was able to find the "bad" document in the oplog.rs in the primary
but I
Post by r***@gmail.com
am not allow to delete it from the oplog.
There was also a tmp collection that had the bad document and was able
to
Post by r***@gmail.com
drop the collection.
Is there any other way to delete it from oplog or thru some other way?
Post by Asya Kamsky
The problem is even if you clean out the "bad" document manually,
there is something that's inserted this document. Unless you track
it down (and fix it or upgrade *it*) it will happen again.
I guess if you can find and remove/fix the bad document you can then
restart all your mongod with objcheck option on (it became default in
2.4) but I think the core of SERVER-12209 is that the checking was
only done at the top level and the illegal field name is embedded one
level below that.
Asya
Post by r***@gmail.com
Thanks for your quick response.
I have restarted one of the notes without replication so we can keep the
application running.
Are there any options other than the upgrade?
Post by r***@gmail.com
i have a mongo cluster with 3 nodes. suddenly all the mongo access stopped.
I found the following exception in the log.
Fri Mar 13 16:19:54 [repl writer worker 1] ERROR: writer worker
caught
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
exception: invalid operator: $substr on: { ts: Timestamp
1426260605000|14,
"mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [ undefined,
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATE..."
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
} }
Fri Mar 13 16:19:54 [repl writer worker 1] Fatal Assertion 16360
0xb10fc1 0xad61c3 0x9adba6 0xae44dd 0xb56d39 0x3421607851
0x34212e767d
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb10fc1]
mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xad61c3]
mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x156)
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
[0x9adba6]
mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xae44dd]
mongod() [0xb56d39]
#####################################
This looks similar to a post in this group session. Is it possible
that
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
it
hit the bug SERVER-12206? Im running mongodb 2.2.6
Thanks
--
You received this message because you are subscribed to the Google Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
send
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/c560caf1-364d-4d85-84e1-50aa8a472e78%40googlegroups.com.
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google
Groups
Post by r***@gmail.com
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups
Post by r***@gmail.com
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
an
<javascript:>.
Post by r***@gmail.com
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/8bfd4671-ad15-4220-abdd-2c7933f6528b%40googlegroups.com.
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/fe272895-56e3-4fa3-b6a2-11452d2657a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
r***@gmail.com
2015-03-17 21:28:29 UTC
Permalink
Asya,
Just to make sure I get this right. I will be using the find and modify
function to update the op value as ex. below:
db.oplog.rs.findAndModify({ query: { ns :
/mgwhap.tmp.mr.mongodbcollections1_7/ }, update: { op : "n" } });
Can you please confirm?

Is there a way to find out how full is the oplog (current size is 100MB)?
Will replica members become stale if oplog is filled up?
Once oplog is filled up the new documents will overwrite the existing
documents in the oplog?

Thanks in advance.
Post by r***@gmail.com
Okay. Sounds good. Will give it a try.
Thanks for your help.
Post by Asya Kamsky
For the oplog you can update the entry changing the "op" field from
"i" to "n", that should do it to skip it.
Asya
Post by r***@gmail.com
I believe someone was running a map reduce function manually and
inserted
Post by r***@gmail.com
the document so it was no really an app.
I was able to find the "bad" document in the oplog.rs in the primary
but I
Post by r***@gmail.com
am not allow to delete it from the oplog.
There was also a tmp collection that had the bad document and was able
to
Post by r***@gmail.com
drop the collection.
Is there any other way to delete it from oplog or thru some other way?
Post by Asya Kamsky
The problem is even if you clean out the "bad" document manually,
there is something that's inserted this document. Unless you track
it down (and fix it or upgrade *it*) it will happen again.
I guess if you can find and remove/fix the bad document you can then
restart all your mongod with objcheck option on (it became default in
2.4) but I think the core of SERVER-12209 is that the checking was
only done at the top level and the illegal field name is embedded one
level below that.
Asya
Post by r***@gmail.com
Thanks for your quick response.
I have restarted one of the notes without replication so we can
keep
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
the
application running.
Are there any options other than the upgrade?
Post by r***@gmail.com
i have a mongo cluster with 3 nodes. suddenly all the mongo access stopped.
I found the following exception in the log.
Fri Mar 13 16:19:54 [repl writer worker 1] ERROR: writer worker
caught
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
exception: invalid operator: $substr on: { ts: Timestamp 1426260605000|14,
"mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [ undefined,
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATE..."
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
} }
Fri Mar 13 16:19:54 [repl writer worker 1] Fatal Assertion 16360
0xb10fc1 0xad61c3 0x9adba6 0xae44dd 0xb56d39 0x3421607851
0x34212e767d
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb10fc1]
mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xad61c3]
mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x156)
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
[0x9adba6]
mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xae44dd]
mongod() [0xb56d39]
#####################################
This looks similar to a post in this group session. Is it possible
that
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
it
hit the bug SERVER-12206? Im running mongodb 2.2.6
Thanks
--
You received this message because you are subscribed to the Google Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
send
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/c560caf1-364d-4d85-84e1-50aa8a472e78%40googlegroups.com.
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google
Groups
Post by r***@gmail.com
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups
Post by r***@gmail.com
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
an
Post by r***@gmail.com
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/8bfd4671-ad15-4220-abdd-2c7933f6528b%40googlegroups.com.
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/d455b22f-5d76-4153-8708-0df6b4319621%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-03-17 22:57:48 UTC
Permalink
You should not be using findAndModify - you should just use "update".

And that is *not* correct clause - this would match all the operations
on that namespace... plus you are matching it as regex which is also
bad.

First thing, *please* back up the instance and the oplog in particular. Use:
mongodump -d local -c oplog.rs to dump out the oplog.

Now, you need to find the record that's causing this problem in the
oplog, and we know from the error message it's this:

{ ts: Timestamp 1426260605000|14, h: 1590251066536574937, v: 2, op:
"i", ns: "mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [
undefined, 0.0, 10.0 ] }, value:
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATE..."
} }

So first run this:

use local
db.oplog.rs.find( { ts: Timestamp(1426260605000,14), h:
NumberLong("1590251066536574937"), op: "i", ns:
"mgwhap.tmp.mr.mongodbcollections1_7" } )

This should return one and exactly one record and it should look like
the one in the logs (i.e. its _id value will be $substr, etc.)

Now, *only* if you got back *one* record to the find query, run the following:

db.oplog.rs.update( { ts: Timestamp(1426260605000,14), h:
NumberLong("1590251066536574937"), op: "i", ns:
"mgwhap.tmp.mr.mongodbcollections1_7" }, {$set: {op:"n" } } )

If you got something else for the find, please come back here and let
us know what it was.

Asya
Post by r***@gmail.com
Asya,
Just to make sure I get this right. I will be using the find and modify
/mgwhap.tmp.mr.mongodbcollections1_7/ }, update: { op : "n" } });
Can you please confirm?
Is there a way to find out how full is the oplog (current size is 100MB)?
Will replica members become stale if oplog is filled up?
Once oplog is filled up the new documents will overwrite the existing
documents in the oplog?
Thanks in advance.
Post by r***@gmail.com
Okay. Sounds good. Will give it a try.
Thanks for your help.
Post by Asya Kamsky
For the oplog you can update the entry changing the "op" field from
"i" to "n", that should do it to skip it.
Asya
Post by r***@gmail.com
I believe someone was running a map reduce function manually and inserted
the document so it was no really an app.
I was able to find the "bad" document in the oplog.rs in the primary but I
am not allow to delete it from the oplog.
There was also a tmp collection that had the bad document and was able to
drop the collection.
Is there any other way to delete it from oplog or thru some other way?
Post by Asya Kamsky
The problem is even if you clean out the "bad" document manually,
there is something that's inserted this document. Unless you track
it down (and fix it or upgrade *it*) it will happen again.
I guess if you can find and remove/fix the bad document you can then
restart all your mongod with objcheck option on (it became default in
2.4) but I think the core of SERVER-12209 is that the checking was
only done at the top level and the illegal field name is embedded one
level below that.
Asya
Post by r***@gmail.com
Thanks for your quick response.
I have restarted one of the notes without replication so we can keep
the
application running.
Are there any options other than the upgrade?
Post by r***@gmail.com
i have a mongo cluster with 3 nodes. suddenly all the mongo access stopped.
I found the following exception in the log.
Fri Mar 13 16:19:54 [repl writer worker 1] ERROR: writer worker caught
exception: invalid operator: $substr on: { ts: Timestamp 1426260605000|14,
"mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [ undefined,
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATE..."
} }
Fri Mar 13 16:19:54 [repl writer worker 1] Fatal Assertion 16360
0xb10fc1 0xad61c3 0x9adba6 0xae44dd 0xb56d39 0x3421607851 0x34212e767d
mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb10fc1]
mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xad61c3]
mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x156)
[0x9adba6]
mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xae44dd]
mongod() [0xb56d39]
#####################################
This looks similar to a post in this group session. Is it possible that
it
hit the bug SERVER-12206? Im running mongodb 2.2.6
Thanks
--
You received this message because you are subscribed to the Google Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/c560caf1-364d-4d85-84e1-50aa8a472e78%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/8bfd4671-ad15-4220-abdd-2c7933f6528b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/d455b22f-5d76-4153-8708-0df6b4319621%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJCCoYtuxpf3NzUXX5BnpkOt68w1_wYi2Wnc%3DMv4vS1XNQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
r***@gmail.com
2015-03-18 16:11:05 UTC
Permalink
Ok. I was lucky that was able to find it using the find operation. it seems
this is the only transaction on that collection.
I will use the commands you send me to find and update the "bad doc"

I could not find out if anyone submitted this map_reduce request to this
environment. Is it possible that mongodb generated
this document automatically for replication purposes?
Is there a way to find out what command caused this issue?

Thanks.
Post by Asya Kamsky
You should not be using findAndModify - you should just use "update".
And that is *not* correct clause - this would match all the operations
on that namespace... plus you are matching it as regex which is also
bad.
First thing, *please* back up the instance and the oplog in particular.
mongodump -d local -c oplog.rs to dump out the oplog.
Now, you need to find the record that's causing this problem in the
"i", ns: "mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATE..."
} }
use local
"mgwhap.tmp.mr.mongodbcollections1_7" } )
This should return one and exactly one record and it should look like
the one in the logs (i.e. its _id value will be $substr, etc.)
"mgwhap.tmp.mr.mongodbcollections1_7" }, {$set: {op:"n" } } )
If you got something else for the find, please come back here and let
us know what it was.
Asya
Post by r***@gmail.com
Asya,
Just to make sure I get this right. I will be using the find and modify
/mgwhap.tmp.mr.mongodbcollections1_7/ }, update: { op : "n" } });
Can you please confirm?
Is there a way to find out how full is the oplog (current size is
100MB)?
Post by r***@gmail.com
Will replica members become stale if oplog is filled up?
Once oplog is filled up the new documents will overwrite the existing
documents in the oplog?
Thanks in advance.
Post by r***@gmail.com
Okay. Sounds good. Will give it a try.
Thanks for your help.
Post by Asya Kamsky
For the oplog you can update the entry changing the "op" field from
"i" to "n", that should do it to skip it.
Asya
Post by r***@gmail.com
I believe someone was running a map reduce function manually and inserted
the document so it was no really an app.
I was able to find the "bad" document in the oplog.rs in the
primary
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
but I
am not allow to delete it from the oplog.
There was also a tmp collection that had the bad document and was
able
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
to
drop the collection.
Is there any other way to delete it from oplog or thru some other
way?
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
The problem is even if you clean out the "bad" document manually,
there is something that's inserted this document. Unless you
track
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
it down (and fix it or upgrade *it*) it will happen again.
I guess if you can find and remove/fix the bad document you can
then
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
restart all your mongod with objcheck option on (it became default
in
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
2.4) but I think the core of SERVER-12209 is that the checking was
only done at the top level and the illegal field name is embedded
one
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
level below that.
Asya
Post by r***@gmail.com
Thanks for your quick response.
I have restarted one of the notes without replication so we can keep
the
application running.
Are there any options other than the upgrade?
On Saturday, March 14, 2015 at 1:10:11 PM UTC-4,
Post by r***@gmail.com
i have a mongo cluster with 3 nodes. suddenly all the mongo
access
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
stopped.
I found the following exception in the log.
Fri Mar 13 16:19:54 [repl writer worker 1] ERROR: writer worker caught
exception: invalid operator: $substr on: { ts: Timestamp
1426260605000|14,
"mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [
undefined,
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATE..."
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
} }
Fri Mar 13 16:19:54 [repl writer worker 1] Fatal Assertion
16360
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
0xb10fc1 0xad61c3 0x9adba6 0xae44dd 0xb56d39 0x3421607851 0x34212e767d
mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb10fc1]
mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xad61c3]
mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x156)
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
[0x9adba6]
mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xae44dd]
mongod() [0xb56d39]
#####################################
This looks similar to a post in this group session. Is it
possible
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
that
it
hit the bug SERVER-12206? Im running mongodb 2.2.6
Thanks
--
You received this message because you are subscribed to the
Google
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the
Google
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/c560caf1-364d-4d85-84e1-50aa8a472e78%40googlegroups.com.
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
send
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/8bfd4671-ad15-4220-abdd-2c7933f6528b%40googlegroups.com.
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google
Groups
Post by r***@gmail.com
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups
Post by r***@gmail.com
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
an
<javascript:>.
Post by r***@gmail.com
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/d455b22f-5d76-4153-8708-0df6b4319621%40googlegroups.com.
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/8e70d0eb-4eea-42da-aa19-2bd034f6a8d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
r***@gmail.com
2015-03-19 23:38:28 UTC
Permalink
Hi Asya,
Any help on finding out what caused the issue?
Post by r***@gmail.com
Ok. I was lucky that was able to find it using the find operation. it
seems this is the only transaction on that collection.
I will use the commands you send me to find and update the "bad doc"
I could not find out if anyone submitted this map_reduce request to this
environment. Is it possible that mongodb generated
this document automatically for replication purposes?
Is there a way to find out what command caused this issue?
Thanks.
Post by Asya Kamsky
You should not be using findAndModify - you should just use "update".
And that is *not* correct clause - this would match all the operations
on that namespace... plus you are matching it as regex which is also
bad.
First thing, *please* back up the instance and the oplog in particular.
mongodump -d local -c oplog.rs to dump out the oplog.
Now, you need to find the record that's causing this problem in the
"i", ns: "mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATE..."
} }
use local
"mgwhap.tmp.mr.mongodbcollections1_7" } )
This should return one and exactly one record and it should look like
the one in the logs (i.e. its _id value will be $substr, etc.)
"mgwhap.tmp.mr.mongodbcollections1_7" }, {$set: {op:"n" } } )
If you got something else for the find, please come back here and let
us know what it was.
Asya
Post by r***@gmail.com
Asya,
Just to make sure I get this right. I will be using the find and modify
/mgwhap.tmp.mr.mongodbcollections1_7/ }, update: { op : "n" } });
Can you please confirm?
Is there a way to find out how full is the oplog (current size is
100MB)?
Post by r***@gmail.com
Will replica members become stale if oplog is filled up?
Once oplog is filled up the new documents will overwrite the existing
documents in the oplog?
Thanks in advance.
Post by r***@gmail.com
Okay. Sounds good. Will give it a try.
Thanks for your help.
Post by Asya Kamsky
For the oplog you can update the entry changing the "op" field from
"i" to "n", that should do it to skip it.
Asya
Post by r***@gmail.com
I believe someone was running a map reduce function manually and inserted
the document so it was no really an app.
I was able to find the "bad" document in the oplog.rs in the
primary
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
but I
am not allow to delete it from the oplog.
There was also a tmp collection that had the bad document and was
able
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
to
drop the collection.
Is there any other way to delete it from oplog or thru some other
way?
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
The problem is even if you clean out the "bad" document manually,
there is something that's inserted this document. Unless you
track
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
it down (and fix it or upgrade *it*) it will happen again.
I guess if you can find and remove/fix the bad document you can
then
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
restart all your mongod with objcheck option on (it became default
in
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
2.4) but I think the core of SERVER-12209 is that the checking was
only done at the top level and the illegal field name is embedded
one
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
level below that.
Asya
Post by r***@gmail.com
Thanks for your quick response.
I have restarted one of the notes without replication so we can keep
the
application running.
Are there any options other than the upgrade?
On Saturday, March 14, 2015 at 1:10:11 PM UTC-4,
Post by r***@gmail.com
i have a mongo cluster with 3 nodes. suddenly all the mongo
access
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
stopped.
I found the following exception in the log.
Fri Mar 13 16:19:54 [repl writer worker 1] ERROR: writer worker
caught
exception: invalid operator: $substr on: { ts: Timestamp
1426260605000|14,
"mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [
undefined,
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATE..."
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
} }
Fri Mar 13 16:19:54 [repl writer worker 1] Fatal Assertion
16360
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
0xb10fc1 0xad61c3 0x9adba6 0xae44dd 0xb56d39 0x3421607851
0x34212e767d
mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb10fc1]
mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xad61c3]
mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x156)
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
[0x9adba6]
mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xae44dd]
mongod() [0xb56d39]
#####################################
This looks similar to a post in this group session. Is it
possible
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
that
it
hit the bug SERVER-12206? Im running mongodb 2.2.6
Thanks
--
You received this message because you are subscribed to the
Google
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the
Google
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from
it,
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
send
an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/c560caf1-364d-4d85-84e1-50aa8a472e78%40googlegroups.com.
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
send
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/8bfd4671-ad15-4220-abdd-2c7933f6528b%40googlegroups.com.
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google
Groups
Post by r***@gmail.com
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups
Post by r***@gmail.com
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
an
Post by r***@gmail.com
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/d455b22f-5d76-4153-8708-0df6b4319621%40googlegroups.com.
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/f6a1daed-7af1-4254-832d-9d16ff540c58%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-03-20 05:20:18 UTC
Permalink
The name of the collection suggests that it's a map reduce created
collection...
Post by r***@gmail.com
Ok. I was lucky that was able to find it using the find operation. it
seems this is the only transaction on that collection.
I will use the commands you send me to find and update the "bad doc"
I could not find out if anyone submitted this map_reduce request to this
environment. Is it possible that mongodb generated
this document automatically for replication purposes?
Is there a way to find out what command caused this issue?
Thanks.
Post by Asya Kamsky
You should not be using findAndModify - you should just use "update".
And that is *not* correct clause - this would match all the operations
on that namespace... plus you are matching it as regex which is also
bad.
First thing, *please* back up the instance and the oplog in particular.
mongodump -d local -c oplog.rs to dump out the oplog.
Now, you need to find the record that's causing this problem in the
"i", ns: "mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCR
IBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCH
CREATESUBSCRIBERONSWITCHCREATE..."
} }
use local
"mgwhap.tmp.mr.mongodbcollections1_7" } )
This should return one and exactly one record and it should look like
the one in the logs (i.e. its _id value will be $substr, etc.)
"mgwhap.tmp.mr.mongodbcollections1_7" }, {$set: {op:"n" } } )
If you got something else for the find, please come back here and let
us know what it was.
Asya
Post by r***@gmail.com
Asya,
Just to make sure I get this right. I will be using the find and modify
/mgwhap.tmp.mr.mongodbcollections1_7/ }, update: { op : "n" } });
Can you please confirm?
Is there a way to find out how full is the oplog (current size is
100MB)?
Post by r***@gmail.com
Will replica members become stale if oplog is filled up?
Once oplog is filled up the new documents will overwrite the existing
documents in the oplog?
Thanks in advance.
Post by r***@gmail.com
Okay. Sounds good. Will give it a try.
Thanks for your help.
Post by Asya Kamsky
For the oplog you can update the entry changing the "op" field from
"i" to "n", that should do it to skip it.
Asya
Post by r***@gmail.com
I believe someone was running a map reduce function manually and inserted
the document so it was no really an app.
I was able to find the "bad" document in the oplog.rs in the
primary
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
but I
am not allow to delete it from the oplog.
There was also a tmp collection that had the bad document and was
able
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
to
drop the collection.
Is there any other way to delete it from oplog or thru some other
way?
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
The problem is even if you clean out the "bad" document manually,
there is something that's inserted this document. Unless you
track
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
it down (and fix it or upgrade *it*) it will happen again.
I guess if you can find and remove/fix the bad document you can
then
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
restart all your mongod with objcheck option on (it became default
in
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
2.4) but I think the core of SERVER-12209 is that the checking was
only done at the top level and the illegal field name is embedded
one
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
level below that.
Asya
Post by r***@gmail.com
Thanks for your quick response.
I have restarted one of the notes without replication so we can keep
the
application running.
Are there any options other than the upgrade?
On Saturday, March 14, 2015 at 1:10:11 PM UTC-4,
Post by r***@gmail.com
i have a mongo cluster with 3 nodes. suddenly all the mongo
access
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
stopped.
I found the following exception in the log.
Fri Mar 13 16:19:54 [repl writer worker 1] ERROR: writer worker
caught
exception: invalid operator: $substr on: { ts: Timestamp
1426260605000|14,
"mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [
undefined,
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCR
IBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCH
CREATESUBSCRIBERONSWITCHCREATE..."
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
} }
Fri Mar 13 16:19:54 [repl writer worker 1] Fatal Assertion
16360
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
0xb10fc1 0xad61c3 0x9adba6 0xae44dd 0xb56d39 0x3421607851
0x34212e767d
mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb10fc1]
mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xad61c3]
mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_
7BSONObjESaIS2_EEPNS0_8SyncTailE+0x156)
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
[0x9adba6]
mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xae44dd]
mongod() [0xb56d39]
#####################################
This looks similar to a post in this group session. Is it
possible
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
that
it
hit the bug SERVER-12206? Im running mongodb 2.2.6
Thanks
--
You received this message because you are subscribed to the
Google
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the
Google
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from
it,
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
send
an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/c560caf1-
364d-4d85-84e1-50aa8a472e78%40googlegroups.com.
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
send
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/8bfd4671-
ad15-4220-abdd-2c7933f6528b%40googlegroups.com.
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google
Groups
Post by r***@gmail.com
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups
Post by r***@gmail.com
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
an
Post by r***@gmail.com
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/d455b22f-
5d76-4153-8708-0df6b4319621%40googlegroups.com.
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/8e70d0eb-4eea-42da-aa19-2bd034f6a8d1%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/8e70d0eb-4eea-42da-aa19-2bd034f6a8d1%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJBOmXwsX1pRjT%3DacyDToDn_841abaBSMMnjboqOXp3hjQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
r***@gmail.com
2015-03-23 14:21:50 UTC
Permalink
Do you mean that it was created by the application or by a user as opposed
to be created internally by mongodb?

BTW. I applied the workaround you provided and it worked perfectly. Thanks
for your help again.
Post by Asya Kamsky
The name of the collection suggests that it's a map reduce created
collection...
Post by r***@gmail.com
Ok. I was lucky that was able to find it using the find operation. it
seems this is the only transaction on that collection.
I will use the commands you send me to find and update the "bad doc"
I could not find out if anyone submitted this map_reduce request to this
environment. Is it possible that mongodb generated
this document automatically for replication purposes?
Is there a way to find out what command caused this issue?
Thanks.
Post by Asya Kamsky
You should not be using findAndModify - you should just use "update".
And that is *not* correct clause - this would match all the operations
on that namespace... plus you are matching it as regex which is also
bad.
First thing, *please* back up the instance and the oplog in particular.
mongodump -d local -c oplog.rs to dump out the oplog.
Now, you need to find the record that's causing this problem in the
"i", ns: "mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCR
IBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCH
CREATESUBSCRIBERONSWITCHCREATE..."
} }
use local
"mgwhap.tmp.mr.mongodbcollections1_7" } )
This should return one and exactly one record and it should look like
the one in the logs (i.e. its _id value will be $substr, etc.)
"mgwhap.tmp.mr.mongodbcollections1_7" }, {$set: {op:"n" } } )
If you got something else for the find, please come back here and let
us know what it was.
Asya
Post by r***@gmail.com
Asya,
Just to make sure I get this right. I will be using the find and
modify
Post by r***@gmail.com
/mgwhap.tmp.mr.mongodbcollections1_7/ }, update: { op : "n" } });
Can you please confirm?
Is there a way to find out how full is the oplog (current size is
100MB)?
Post by r***@gmail.com
Will replica members become stale if oplog is filled up?
Once oplog is filled up the new documents will overwrite the existing
documents in the oplog?
Thanks in advance.
Post by r***@gmail.com
Okay. Sounds good. Will give it a try.
Thanks for your help.
Post by Asya Kamsky
For the oplog you can update the entry changing the "op" field from
"i" to "n", that should do it to skip it.
Asya
Post by r***@gmail.com
I believe someone was running a map reduce function manually and inserted
the document so it was no really an app.
I was able to find the "bad" document in the oplog.rs in the
primary
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
but I
am not allow to delete it from the oplog.
There was also a tmp collection that had the bad document and was
able
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
to
drop the collection.
Is there any other way to delete it from oplog or thru some other
way?
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
The problem is even if you clean out the "bad" document manually,
there is something that's inserted this document. Unless you
track
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
it down (and fix it or upgrade *it*) it will happen again.
I guess if you can find and remove/fix the bad document you can
then
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
restart all your mongod with objcheck option on (it became
default in
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
2.4) but I think the core of SERVER-12209 is that the checking
was
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
only done at the top level and the illegal field name is embedded
one
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
level below that.
Asya
Post by r***@gmail.com
Thanks for your quick response.
I have restarted one of the notes without replication so we
can
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
keep
the
application running.
Are there any options other than the upgrade?
On Saturday, March 14, 2015 at 1:10:11 PM UTC-4,
Post by r***@gmail.com
i have a mongo cluster with 3 nodes. suddenly all the mongo
access
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
stopped.
I found the following exception in the log.
Fri Mar 13 16:19:54 [repl writer worker 1] ERROR: writer
worker
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
caught
exception: invalid operator: $substr on: { ts: Timestamp
1426260605000|14,
"mgwhap.tmp.mr.mongodbcollections1_7", o: { _id: { $substr: [
undefined,
"CREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCR
IBERONSWITCHCREATESUBSCRIBERONSWITCHCREATESUBSCRIBERONSWITCH
CREATESUBSCRIBERONSWITCHCREATE..."
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
} }
Fri Mar 13 16:19:54 [repl writer worker 1] Fatal Assertion
16360
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
0xb10fc1 0xad61c3 0x9adba6 0xae44dd 0xb56d39 0x3421607851
0x34212e767d
mongod(_ZN5mongo15printStackTraceERSo+0x21) [0xb10fc1]
mongod(_ZN5mongo13fassertFailedEi+0xa3) [0xad61c3]
mongod(_ZN5mongo7replset14multiSyncApplyERKSt6vectorINS_
7BSONObjESaIS2_EEPNS0_8SyncTailE+0x156)
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
[0x9adba6]
mongod(_ZN5mongo10threadpool6Worker4loopEv+0x26d) [0xae44dd]
mongod() [0xb56d39]
#####################################
This looks similar to a post in this group session. Is it
possible
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by r***@gmail.com
that
it
hit the bug SERVER-12206? Im running mongodb 2.2.6
Thanks
--
You received this message because you are subscribed to the
Google
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the
Google
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from
it,
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
send
an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/c560caf1-
364d-4d85-84e1-50aa8a472e78%40googlegroups.com.
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25%
off!
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
--
You received this message because you are subscribed to the Google Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
send
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/8bfd4671-
ad15-4220-abdd-2c7933f6528b%40googlegroups.com.
Post by r***@gmail.com
Post by r***@gmail.com
Post by Asya Kamsky
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google
Groups
Post by r***@gmail.com
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups
Post by r***@gmail.com
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
an
Post by r***@gmail.com
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/d455b22f-
5d76-4153-8708-0df6b4319621%40googlegroups.com.
Post by r***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/8e70d0eb-4eea-42da-aa19-2bd034f6a8d1%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/8e70d0eb-4eea-42da-aa19-2bd034f6a8d1%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
MongoDB World is back! June 1-2 in NYC. Use code ASYA for 25% off!
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/9bfba743-c9ee-4940-bb48-e826d85ef0df%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Loading...