Discussion:
queries on a capped collection with 2dspehere index become worse and now crash mongod
(too old to reply)
Chris
2017-01-24 14:14:13 UTC
Permalink
Hi,

we have a capped collection with couple of hundred million documents + a
2dshpere index on location [lon,lat] besides a datetime index and a id
index (not _id).

- Some month $geoNear queries on this collection running fine and very
well.
- At present we can't query with a filter on time & lat, lon.
- If the behaviour became worse, we did a reindex, but it doesn't really
help.

Always vsize (mongostats) grows to the available RAM and then mongod
crash.
- With $near it is the same.
- Simple find queries work, but as soon as the loc 2dspehere is in the game
the problems occur.
- ulimit -v ... also not works.

The only thing what have really changed is that documents distributed in
a smaller date range.

Is " dump && drop db && restore "" the last resort? Is there an
explanation?

Thanks for any help.
Chris
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/6ae2fa54-b648-421a-9144-403465325467%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Kevin Adistambha' via mongodb-user
2017-01-31 22:47:36 UTC
Permalink
Hi Chris

Could you provide more information regarding your deployment, e.g.:

- Your MongoDB version and your O/S version
- The storage engine (WT or MMAPv1)
- The details of your hardware (CPU, RAM, how much swap is configured,
whether it’s a virtual machine, etc.)
- Your topology details (standalone, replica set, or sharded cluster,
how many shards, etc.)
- Output of rs.status() and/or sh.status() if applicable

we have a capped collection with couple of hundred million documents + a
2dshpere index on location [lon,lat] besides a datetime index and a id
index (not _id).

Could you post some example documents and the indexes on the collection?
Please also post the explain results
<https://docs.mongodb.com/manual/reference/explain-results/> of the
relevant queries.


- Some month $geoNear queries on this collection running fine and very
well.
- At present we can’t query with a filter on time & lat, lon.

What changed between today and when the time the queries ran well? Are
there changes in workload, lots of documents inserted, etc?

Always vsize (mongostats) grows to the available RAM and then mongod crash.

MongoDB will try to use as much memory as required in the machine to
provide you with the best performance, so the increase in RAM usage is
expected (depending on workload) and is by design. Regarding the crash, do
you have the relevant information from any of the logs (dmesg, mongod logs,
etc.) that could point to the reason of the crash? Having said that, it is
possible that either limiting virtual memory use of MongoDB using ulimit -v
or the lack of swap could cause MongoDB to crash.


- Simple find queries work, but as soon as the loc 2dspehere is in the
game the problems occur.

Could you post both the simple find query, the 2dsphere query, and their
relevant explain results please?

The only thing what have really changed is that documents distributed in a
smaller date range.

Could you elaborate on this? Some examples would be helpful.

As a general suggestion, you may want to double check if the machine was
setup using the recommended settings. Please see:

- Production Notes
<https://docs.mongodb.com/manual/administration/production-notes/>
- Operations Checklist
<https://docs.mongodb.com/manual/administration/production-checklist-operations/>
- UNIX ulimit Settings
<https://docs.mongodb.com/manual/reference/ulimit/>

Best regards,
Kevin
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/45ead323-d657-4124-8dc0-5a220a4eb07c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Chris
2017-02-02 20:05:50 UTC
Permalink
Hi Kevin,

thanks for your help. See detailed infos below.

Best Chris


Hi Chris
Post by 'Kevin Adistambha' via mongodb-user
- Your MongoDB version and your O/S version
mongodb v3.4.1 on Ubuntu 14.04.5 LTS x86_64
- The storage engine (WT or MMAPv1)
WiredTiger
- The details of your hardware (CPU, RAM
VM (KVM), 8 vCores, 24Gb RAM, 256Mb swap
- , how much swap is configured, whether it’s a virtual machine, etc.)
- Your topology details (standalone, replica set, or sharded cluster,
how many shards, etc.)
standalone
- Output of rs.status() and/or sh.status() if applicable
not applicable
OOM-killer action
Post by 'Kevin Adistambha' via mongodb-user
we have a capped collection with couple of hundred million documents + a
2dshpere index on location [lon,lat] besides a datetime index and a id
index (not _id).
Could you post some example documents and the indexes on the collection?
Please also post the explain results
<https://docs.mongodb.com/manual/reference/explain-results/> of the
relevant queries.
Typical document:

{'_id': ObjectId('583e19effa55f35d2c237da9'),
'id1': 'f00b78553d56c34f37394ea35ba4b505daa8',
'e': 'impression',
'atype': '3',
'its': datetime.datetime(2016, 11, 30, 0, 14, 39, 347000),
'loc': {'coordinates': [8.6999998092651, 52.016700744629], 'type':
'Point'},
'ref': 'somestring',
'ua': 'SomeLongerStringWithMax80characters',
'another_id: '608754c601d3b929191be3a98829012e'}
Post by 'Kevin Adistambha' via mongodb-user
- Some month $geoNear queries on this collection running fine and very
well.
- At present we can’t query with a filter on time & lat, lon.
What changed between today and when the time the queries ran well? Are
there changes in workload, lots of documents inserted, etc?
Nothing what I is in my mind, however it is a capped collection. When the
capped collection hits the limit first couple of times.
The datetime range was 90 days , this decreased to 40 days.
Post by 'Kevin Adistambha' via mongodb-user
Always vsize (mongostats) grows to the available RAM and then mongod crash.
MongoDB will try to use as much memory as required in the machine to
provide you with the best performance, so the increase in RAM usage is
expected (depending on workload) and is by design. Regarding the crash, do
you have the relevant information from any of the logs (dmesg, mongod
logs, etc.) that could point to the reason of the crash? Having said that,
it is possible that either limiting virtual memory use of MongoDB using ulimit
-v or the lack of swap could cause MongoDB to crash.
- Simple find queries work, but as soon as the loc 2dspehere is in the
game the problems occur.
Could you post both the simple find query, the 2dsphere query, and their
relevant explain results please?
ITS query works
Post by 'Kevin Adistambha' via mongodb-user
db.geo.find({ its: {
... "$gt": new Date('2017-01-16T00:00:00'),
... "$lt": new Date('2017-01-17T00:00:00')
... }
... }).explain()
{
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "history.geo",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"its" : {
"$lt" :
ISODate("2017-01-16T23:00:00Z")
}
},
{
"its" : {
"$gt" :
ISODate("2017-01-15T23:00:00Z")
}
}
]
},
"winningPlan" : {
"stage" : "FETCH",
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"its" : -1
},
"indexName" : "its_-1",
"isMultiKey" : false,
"multiKeyPaths" : {
"its" : [ ]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"its" : [
"(new Date(1484607600000),
new Date(1484521200000))"
]
}
}
},
"rejectedPlans" : [ ]
},
"serverInfo" : {
"version" : "3.4.1",
},
"ok" : 1
}



Query with $near, actually I used $geoNear, anyway crashing alway even
with a smaller date window range.


db.geo.find({'$and':[
... {'its':{ "$gt": new Date('2017-01-16T00:00:00'),
... "$lt": new Date('2017-01-17T00:00:00')
... }},
... {"loc":{"$near": {
... "$geometry": {
... type: "Point" ,
... coordinates: [ 13.40495400,52.52000660 ],
... "$maxDistance": 1000,
... }}}}]}).explain()
{
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "history.geo_locations",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"its" : {
"$lt" :
ISODate("2017-01-16T23:00:00Z")
}
},
{
"its" : {
"$gt" :
ISODate("2017-01-15T23:00:00Z")
}
},
{
"loc" : {
"$near" : {
"$geometry" : {
"type" :
"Point",

"coordinates" : [

13.404954,

52.5200066
],

"$maxDistance" : 1000
}
}
}
}
]
},
"winningPlan" : {
"stage" : "FETCH",
"filter" : {
"$and" : [
{
"its" : {
"$lt" :
ISODate("2017-01-16T23:00:00Z")
}
},
{
"its" : {
"$gt" :
ISODate("2017-01-15T23:00:00Z")
}
}
]
},
"inputStage" : {
"stage" : "GEO_NEAR_2DSPHERE",
"keyPattern" : {
"loc" : "2dsphere"
},
"indexName" : "loc_2dsphere",
"indexVersion" : 2
}
},
"rejectedPlans" : [ ]
},

After some minutes.

Error: error doing query: failed: network error while attempting to run
command 'find' on host ...'
2017-02-02T20:54:26.881+0100 I NETWORK [thread1] trying reconnect to
....... failed
2017-02-02T20:54:26.885+0100 W NETWORK [thread1] Failed to connect to
..... reason: errno:111 Connection refused
2017-02-02T20:54:26.886+0100 I NETWORK [thread1] reconnect .........
failed failed
Post by 'Kevin Adistambha' via mongodb-user
The only thing what have really changed is that documents distributed in a
smaller date range.
Could you elaborate on this? Some examples would be helpful.
See above about the capped collection.
Post by 'Kevin Adistambha' via mongodb-user
As a general suggestion, you may want to double check if the machine was
- Production Notes
<https://docs.mongodb.com/manual/administration/production-notes/>
- Operations Checklist
<https://docs.mongodb.com/manual/administration/production-checklist-operations/>
- UNIX ulimit Settings
<https://docs.mongodb.com/manual/reference/ulimit/>
Best regards,
Kevin
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/41c99d19-be40-4aa6-bbc7-dde3c8bf9243%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Asya Kamsky' via mongodb-user
2017-02-03 20:26:48 UTC
Permalink
Is there a reason you're using $near (which is meant for flat 2d geometry)
with a 2dsphere index?

It's possible you are running into SERVER-22224
<https://jira.mongodb.org/browse/SERVER-22224>
Post by Chris
Hi Kevin,
thanks for your help. See detailed infos below.
Best Chris
Hi Chris
Post by 'Kevin Adistambha' via mongodb-user
- Your MongoDB version and your O/S version
mongodb v3.4.1 on Ubuntu 14.04.5 LTS x86_64
- The storage engine (WT or MMAPv1)
WiredTiger
- The details of your hardware (CPU, RAM
VM (KVM), 8 vCores, 24Gb RAM, 256Mb swap
- , how much swap is configured, whether it’s a virtual machine, etc.)
- Your topology details (standalone, replica set, or sharded cluster,
how many shards, etc.)
standalone
- Output of rs.status() and/or sh.status() if applicable
not applicable
OOM-killer action
Post by 'Kevin Adistambha' via mongodb-user
we have a capped collection with couple of hundred million documents + a
2dshpere index on location [lon,lat] besides a datetime index and a id
index (not _id).
Could you post some example documents and the indexes on the collection?
Please also post the explain results
<https://docs.mongodb.com/manual/reference/explain-results/> of the
relevant queries.
{'_id': ObjectId('583e19effa55f35d2c237da9'),
'id1': 'f00b78553d56c34f37394ea35ba4b505daa8',
'e': 'impression',
'atype': '3',
'its': datetime.datetime(2016, 11, 30, 0, 14, 39, 347000),
'Point'},
'ref': 'somestring',
'ua': 'SomeLongerStringWithMax80characters',
'another_id: '608754c601d3b929191be3a98829012e'}
Post by 'Kevin Adistambha' via mongodb-user
- Some month $geoNear queries on this collection running fine and
very well.
- At present we can’t query with a filter on time & lat, lon.
What changed between today and when the time the queries ran well? Are
there changes in workload, lots of documents inserted, etc?
Nothing what I is in my mind, however it is a capped collection. When the
capped collection hits the limit first couple of times.
The datetime range was 90 days , this decreased to 40 days.
Post by 'Kevin Adistambha' via mongodb-user
Always vsize (mongostats) grows to the available RAM and then mongod crash.
MongoDB will try to use as much memory as required in the machine to
provide you with the best performance, so the increase in RAM usage is
expected (depending on workload) and is by design. Regarding the crash, do
you have the relevant information from any of the logs (dmesg, mongod
logs, etc.) that could point to the reason of the crash? Having said that,
it is possible that either limiting virtual memory use of MongoDB using ulimit
-v or the lack of swap could cause MongoDB to crash.
- Simple find queries work, but as soon as the loc 2dspehere is in
the game the problems occur.
Could you post both the simple find query, the 2dsphere query, and their
relevant explain results please?
ITS query works
Post by 'Kevin Adistambha' via mongodb-user
db.geo.find({ its: {
... "$gt": new Date('2017-01-16T00:00:00'),
... "$lt": new Date('2017-01-17T00:00:00')
... }
... }).explain()
{
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "history.geo",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"its" : {
ISODate("2017-01-16T23:00:00Z")
}
},
{
"its" : {
ISODate("2017-01-15T23:00:00Z")
}
}
]
},
"winningPlan" : {
"stage" : "FETCH",
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"its" : -1
},
"indexName" : "its_-1",
"isMultiKey" : false,
"multiKeyPaths" : {
"its" : [ ]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"its" : [
"(new Date(1484607600000),
new Date(1484521200000))"
]
}
}
},
"rejectedPlans" : [ ]
},
"serverInfo" : {
"version" : "3.4.1",
},
"ok" : 1
}
Query with $near, actually I used $geoNear, anyway crashing alway even
with a smaller date window range.
db.geo.find({'$and':[
... {'its':{ "$gt": new Date('2017-01-16T00:00:00'),
... "$lt": new Date('2017-01-17T00:00:00')
... }},
... {"loc":{"$near": {
... "$geometry": {
... type: "Point" ,
... coordinates: [ 13.40495400,52.52000660 ],
... "$maxDistance": 1000,
... }}}}]}).explain()
{
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "history.geo_locations",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"its" : {
ISODate("2017-01-16T23:00:00Z")
}
},
{
"its" : {
ISODate("2017-01-15T23:00:00Z")
}
},
{
"loc" : {
"$near" : {
"$geometry" : {
"Point",
"coordinates" : [
13.404954,
52.5200066
],
"$maxDistance" : 1000
}
}
}
}
]
},
"winningPlan" : {
"stage" : "FETCH",
"filter" : {
"$and" : [
{
"its" : {
ISODate("2017-01-16T23:00:00Z")
}
},
{
"its" : {
ISODate("2017-01-15T23:00:00Z")
}
}
]
},
"inputStage" : {
"stage" : "GEO_NEAR_2DSPHERE",
"keyPattern" : {
"loc" : "2dsphere"
},
"indexName" : "loc_2dsphere",
"indexVersion" : 2
}
},
"rejectedPlans" : [ ]
},
After some minutes.
Error: error doing query: failed: network error while attempting to run
command 'find' on host ...'
2017-02-02T20:54:26.881+0100 I NETWORK [thread1] trying reconnect to
....... failed
2017-02-02T20:54:26.885+0100 W NETWORK [thread1] Failed to connect to
..... reason: errno:111 Connection refused
2017-02-02T20:54:26.886+0100 I NETWORK [thread1] reconnect .........
failed failed
Post by 'Kevin Adistambha' via mongodb-user
The only thing what have really changed is that documents distributed in
a smaller date range.
Could you elaborate on this? Some examples would be helpful.
See above about the capped collection.
Post by 'Kevin Adistambha' via mongodb-user
As a general suggestion, you may want to double check if the machine was
- Production Notes
<https://docs.mongodb.com/manual/administration/production-notes/>
- Operations Checklist
<https://docs.mongodb.com/manual/administration/production-checklist-operations/>
- UNIX ulimit Settings
<https://docs.mongodb.com/manual/reference/ulimit/>
Best regards,
Kevin
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/
msgid/mongodb-user/41c99d19-be40-4aa6-bbc7-dde3c8bf9243%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/41c99d19-be40-4aa6-bbc7-dde3c8bf9243%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJD38ikBRyW5amC9%3D_7tyoKt37MFF%2BV7NOdsjF_0WsdMEA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Chris
2017-02-03 20:49:44 UTC
Permalink
No actually we used $geoNear wiitihin aggregation and this works quite good
until this problems.
What is your suggestion to querying documents (within ,in, max distance
x around lat,lon) more approriate

Thanks
Christian
Post by 'Asya Kamsky' via mongodb-user
Is there a reason you're using $near (which is meant for flat 2d geometry)
with a 2dsphere index?
It's possible you are running into SERVER-22224
<https://jira.mongodb.org/browse/SERVER-22224>
Post by Chris
Hi Kevin,
thanks for your help. See detailed infos below.
Best Chris
Hi Chris
Post by 'Kevin Adistambha' via mongodb-user
- Your MongoDB version and your O/S version
mongodb v3.4.1 on Ubuntu 14.04.5 LTS x86_64
- The storage engine (WT or MMAPv1)
WiredTiger
- The details of your hardware (CPU, RAM
VM (KVM), 8 vCores, 24Gb RAM, 256Mb swap
- , how much swap is configured, whether it’s a virtual machine, etc.)
- Your topology details (standalone, replica set, or sharded
cluster, how many shards, etc.)
standalone
- Output of rs.status() and/or sh.status() if applicable
not applicable
OOM-killer action
Post by 'Kevin Adistambha' via mongodb-user
we have a capped collection with couple of hundred million documents + a
2dshpere index on location [lon,lat] besides a datetime index and a id
index (not _id).
Could you post some example documents and the indexes on the collection?
Please also post the explain results
<https://docs.mongodb.com/manual/reference/explain-results/> of the
relevant queries.
{'_id': ObjectId('583e19effa55f35d2c237da9'),
'id1': 'f00b78553d56c34f37394ea35ba4b505daa8',
'e': 'impression',
'atype': '3',
'its': datetime.datetime(2016, 11, 30, 0, 14, 39, 347000),
'Point'},
'ref': 'somestring',
'ua': 'SomeLongerStringWithMax80characters',
'another_id: '608754c601d3b929191be3a98829012e'}
Post by 'Kevin Adistambha' via mongodb-user
- Some month $geoNear queries on this collection running fine and
very well.
- At present we can’t query with a filter on time & lat, lon.
What changed between today and when the time the queries ran well? Are
there changes in workload, lots of documents inserted, etc?
Nothing what I is in my mind, however it is a capped collection. When
the capped collection hits the limit first couple of times.
The datetime range was 90 days , this decreased to 40 days.
Post by 'Kevin Adistambha' via mongodb-user
Always vsize (mongostats) grows to the available RAM and then mongod crash.
MongoDB will try to use as much memory as required in the machine to
provide you with the best performance, so the increase in RAM usage is
expected (depending on workload) and is by design. Regarding the crash, do
you have the relevant information from any of the logs (dmesg, mongod
logs, etc.) that could point to the reason of the crash? Having said that,
it is possible that either limiting virtual memory use of MongoDB using ulimit
-v or the lack of swap could cause MongoDB to crash.
- Simple find queries work, but as soon as the loc 2dspehere is in
the game the problems occur.
Could you post both the simple find query, the 2dsphere query, and their
relevant explain results please?
ITS query works
Post by 'Kevin Adistambha' via mongodb-user
db.geo.find({ its: {
... "$gt": new Date('2017-01-16T00:00:00'),
... "$lt": new Date('2017-01-17T00:00:00')
... }
... }).explain()
{
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "history.geo",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"its" : {
ISODate("2017-01-16T23:00:00Z")
}
},
{
"its" : {
ISODate("2017-01-15T23:00:00Z")
}
}
]
},
"winningPlan" : {
"stage" : "FETCH",
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"its" : -1
},
"indexName" : "its_-1",
"isMultiKey" : false,
"multiKeyPaths" : {
"its" : [ ]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"its" : [
"(new
Date(1484607600000), new Date(1484521200000))"
]
}
}
},
"rejectedPlans" : [ ]
},
"serverInfo" : {
"version" : "3.4.1",
},
"ok" : 1
}
Query with $near, actually I used $geoNear, anyway crashing alway even
with a smaller date window range.
db.geo.find({'$and':[
... {'its':{ "$gt": new Date('2017-01-16T00:00:00'),
... "$lt": new Date('2017-01-17T00:00:00')
... }},
... {"loc":{"$near": {
... "$geometry": {
... type: "Point" ,
... coordinates: [ 13.40495400,52.52000660 ],
... "$maxDistance": 1000,
... }}}}]}).explain()
{
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "history.geo_locations",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"its" : {
ISODate("2017-01-16T23:00:00Z")
}
},
{
"its" : {
ISODate("2017-01-15T23:00:00Z")
}
},
{
"loc" : {
"$near" : {
"$geometry" : {
"Point",
"coordinates" : [
13.404954,
52.5200066
],
"$maxDistance" : 1000
}
}
}
}
]
},
"winningPlan" : {
"stage" : "FETCH",
"filter" : {
"$and" : [
{
"its" : {
ISODate("2017-01-16T23:00:00Z")
}
},
{
"its" : {
ISODate("2017-01-15T23:00:00Z")
}
}
]
},
"inputStage" : {
"stage" : "GEO_NEAR_2DSPHERE",
"keyPattern" : {
"loc" : "2dsphere"
},
"indexName" : "loc_2dsphere",
"indexVersion" : 2
}
},
"rejectedPlans" : [ ]
},
After some minutes.
Error: error doing query: failed: network error while attempting to run
command 'find' on host ...'
2017-02-02T20:54:26.881+0100 I NETWORK [thread1] trying reconnect to
....... failed
2017-02-02T20:54:26.885+0100 W NETWORK [thread1] Failed to connect to
..... reason: errno:111 Connection refused
2017-02-02T20:54:26.886+0100 I NETWORK [thread1] reconnect .........
failed failed
Post by 'Kevin Adistambha' via mongodb-user
The only thing what have really changed is that documents distributed in
a smaller date range.
Could you elaborate on this? Some examples would be helpful.
See above about the capped collection.
Post by 'Kevin Adistambha' via mongodb-user
As a general suggestion, you may want to double check if the machine was
- Production Notes
<https://docs.mongodb.com/manual/administration/production-notes/>
- Operations Checklist
<https://docs.mongodb.com/manual/administration/production-checklist-operations/>
- UNIX ulimit Settings
<https://docs.mongodb.com/manual/reference/ulimit/>
Best regards,
Kevin
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/41c99d19-be40-4aa6-bbc7-dde3c8bf9243%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/41c99d19-be40-4aa6-bbc7-dde3c8bf9243%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/46a826a4-1a75-4518-8fbb-9ee92e741cb5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
'Asya Kamsky' via mongodb-user
2017-02-04 05:15:54 UTC
Permalink
Yes, I think $geoWithin will be better (or $nearSphere if you need it to be
ordered).

Asya
Post by Chris
No actually we used $geoNear wiitihin aggregation and this works quite
good until this problems.
What is your suggestion to querying documents (within ,in, max distance
x around lat,lon) more approriate
Thanks
Christian
Is there a reason you're using $near (which is meant for flat 2d geometry)
with a 2dsphere index?
It's possible you are running into SERVER-22224
<https://jira.mongodb.org/browse/SERVER-22224>
Hi Kevin,
thanks for your help. See detailed infos below.
Best Chris
Hi Chris
- Your MongoDB version and your O/S version
mongodb v3.4.1 on Ubuntu 14.04.5 LTS x86_64
- The storage engine (WT or MMAPv1)
WiredTiger
- The details of your hardware (CPU, RAM
VM (KVM), 8 vCores, 24Gb RAM, 256Mb swap
- , how much swap is configured, whether it’s a virtual machine, etc.)
- Your topology details (standalone, replica set, or sharded cluster,
how many shards, etc.)
standalone
- Output of rs.status() and/or sh.status() if applicable
not applicable
OOM-killer action
we have a capped collection with couple of hundred million documents + a
2dshpere index on location [lon,lat] besides a datetime index and a id
index (not _id).
Could you post some example documents and the indexes on the collection?
Please also post the explain results
<https://docs.mongodb.com/manual/reference/explain-results/> of the
relevant queries.
{'_id': ObjectId('583e19effa55f35d2c237da9'),
'id1': 'f00b78553d56c34f37394ea35ba4b505daa8',
'e': 'impression',
'atype': '3',
'its': datetime.datetime(2016, 11, 30, 0, 14, 39, 347000),
'Point'},
'ref': 'somestring',
'ua': 'SomeLongerStringWithMax80characters',
'another_id: '608754c601d3b929191be3a98829012e'}
- Some month $geoNear queries on this collection running fine and very
well.
- At present we can’t query with a filter on time & lat, lon.
What changed between today and when the time the queries ran well? Are
there changes in workload, lots of documents inserted, etc?
Nothing what I is in my mind, however it is a capped collection. When the
capped collection hits the limit first couple of times.
The datetime range was 90 days , this decreased to 40 days.
Always vsize (mongostats) grows to the available RAM and then mongod crash.
MongoDB will try to use as much memory as required in the machine to
provide you with the best performance, so the increase in RAM usage is
expected (depending on workload) and is by design. Regarding the crash, do
you have the relevant information from any of the logs (dmesg, mongod
logs, etc.) that could point to the reason of the crash? Having said that,
it is possible that either limiting virtual memory use of MongoDB using ulimit
-v or the lack of swap could cause MongoDB to crash.
- Simple find queries work, but as soon as the loc 2dspehere is in the
game the problems occur.
Could you post both the simple find query, the 2dsphere query, and their
relevant explain results please?
ITS query works
Post by 'Kevin Adistambha' via mongodb-user
db.geo.find({ its: {
... "$gt": new Date('2017-01-16T00:00:00'),
... "$lt": new Date('2017-01-17T00:00:00')
... }
... }).explain()
{
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "history.geo",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"its" : {
ISODate("2017-01-16T23:00:00Z")
}
},
{
"its" : {
ISODate("2017-01-15T23:00:00Z")
}
}
]
},
"winningPlan" : {
"stage" : "FETCH",
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"its" : -1
},
"indexName" : "its_-1",
"isMultiKey" : false,
"multiKeyPaths" : {
"its" : [ ]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"its" : [
"(new Date(1484607600000),
new Date(1484521200000))"
]
}
}
},
"rejectedPlans" : [ ]
},
"serverInfo" : {
"version" : "3.4.1",
},
"ok" : 1
}
Query with $near, actually I used $geoNear, anyway crashing alway even
with a smaller date window range.
db.geo.find({'$and':[
... {'its':{ "$gt": new Date('2017-01-16T00:00:00'),
... "$lt": new Date('2017-01-17T00:00:00')
... }},
... {"loc":{"$near": {
... "$geometry": {
... type: "Point" ,
... coordinates: [ 13.40495400,52.52000660 ],
... "$maxDistance": 1000,
... }}}}]}).explain()
{
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "history.geo_locations",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [
{
"its" : {
ISODate("2017-01-16T23:00:00Z")
}
},
{
"its" : {
ISODate("2017-01-15T23:00:00Z")
}
},
{
"loc" : {
"$near" : {
"$geometry" : {
"Point",
"coordinates" : [
13.404954,
52.5200066
],
"$maxDistance" : 1000
}
}
}
}
]
},
"winningPlan" : {
"stage" : "FETCH",
"filter" : {
"$and" : [
{
"its" : {
ISODate("2017-01-16T23:00:00Z")
}
},
{
"its" : {
ISODate("2017-01-15T23:00:00Z")
}
}
]
},
"inputStage" : {
"stage" : "GEO_NEAR_2DSPHERE",
"keyPattern" : {
"loc" : "2dsphere"
},
"indexName" : "loc_2dsphere",
"indexVersion" : 2
}
},
"rejectedPlans" : [ ]
},
After some minutes.
Error: error doing query: failed: network error while attempting to run
command 'find' on host ...'
2017-02-02T20:54:26.881+0100 I NETWORK [thread1] trying reconnect to
....... failed
2017-02-02T20:54:26.885+0100 W NETWORK [thread1] Failed to connect to
..... reason: errno:111 Connection refused
2017-02-02T20:54:26.886+0100 I NETWORK [thread1] reconnect .........
failed failed
The only thing what have really changed is that documents distributed in a
smaller date range.
Could you elaborate on this? Some examples would be helpful.
See above about the capped collection.
As a general suggestion, you may want to double check if the machine was
- Production Notes
<https://docs.mongodb.com/manual/administration/production-notes/>
- Operations Checklist
<https://docs.mongodb.com/manual/administration/production-checklist-operations/>
- UNIX ulimit Settings
<https://docs.mongodb.com/manual/reference/ulimit/>
Best regards,
Kevin
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/41c99d19-be40-4aa6-bbc7-dde3c8bf9243%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/41c99d19-be40-4aa6-bbc7-dde3c8bf9243%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
Asya Kamsky
Lead Product Manager
MongoDB
Download MongoDB - mongodb.org/downloads
We're Hiring! - https://www.mongodb.com/careers
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/46a826a4-1a75-4518-8fbb-9ee92e741cb5%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/46a826a4-1a75-4518-8fbb-9ee92e741cb5%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJAfXgo_HHCvzH3ta7uQVfCTXLknujeVBY3-57Ps9GjMZQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Loading...