[mongodb-user] How to avoid AutoReconnect when doing query to the Replica Set

[Jiayu Song]

This is the case: I deploy one MongoDB primiry server with one replica. I
have over 200K document in a collection. When I adopted MongoEngine to
query the collection. After about 100K document scanned, it usually showed
AutoReconnect. Is there any tips to avoid AutoReconnect or anything I can
do for diagnosis? I've already tried to set Timeout keyword argument and
also set the batch_size, but problem still exists.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/20b1169a-9986-4ac6-aaf4-26f4c25807cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

['Wan Bachtiar' via mongodb-user]

When I adopted MongoEngine to query the collection. After about 100K
document scanned, it usually showed AutoReconnect

Hi Jiayu,

For others to help answer your question better, could you provide:

- MongoDB server version
- MongoEngine version
- A minimal, complete and reproducible example code
- The full AutoReconnect message and where do you see it ? i.e. MongoDB
server log, application server log, stack trace etc
- What happens when it shows the AutoReconnect message ? i.e.
application stop ?

Regards,
Wan.
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/8b4a2a44-258d-4ad3-8060-1502e8c3de15%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jiayu Song]

MongoDB Server Version: 3.6
MongoEngine Version 0.15.0

A minimal, complete and reproducible example code(bear me for language
other than English):

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

import os
import sys
import xlsxwriter

sys.path.append(os.path.dirname(os.path.dirname(os.path.dirname(__file__))))
from models.question_models import XxQuestion
from models.paper_models import XxPaper
from models.workbook_models import XxWorkbookContent, XxWorkbook
from models.user_models import ZyUser

from config import ZyConst
from utils.time_utils import format_datetime
from utils.logger.vox_logger import get_logger
from utils.memory_database import MemDB
from utils.table_utils import TableExportHelper

from pymongo import ReadPreference

admin = ZyUser.get_admin()
logger = get_logger('for_debug', deploy_stage='test', log_source='for_debug')


xls = xlsxwriter.Workbook('题目新鲜床.xlsx')

table = TableExportHelper("已䞊线题目时闎(按月计算)")
table.set_head_define("序号", lambda item1: item1["序号"])
table.set_head_define("日期", lambda item2: item2["日期"])
table.set_head_define("教研圕入题目", lambda item3: item3["教研圕入题目"])
table.set_head_define("自劚生成题目", lambda item4: item4["自劚生成题目"])

mem_db_obj = MemDB()
mem_db_obj.create_index(['date'], unique=True)

if __name__ == '__main__':
interested_questions = XxQuestion.objects(subject_id__in=[101, 102, 103], ol_status=1, deleted_at=None,
read_preference=ReadPreference.SECONDARY_PREFERRED).no_cache().timeout(False)
for each_question in interested_questions:
workbook_id_list = []
final_min_time = each_question.created_at

# 计算教蟅的出版时闎和创建时闎的最小倌
workbook_content_objects = XxWorkbookContent.objects(questions__id=each_question.id, ol_status=ZyConst.OL_ONLINE,
deleted_at=None, read_preference=ReadPreference.SECONDARY_PREFERRED).all()
if not workbook_content_objects:
pass
else:
for workbook_content_item in workbook_content_objects:
workbook_id_list.append(workbook_content_item.workbook_id)
workbook_objects = XxWorkbook.objects(id__in=workbook_id_list, ol_status=ZyConst.OL_ONLINE, deleted_at=None,
read_preference=ReadPreference.SECONDARY_PREFERRED).all()
if not workbook_objects:
pass
else:
for workbook_object in workbook_objects:
if workbook_object.date and workbook_object.date < final_min_time:
final_min_time = workbook_object.date
elif workbook_object.created_at < final_min_time:
final_min_time = workbook_object.created_at

# 计算试卷的试卷时闎和创建时闎的最小倌
paper_objects = XxPaper.objects(questions__id=each_question.id, ol_status=ZyConst.OL_ONLINE, deleted_at=None,
read_preference=ReadPreference.SECONDARY_PREFERRED).all()
if not paper_objects:
pass
else:
for paper_object in paper_objects:
if paper_object.date and paper_object.date < final_min_time:
final_min_time = paper_object.date
elif paper_object.date < final_min_time:
final_min_time = paper_object.created_at

formatted_date = format_datetime(final_min_time, '%Y-%m')

entry = mem_db_obj.search_on_index('date', date=formatted_date)
if not entry:
entry = {
'date': formatted_date,
'teaching_input': 0,
'auto_gen': 0,
}
mem_db_obj.insert(entry)
if each_question.creator_id != admin.id:
entry['teaching_input'] += 1
else:
entry['auto_gen'] += 1

i = 1
for entry in mem_db_obj.data:
table.add_row(item1={"序号": i}, item2={"日期": entry['date']}, item3={"教研圕入题目": entry['teaching_input']},
item4={"自劚生成题目": entry['auto_gen']})
i += 1
table.save_to_sheet(xls)
xls.close()



AutoReconnect message showed on PyCharm 'Run' Console:

Traceback (most recent call last):
File "/Users/jiayusong/question-bank/tools/question/question_freshness.py"
, line 40, in <module>
File
"/anaconda3/envs/snakes/lib/python2.7/site-packages/mongoengine/queryset/base.py"
, line 1484, in __next__
raw_doc = six.next(self._cursor)
File
"/anaconda3/envs/snakes/lib/python2.7/site-packages/pymongo/cursor.py",
line 1189, in next
if len(self.__data) or self._refresh():
File
"/anaconda3/envs/snakes/lib/python2.7/site-packages/pymongo/cursor.py",
line 1126, in _refresh
self.__send_message(g)
File
"/anaconda3/envs/snakes/lib/python2.7/site-packages/pymongo/cursor.py",
line 931, in __send_message
operation, exhaust=self.__exhaust, address=self.__address)
File
"/anaconda3/envs/snakes/lib/python2.7/site-packages/pymongo/mongo_client.py"
, line 1145, in _send_message_with_response
exhaust)
File
"/anaconda3/envs/snakes/lib/python2.7/site-packages/pymongo/mongo_client.py"
, line 1156, in _reset_on_error
return func(*args, **kwargs)
File
"/anaconda3/envs/snakes/lib/python2.7/site-packages/pymongo/server.py",
line 106, in send_message_with_response
reply = sock_info.receive_message(request_id)
File "/anaconda3/envs/snakes/lib/python2.7/site-packages/pymongo/pool.py",
line 612, in receive_message
self._raise_connection_failure(error)
File "/anaconda3/envs/snakes/lib/python2.7/site-packages/pymongo/pool.py",
line 743, in _raise_connection_failure
_raise_connection_failure(self.address, error)
File "/anaconda3/envs/snakes/lib/python2.7/site-packages/pymongo/pool.py",
line 283, in _raise_connection_failure
raise AutoReconnect(msg)
pymongo.errors.AutoReconnect: 10.7.1.36:47501: [Errno 54] Connection reset
by peer


The phenomenon when the script stopped:

*It is actually a script that generally ran on about 8 million documents in
MongoDB. It showed the message:*

Process finished with exit code 1





Regards,
Jiayu

Post by 'Wan Bachtiar' via mongodb-user
When I adopted MongoEngine to query the collection. After about 100K
document scanned, it usually showed AutoReconnect
Hi Jiayu,
- MongoDB server version
- MongoEngine version
- A minimal, complete and reproducible example code
- The full AutoReconnect message and where do you see it ? i.e.
MongoDB server log, application server log, stack trace etc
- What happens when it shows the AutoReconnect message ? i.e.
application stop ?
Regards,
Wan.
​

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/748374bf-a125-47b5-b717-f38acd0cd431%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.