[mongodb-user] Mongodb crashed with "File too large"

Discussion:

a***@seoclarity.net

2017-02-09 15:28:41 UTC

Hi There

we have a 8TB db. and Yesterday db crashed without any reasons. after dig
into log and we are seeing the below errors.

i tried to restart that node,it won't start.also we had replication on
another server but that server e crashed too. please take a look

2017-02-08T09:55:24.661-0600 E STORAGE [thread1] WiredTiger (27)
[1486569324:661608][9029:0x7f8bef870700],
file:collection-9-2207001049530720609.wt
<file://collection-9-2207001049530720609.wt/>, eviction-worker:
collection-9-2207001049530720609.wt: handle-write: pwrite: failed to write
4096 bytes at offset 2194719883264: File too large
2017-02-08T09:55:24.661-0600 E STORAGE [thread1] WiredTiger (27)
[1486569324:661933][9029:0x7f8bef870700], eviction-worker: cache eviction
worker error: File too large
2017-02-08T09:55:24.661-0600 E STORAGE [thread1] WiredTiger (-31804)
[1486569324:661979][9029:0x7f8bef870700], eviction-worker: the process must
exit and restart: WT_PANIC: WiredTiger library panic
2017-02-08T09:55:24.662-0600 I - [thread1] Fatal Assertion 28558
2017-02-08T09:55:24.662-0600 E STORAGE [initandlisten] WiredTiger (-31804)
[1486569324:661994][9029:0x7f8bf3298cc0], file:sizeStorer.wt
<file://sizestorer.wt/>, txn-recover: Operation failed during recovery:
WT_PANIC: WiredTiger library panic
2017-02-08T09:55:24.662-0600 I - [thread1]
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/7f90b539-6596-4f8d-953e-29d328f8e08c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

a***@seoclarity.net

2017-02-09 15:29:07 UTC

Permalink

2017-02-08T10:58:35.004-0600 I - [initandlisten] Detected data files in /opt/R1/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.

2017-02-08T10:58:35.004-0600 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=15G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),

2017-02-08T10:58:38.297-0600 E STORAGE [thread1] WiredTiger (27) [1486573118:296987][9532:0x7f4f6a07e700], file:collection-9-2207001049530720609.wt, eviction-worker: collection-9-2207001049530720609.wt: handle-write: pwrite: failed to write 4096 bytes at offset 2194719883264: File too large

2017-02-08T10:58:38.297-0600 E STORAGE [thread1] WiredTiger (27) [1486573118:297392][9532:0x7f4f6a07e700], eviction-worker: cache eviction worker error: File too large

2017-02-08T10:58:38.297-0600 E STORAGE [thread1] WiredTiger (-31804) [1486573118:297433][9532:0x7f4f6a07e700], eviction-worker: the process must exit and restart: WT_PANIC: WiredTiger library panic

2017-02-08T10:58:38.297-0600 I - [thread1] Fatal Assertion 28558

2017-02-08T10:58:38.297-0600 I - [thread1]

***aborting after fassert() failure

2017-02-08T10:58:38.297-0600 E STORAGE [initandlisten] WiredTiger (-31804) [1486573118:297567][9532:0x7f4f6caa4cc0], file:sizeStorer.wt, txn-recover: Operation failed during recovery: WT_PANIC: WiredTiger library panic

2017-02-08T10:58:38.297-0600 I - [initandlisten] Fatal Assertion 28558

2017-02-08T10:58:38.297-0600 I - [initandlisten]

***aborting after fassert() failure

2017-02-08T10:58:38.304-0600 F - [thread1] Got signal: 6 (Aborted).

0x1315982 0x1314ad9 0x13152e2 0x7f4f6b71f340 0x7f4f6b380cc9 0x7f4f6b3840d8 0x129f992 0x109ad23 0x1a7676c 0x1a76c2d 0x1a77014 0x1a22473 0x7f4f6b717182 0x7f4f6b44447d

----- BEGIN BACKTRACE -----

{"backtrace":[{"b":"400000","o":"F15982","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F14AD9"},{"b":"400000","o":"F152E2"},{"b":"7F4F6B70F000","o":"10340"},{"b":"7F4F6B34A000","o":"36CC9","s":"gsignal"},{"b":"7F4F6B34A000","o":"3A0D8","s":"abort"},{"b":"400000","o":"E9F992","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C9AD23"},{"b":"400000","o":"167676C","s":"__wt_eventv"},{"b":"400000","o":"1676C2D","s":"__wt_err"},{"b":"400000","o":"1677014","s":"__wt_panic"},{"b":"400000","o":"1622473"},{"b":"7F4F6B70F000","o":"8182"},{"b":"7F4F6B34A000","o":"FA47D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.6", "gitVersion" : "05552b562c7a0b3143a729aaa0838e558dc49b25", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.2.0-36-generic", "version" : "#41~14.04.1-Ubuntu SMP Tue Apr 19 17:03:32 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "B35B0D2B9CB934315BA086AB5C88C919B9B5BA50" }, { "b" : "7FFFFD9CD000", "elfType" : 3, "buildId" : "8B2559B12BD6E6C7212A35F980FB4E639F6DAB3B" }, { "b" : "7F4F6C631000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "74864DB9D5F69D39A67E4755012FB6573C469B3D" }, { "b" : "7F4F6C255000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "AAE7CFF8351B730830BDBCE0DCABBE06574B7144" }, { "b" : "7F4F6C04D000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "B376100CAB1EAC4E5DE066EACFC282BF7C0B54F3" }, { "b" : "7F4F6BE49000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "67699FFDA9FD2A552032E0652A242E82D65AA10D" }, { "b" : "7F4F6BB43000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "EF3F6DFFA1FBE48436EC6F45CD3AABA157064BB4" }, { "b" : "7F4F6B92D000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F4F6B70F000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "AF06068681750736E0524DF17D5A86CB2C3F765C" }, { "b" : "7F4F6B34A000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "5382058B69031CAA9B9996C11061CD164C9398FF" }, { "b" : "7F4F6C890000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "2A816C3EBBA4E12813FBD34B06FBD25BC892A67F" } ] }}

mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1315982]

mongod(+0xF14AD9) [0x1314ad9]

mongod(+0xF152E2) [0x13152e2]

libpthread.so.0(+0x10340) [0x7f4f6b71f340]

libc.so.6(gsignal+0x39) [0x7f4f6b380cc9]

libc.so.6(abort+0x148) [0x7f4f6b3840d8]

mongod(_ZN5mongo13fassertFailedEi+0x82) [0x129f992]

mongod(+0xC9AD23) [0x109ad23]

mongod(__wt_eventv+0x42C) [0x1a7676c]

mongod(__wt_err+0x8D) [0x1a76c2d]

mongod(__wt_panic+0x24) [0x1a77014]

mongod(+0x1622473) [0x1a22473]

libpthread.so.0(+0x8182) [0x7f4f6b717182]

libc.so.6(clone+0x6D) [0x7f4f6b44447d]

----- END BACKTRACE -----

Post by a***@seoclarity.net
Hi There
we have a 8TB db. and Yesterday db crashed without any reasons. after dig
into log and we are seeing the below errors.
i tried to restart that node,it won't start.also we had replication on
another server but that server e crashed too. please take a look
2017-02-08T09:55:24.661-0600 E STORAGE [thread1] WiredTiger (27)
[1486569324:661608][9029:0x7f8bef870700],
collection-9-2207001049530720609.wt: handle-write: pwrite: failed to write
4096 bytes at offset 2194719883264: File too large
2017-02-08T09:55:24.661-0600 E STORAGE [thread1] WiredTiger (27)
[1486569324:661933][9029:0x7f8bef870700], eviction-worker: cache eviction
worker error: File too large
2017-02-08T09:55:24.661-0600 E STORAGE [thread1] WiredTiger (-31804)
[1486569324:661979][9029:0x7f8bef870700], eviction-worker: the process
must exit and restart: WT_PANIC: WiredTiger library panic
2017-02-08T09:55:24.662-0600 I - [thread1] Fatal Assertion 28558
2017-02-08T09:55:24.662-0600 E STORAGE [initandlisten] WiredTiger
(-31804) [1486569324:661994][9029:0x7f8bf3298cc0], file:sizeStorer.wt,
txn-recover: Operation failed during recovery: WT_PANIC: WiredTiger library
panic
2017-02-08T09:55:24.662-0600 I - [thread1]

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/34c20802-5423-4b48-9385-05e5004d8ce8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Weishan Ang

2017-02-09 16:09:18 UTC

Permalink

Hello!

What filesystem are you using?

Post by a***@seoclarity.net
2017-02-08T10:58:35.004-0600 I - [initandlisten] Detected data files in /opt/R1/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2017-02-08T10:58:35.004-0600 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=15G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2017-02-08T10:58:38.297-0600 E STORAGE [thread1] WiredTiger (27) [1486573118:296987][9532:0x7f4f6a07e700], file:collection-9-2207001049530720609.wt, eviction-worker: collection-9-2207001049530720609.wt: handle-write: pwrite: failed to write 4096 bytes at offset 2194719883264: File too large
2017-02-08T10:58:38.297-0600 E STORAGE [thread1] WiredTiger (27) [1486573118:297392][9532:0x7f4f6a07e700], eviction-worker: cache eviction worker error: File too large
2017-02-08T10:58:38.297-0600 E STORAGE [thread1] WiredTiger (-31804) [1486573118:297433][9532:0x7f4f6a07e700], eviction-worker: the process must exit and restart: WT_PANIC: WiredTiger library panic
2017-02-08T10:58:38.297-0600 I - [thread1] Fatal Assertion 28558
2017-02-08T10:58:38.297-0600 I - [thread1]
***aborting after fassert() failure
2017-02-08T10:58:38.297-0600 E STORAGE [initandlisten] WiredTiger (-31804) [1486573118:297567][9532:0x7f4f6caa4cc0], file:sizeStorer.wt, txn-recover: Operation failed during recovery: WT_PANIC: WiredTiger library panic
2017-02-08T10:58:38.297-0600 I - [initandlisten] Fatal Assertion 28558
2017-02-08T10:58:38.297-0600 I - [initandlisten]
***aborting after fassert() failure
2017-02-08T10:58:38.304-0600 F - [thread1] Got signal: 6 (Aborted).
0x1315982 0x1314ad9 0x13152e2 0x7f4f6b71f340 0x7f4f6b380cc9 0x7f4f6b3840d8 0x129f992 0x109ad23 0x1a7676c 0x1a76c2d 0x1a77014 0x1a22473 0x7f4f6b717182 0x7f4f6b44447d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"400000","o":"F15982","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"F14AD9"},{"b":"400000","o":"F152E2"},{"b":"7F4F6B70F000","o":"10340"},{"b":"7F4F6B34A000","o":"36CC9","s":"gsignal"},{"b":"7F4F6B34A000","o":"3A0D8","s":"abort"},{"b":"400000","o":"E9F992","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C9AD23"},{"b":"400000","o":"167676C","s":"__wt_eventv"},{"b":"400000","o":"1676C2D","s":"__wt_err"},{"b":"400000","o":"1677014","s":"__wt_panic"},{"b":"400000","o":"1622473"},{"b":"7F4F6B70F000","o":"8182"},{"b":"7F4F6B34A000","o":"FA47D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.6", "gitVersion" : "05552b562c7a0b3143a729aaa0838e558dc49b25", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.2.0-36-generic", "version" : "#41~14.04.1-Ubuntu SMP Tue Apr 19 17:03:32 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "B35B0D2B9CB934315BA086AB5C88C919B9B5BA50" }, { "b" : "7FFFFD9CD000", "elfType" : 3, "buildId" : "8B2559B12BD6E6C7212A35F980FB4E639F6DAB3B" }, { "b" : "7F4F6C631000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "74864DB9D5F69D39A67E4755012FB6573C469B3D" }, { "b" : "7F4F6C255000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "AAE7CFF8351B730830BDBCE0DCABBE06574B7144" }, { "b" : "7F4F6C04D000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "B376100CAB1EAC4E5DE066EACFC282BF7C0B54F3" }, { "b" : "7F4F6BE49000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "67699FFDA9FD2A552032E0652A242E82D65AA10D" }, { "b" : "7F4F6BB43000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "EF3F6DFFA1FBE48436EC6F45CD3AABA157064BB4" }, { "b" : "7F4F6B92D000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F4F6B70F000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "AF06068681750736E0524DF17D5A86CB2C3F765C" }, { "b" : "7F4F6B34A000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "5382058B69031CAA9B9996C11061CD164C9398FF" }, { "b" : "7F4F6C890000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "2A816C3EBBA4E12813FBD34B06FBD25BC892A67F" } ] }}
mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x1315982]
mongod(+0xF14AD9) [0x1314ad9]
mongod(+0xF152E2) [0x13152e2]
libpthread.so.0(+0x10340) [0x7f4f6b71f340]
libc.so.6(gsignal+0x39) [0x7f4f6b380cc9]
libc.so.6(abort+0x148) [0x7f4f6b3840d8]
mongod(_ZN5mongo13fassertFailedEi+0x82) [0x129f992]
mongod(+0xC9AD23) [0x109ad23]
mongod(__wt_eventv+0x42C) [0x1a7676c]
mongod(__wt_err+0x8D) [0x1a76c2d]
mongod(__wt_panic+0x24) [0x1a77014]
mongod(+0x1622473) [0x1a22473]
libpthread.so.0(+0x8182) [0x7f4f6b717182]
libc.so.6(clone+0x6D) [0x7f4f6b44447d]
----- END BACKTRACE -----

Post by a***@seoclarity.net
Hi There
we have a 8TB db. and Yesterday db crashed without any reasons. after dig
into log and we are seeing the below errors.
i tried to restart that node,it won't start.also we had replication on
another server but that server e crashed too. please take a look
2017-02-08T09:55:24.661-0600 E STORAGE [thread1] WiredTiger (27)
[1486569324:661608][9029:0x7f8bef870700],
collection-9-2207001049530720609.wt: handle-write: pwrite: failed to write
4096 bytes at offset 2194719883264: File too large
2017-02-08T09:55:24.661-0600 E STORAGE [thread1] WiredTiger (27)
[1486569324:661933][9029:0x7f8bef870700], eviction-worker: cache
eviction worker error: File too large
2017-02-08T09:55:24.661-0600 E STORAGE [thread1] WiredTiger (-31804)
[1486569324:661979][9029:0x7f8bef870700], eviction-worker: the process
must exit and restart: WT_PANIC: WiredTiger library panic
2017-02-08T09:55:24.662-0600 I - [thread1] Fatal Assertion 28558
2017-02-08T09:55:24.662-0600 E STORAGE [initandlisten] WiredTiger
(-31804) [1486569324:661994][9029:0x7f8bf3298cc0], file:sizeStorer.wt,
txn-recover: Operation failed during recovery: WT_PANIC: WiredTiger library
panic
2017-02-08T09:55:24.662-0600 I - [thread1]

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/1d8448e7-297f-4505-89d0-214fc97950ec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

a***@seoclarity.net

2017-02-09 16:32:05 UTC

Permalink

HI

we are using EXT3

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/1eaa69ed-9c41-4205-9128-24814fccf05f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

'Stephen Steneker' via mongodb-user

2017-02-10 00:49:26 UTC

Permalink

Post by a***@seoclarity.net
we are using EXT3

Hi,

As you noted on SERVER-27948 <https://jira.mongodb.org/browse/SERVER-27948> (which
you also reported for this issue), the maximum file size for EXT3 with 4KiB
blocks is 2TiB so the "File too large" error is expected for a 2 TiB data
file. The solution for this issue is to upgrade your filesystem; more
modern filesystems like XFS and EXT4 support significantly larger limits.

Per the MongoDB Production Notes
<https://docs.mongodb.com/manual/administration/production-notes/#kernel-and-file-systems>,
use of XFS is strongly recommended with WiredTiger. It would also be worth
reviewing ulimit settings
<https://docs.mongodb.com/manual/reference/ulimit/> to ensure adequate
resource limits for your use case.

Regards,
Stennie
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/395bfdde-1eb8-42ff-84d7-e82252d42134%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

a***@seoclarity.net

2017-02-10 16:28:58 UTC

Permalink

thank you

Post by 'Stephen Steneker' via mongodb-user

Post by a***@seoclarity.net
we are using EXT3

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/a44bfdf5-3e50-4556-a144-cb24a9fa6416%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.