Discussion:
mongodb old records missing
(too old to reply)
t***@gmail.com
2015-08-15 22:27:20 UTC
Permalink
I have a mongodb database containing a collection that has remained
unchanged for a period of time -- the underlying .ns and .0, .1 etc files
have not been modified for months. Up until a few weeks ago, I had no
problem reading records from the collection. There were thousands of
records in the collection.

However, today when I attempted to read the records, many of them appeared
missing -- e.g. records that I expected to be there were not, although some
of the records were available. Now, there appear to be only 250 records. =

I copied the database files and did a repair of the (copied) database,
which ran fine apparently, but only the 250 records appeared, and cannot
find the others.

I can see manually that the bSON binary files in the database directory
associated with this database still contain the data that I want to access
(by grepping the contents of the database files, I can see some unique text
strings that are associated just with missing records -- of course they do,
since the files haven't been modified for months).

I cannot think of anything that changed between now and the last time that
I successfully was able to access the records, except that the DB process
crashed once. It was brought back up immediately. Although, as I say, it
doesn't appear that anything modified the files associated with the
database where these records are.

What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/1d24335b-cd66-4f02-80f4-b9be65d91cc0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Chris De Bruyne
2015-08-17 07:26:20 UTC
Permalink
Can you give some more info?

Like which query are you doing to find the documents, what is the structure
of the docs, are there indexes on this collection?
Post by t***@gmail.com
I have a mongodb database containing a collection that has remained
unchanged for a period of time -- the underlying .ns and .0, .1 etc files
have not been modified for months. Up until a few weeks ago, I had no
problem reading records from the collection. There were thousands of
records in the collection.
However, today when I attempted to read the records, many of them appeared
missing -- e.g. records that I expected to be there were not, although some
of the records were available. Now, there appear to be only 250 records. =
I copied the database files and did a repair of the (copied) database,
which ran fine apparently, but only the 250 records appeared, and cannot
find the others.
I can see manually that the bSON binary files in the database directory
associated with this database still contain the data that I want to access
(by grepping the contents of the database files, I can see some unique text
strings that are associated just with missing records -- of course they do,
since the files haven't been modified for months).
I cannot think of anything that changed between now and the last time that
I successfully was able to access the records, except that the DB process
crashed once. It was brought back up immediately. Although, as I say, it
doesn't appear that anything modified the files associated with the
database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/c0342334-554d-4c65-b5fd-b811ce71f419%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Ankit Kakkar
2015-08-17 16:47:07 UTC
Permalink
Hello Tobjan,

Thanks for reaching out to us. To assist us in the investigation of this
case, please provide us with following information:

1) Describe your MongoDB deployment (standalone, replica set, or sharded
cluster)
2) What version of MongoDB are you running? (You can check it with
db.version() <http://docs.mongodb.org/manual/reference/method/db.version>
in the mongo shell)
3) Which query did you use to find or count the documents? Can you please
run that query with explain()
<http://docs.mongodb.org/manual/reference/method/cursor.explain/> and send
us the output?
4) Output of db.collection.getIndexes()
<http://docs.mongodb.org/manual/reference/method/db.collection.getIndexes/>
for the collection where documents appear to be missing.

Regards,
ankit
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what is the
structure of the docs, are there indexes on this collection?
Post by t***@gmail.com
I have a mongodb database containing a collection that has remained
unchanged for a period of time -- the underlying .ns and .0, .1 etc files
have not been modified for months. Up until a few weeks ago, I had no
problem reading records from the collection. There were thousands of
records in the collection.
However, today when I attempted to read the records, many of them
appeared missing -- e.g. records that I expected to be there were not,
although some of the records were available. Now, there appear to be only
250 records. =
I copied the database files and did a repair of the (copied) database,
which ran fine apparently, but only the 250 records appeared, and cannot
find the others.
I can see manually that the bSON binary files in the database directory
associated with this database still contain the data that I want to access
(by grepping the contents of the database files, I can see some unique text
strings that are associated just with missing records -- of course they do,
since the files haven't been modified for months).
I cannot think of anything that changed between now and the last time
that I successfully was able to access the records, except that the DB
process crashed once. It was brought back up immediately. Although, as I
say, it doesn't appear that anything modified the files associated with the
database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/6ca93621-854a-4bcc-87bc-26faef6b6de7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-08-17 22:18:18 UTC
Permalink
In addition to the information below, can you please also check if you are
running with journaling disabled?
Post by t***@gmail.com
I cannot think of anything that changed between now and the last time
that I successfully was able to access the records, except that the DB
process crashed once.
Post by t***@gmail.com
It was brought back up immediately. Although, as I say, it doesn't appear
that anything modified the files associated with the database where these
records are.

If your mongod runs with journaling disabled (which is *NOT* the default)
then any crash could cause data loss.

Asya
Post by t***@gmail.com
Hello Tobjan,
Thanks for reaching out to us. To assist us in the investigation of this
1) Describe your MongoDB deployment (standalone, replica set, or sharded
cluster)
2) What version of MongoDB are you running? (You can check it with
db.version() <http://docs.mongodb.org/manual/reference/method/db.version>
in the mongo shell)
3) Which query did you use to find or count the documents? Can you please
run that query with explain()
<http://docs.mongodb.org/manual/reference/method/cursor.explain/> and
send us the output?
4) Output of db.collection.getIndexes()
<http://docs.mongodb.org/manual/reference/method/db.collection.getIndexes/>
for the collection where documents appear to be missing.
Regards,
ankit
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what is the
structure of the docs, are there indexes on this collection?
Post by t***@gmail.com
I have a mongodb database containing a collection that has remained
unchanged for a period of time -- the underlying .ns and .0, .1 etc files
have not been modified for months. Up until a few weeks ago, I had no
problem reading records from the collection. There were thousands of
records in the collection.
However, today when I attempted to read the records, many of them
appeared missing -- e.g. records that I expected to be there were not,
although some of the records were available. Now, there appear to be only
250 records. =
I copied the database files and did a repair of the (copied) database,
which ran fine apparently, but only the 250 records appeared, and cannot
find the others.
I can see manually that the bSON binary files in the database directory
associated with this database still contain the data that I want to access
(by grepping the contents of the database files, I can see some unique text
strings that are associated just with missing records -- of course they do,
since the files haven't been modified for months).
I cannot think of anything that changed between now and the last time
that I successfully was able to access the records, except that the DB
process crashed once. It was brought back up immediately. Although, as I
say, it doesn't appear that anything modified the files associated with the
database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/6ca93621-854a-4bcc-87bc-26faef6b6de7%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/6ca93621-854a-4bcc-87bc-26faef6b6de7%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJAGn0v%2BjpU2MBY1YRWEptVkuqf7owOpR1mrEeyAJW-H6A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
t***@gmail.com
2015-08-18 16:54:07 UTC
Permalink
Hi Ankit --

Thanks so much for your reply -- it's really appreciated.

Here are some answers to your questions:

1) it's a standalone deployment, with no replica set (I'm using journaling
though)

2) version is 2.4.6

3) Here is the output of explain (done through pymongo):

{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
u'server': ***@myport}

(I think is saying 279 here because that is the number of records shown --
not 250, like I said before, that was a mistake.)

4) Here is the index information (also through pymongo):

{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1), (u'uploadDate',
-1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}

As you can see, this is from a ".files" collection used to manage a gridfs
instance.


Also I think I have some additional information. I copied the database
files to a new location and ran a full repair on the whole database. IN
the new repaired version, there were many fewer files .ns files. As is as
if some records were removed using e.g. gridfs.GridFS.remove ... so in the
new (compacted) version the data was finally deleted. Maybe its somehow
possible that the records were removed from the gridfs collection by
someone running a remove operation? I don't see any evidence of such an
operation in the logs but perhaps it's possible? I've looked carefully
at the logs and am not seeing anything obvious, but maybe dont' know what
to look for.

Of course, that I guess does not modify the underlying files on the
original collection where the (hypothetical) remove would have been run.
If this indeed the case, is there some way to get the data back? After
all, I do have the original .ns files.

Thanks!
Tob
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the investigation of this
1) Describe your MongoDB deployment (standalone, replica set, or sharded
cluster)
2) What version of MongoDB are you running? (You can check it with
db.version() <http://docs.mongodb.org/manual/reference/method/db.version>
in the mongo shell)
3) Which query did you use to find or count the documents? Can you please
run that query with explain()
<http://docs.mongodb.org/manual/reference/method/cursor.explain/> and
send us the output?
4) Output of db.collection.getIndexes()
<http://docs.mongodb.org/manual/reference/method/db.collection.getIndexes/>
for the collection where documents appear to be missing.
Regards,
ankit
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what is the
structure of the docs, are there indexes on this collection?
Post by t***@gmail.com
I have a mongodb database containing a collection that has remained
unchanged for a period of time -- the underlying .ns and .0, .1 etc files
have not been modified for months. Up until a few weeks ago, I had no
problem reading records from the collection. There were thousands of
records in the collection.
However, today when I attempted to read the records, many of them
appeared missing -- e.g. records that I expected to be there were not,
although some of the records were available. Now, there appear to be only
250 records. =
I copied the database files and did a repair of the (copied) database,
which ran fine apparently, but only the 250 records appeared, and cannot
find the others.
I can see manually that the bSON binary files in the database directory
associated with this database still contain the data that I want to access
(by grepping the contents of the database files, I can see some unique text
strings that are associated just with missing records -- of course they do,
since the files haven't been modified for months).
I cannot think of anything that changed between now and the last time
that I successfully was able to access the records, except that the DB
process crashed once. It was brought back up immediately. Although, as I
say, it doesn't appear that anything modified the files associated with the
database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-08-18 17:52:55 UTC
Permalink
Do you have the logs from around the time the database crashed? When was
that relative to when you noticed the records missing?

Since you had journaling on, and you were able to restart mongod without
any errors, I would rule out the crash *causing* records to disappear,
which leaves you with the possibility that they were deleted. If they
were deleted and you don't have any backups from *before* they were
deleted, then there's not much that can be done to recover the data - sure,
it's possible to write code to literally look through the deleted list, but
if any data was inserted after the deletes then that space would be reused
and the old data would be overwritten forever. :(

Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm using journaling
though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of records shown --
not 250, like I said before, that was a mistake.)
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1), (u'uploadDate',
-1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used to manage a gridfs
instance.
Also I think I have some additional information. I copied the database
files to a new location and ran a full repair on the whole database. IN
the new repaired version, there were many fewer files .ns files. As is as
if some records were removed using e.g. gridfs.GridFS.remove ... so in the
new (compacted) version the data was finally deleted. Maybe its somehow
possible that the records were removed from the gridfs collection by
someone running a remove operation? I don't see any evidence of such an
operation in the logs but perhaps it's possible? I've looked carefully
at the logs and am not seeing anything obvious, but maybe dont' know what
to look for.
Of course, that I guess does not modify the underlying files on the
original collection where the (hypothetical) remove would have been run.
If this indeed the case, is there some way to get the data back? After
all, I do have the original .ns files.
Thanks!
Tob
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the investigation of this
1) Describe your MongoDB deployment (standalone, replica set, or sharded
cluster)
2) What version of MongoDB are you running? (You can check it with
db.version() <http://docs.mongodb.org/manual/reference/method/db.version>
in the mongo shell)
3) Which query did you use to find or count the documents? Can you please
run that query with explain()
<http://docs.mongodb.org/manual/reference/method/cursor.explain/> and
send us the output?
4) Output of db.collection.getIndexes()
<http://docs.mongodb.org/manual/reference/method/db.collection.getIndexes/>
for the collection where documents appear to be missing.
Regards,
ankit
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what is the
structure of the docs, are there indexes on this collection?
Post by t***@gmail.com
I have a mongodb database containing a collection that has remained
unchanged for a period of time -- the underlying .ns and .0, .1 etc files
have not been modified for months. Up until a few weeks ago, I had no
problem reading records from the collection. There were thousands of
records in the collection.
However, today when I attempted to read the records, many of them
appeared missing -- e.g. records that I expected to be there were not,
although some of the records were available. Now, there appear to be only
250 records. =
I copied the database files and did a repair of the (copied) database,
which ran fine apparently, but only the 250 records appeared, and cannot
find the others.
I can see manually that the bSON binary files in the database directory
associated with this database still contain the data that I want to access
(by grepping the contents of the database files, I can see some unique text
strings that are associated just with missing records -- of course they do,
since the files haven't been modified for months).
I cannot think of anything that changed between now and the last time
that I successfully was able to access the records, except that the DB
process crashed once. It was brought back up immediately. Although, as I
say, it doesn't appear that anything modified the files associated with the
database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJCcHUpLvpy1PRwYZTTm2ZuK2mj0WPcjLsJf0juzJ%2BRk_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
t***@gmail.com
2015-08-18 17:56:37 UTC
Permalink
Post by Asya Kamsky
Do you have the logs from around the time the database crashed? When was
that relative to when you noticed the records missing?
Yes I have all the logs going back for a year. I noticed the records
missing the day that I wrote the help question originally (several days
ago). The crash happened several weeks ago.


Since you had journaling on, and you were able to restart mongod without
Post by Asya Kamsky
any errors, I would rule out the crash *causing* records to disappear,
which leaves you with the possibility that they were deleted. If they
were deleted and you don't have any backups from *before* they were
deleted, then there's not much that can be done to recover the data - sure,
it's possible to write code to literally look through the deleted list, but
if any data was inserted after the deletes then that space would be reused
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But wouldn't there be some
record in the logs if there was a deleation?

We haven't written any records since the deletes. (The files haven't
changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm using
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of records shown
-- not 250, like I said before, that was a mistake.)
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1), (u'uploadDate',
-1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used to manage a
gridfs instance.
Also I think I have some additional information. I copied the database
files to a new location and ran a full repair on the whole database. IN
the new repaired version, there were many fewer files .ns files. As is as
if some records were removed using e.g. gridfs.GridFS.remove ... so in the
new (compacted) version the data was finally deleted. Maybe its somehow
possible that the records were removed from the gridfs collection by
someone running a remove operation? I don't see any evidence of such an
operation in the logs but perhaps it's possible? I've looked carefully
at the logs and am not seeing anything obvious, but maybe dont' know what
to look for.
Of course, that I guess does not modify the underlying files on the
original collection where the (hypothetical) remove would have been run.
If this indeed the case, is there some way to get the data back? After
all, I do have the original .ns files.
Thanks!
Tob
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the investigation of this
1) Describe your MongoDB deployment (standalone, replica set, or sharded
cluster)
2) What version of MongoDB are you running? (You can check it with
db.version()
<http://docs.mongodb.org/manual/reference/method/db.version> in the
mongo shell)
3) Which query did you use to find or count the documents? Can you
please run that query with explain()
<http://docs.mongodb.org/manual/reference/method/cursor.explain/> and
send us the output?
4) Output of db.collection.getIndexes()
<http://docs.mongodb.org/manual/reference/method/db.collection.getIndexes/>
for the collection where documents appear to be missing.
Regards,
ankit
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what is the
structure of the docs, are there indexes on this collection?
Post by t***@gmail.com
I have a mongodb database containing a collection that has remained
unchanged for a period of time -- the underlying .ns and .0, .1 etc files
have not been modified for months. Up until a few weeks ago, I had no
problem reading records from the collection. There were thousands of
records in the collection.
However, today when I attempted to read the records, many of them
appeared missing -- e.g. records that I expected to be there were not,
although some of the records were available. Now, there appear to be only
250 records. =
I copied the database files and did a repair of the (copied) database,
which ran fine apparently, but only the 250 records appeared, and cannot
find the others.
I can see manually that the bSON binary files in the database
directory associated with this database still contain the data that I want
to access (by grepping the contents of the database files, I can see some
unique text strings that are associated just with missing records -- of
course they do, since the files haven't been modified for months).
I cannot think of anything that changed between now and the last time
that I successfully was able to access the records, except that the DB
process crashed once. It was brought back up immediately. Although, as I
say, it doesn't appear that anything modified the files associated with the
database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-08-19 18:53:23 UTC
Permalink
The only CRUD operations that would be logged would be those that take
longer than 100ms.

Might look something like this:

2015-07-14T12:10:34.559-0500 I WRITE [conn4822] remove blackbox.bigcoll
query: { _id: { $gte: 10000.0 } } ndeleted:90000 keyUpdates:0
writeConflicts:0 numYields:12330 locks:{ Global: { acquireCount: { r:
12331, w: 12331 } }, Database: { acquireCount: { w: 12331 } }, Collection:
{ acquireCount: { w: 12331 } } } 249168ms

Of course this was a huge mass remove of giant docs which is why it took so
long. If the number of documents deleted was small (like if they were
deleted one at a time instead of via single command) then it's much less
likely they would have taken longer than 100ms.

I would suggest searching through the logs with equivalent command to:

grep 'remove <dbname>' *.log.*

where you would replace <dbname> with the name of your actual DB and
replace *.log.* with path to your log files.

Asya
Post by t***@gmail.com
Post by Asya Kamsky
Do you have the logs from around the time the database crashed? When
was that relative to when you noticed the records missing?
Yes I have all the logs going back for a year. I noticed the records
missing the day that I wrote the help question originally (several days
ago). The crash happened several weeks ago.
Since you had journaling on, and you were able to restart mongod without
Post by Asya Kamsky
any errors, I would rule out the crash *causing* records to disappear,
which leaves you with the possibility that they were deleted. If they
were deleted and you don't have any backups from *before* they were
deleted, then there's not much that can be done to recover the data - sure,
it's possible to write code to literally look through the deleted list, but
if any data was inserted after the deletes then that space would be reused
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But wouldn't there be
some record in the logs if there was a deleation?
We haven't written any records since the deletes. (The files haven't
changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm using
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of records shown
-- not 250, like I said before, that was a mistake.)
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1),
(u'uploadDate', -1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used to manage a
gridfs instance.
Also I think I have some additional information. I copied the database
files to a new location and ran a full repair on the whole database. IN
the new repaired version, there were many fewer files .ns files. As is as
if some records were removed using e.g. gridfs.GridFS.remove ... so in the
new (compacted) version the data was finally deleted. Maybe its somehow
possible that the records were removed from the gridfs collection by
someone running a remove operation? I don't see any evidence of such an
operation in the logs but perhaps it's possible? I've looked carefully
at the logs and am not seeing anything obvious, but maybe dont' know what
to look for.
Of course, that I guess does not modify the underlying files on the
original collection where the (hypothetical) remove would have been run.
If this indeed the case, is there some way to get the data back? After
all, I do have the original .ns files.
Thanks!
Tob
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the investigation of
1) Describe your MongoDB deployment (standalone, replica set, or
sharded cluster)
2) What version of MongoDB are you running? (You can check it with
db.version()
<http://docs.mongodb.org/manual/reference/method/db.version> in the
mongo shell)
3) Which query did you use to find or count the documents? Can you
please run that query with explain()
<http://docs.mongodb.org/manual/reference/method/cursor.explain/> and
send us the output?
4) Output of db.collection.getIndexes()
<http://docs.mongodb.org/manual/reference/method/db.collection.getIndexes/>
for the collection where documents appear to be missing.
Regards,
ankit
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what is the
structure of the docs, are there indexes on this collection?
Post by t***@gmail.com
I have a mongodb database containing a collection that has remained
unchanged for a period of time -- the underlying .ns and .0, .1 etc files
have not been modified for months. Up until a few weeks ago, I had no
problem reading records from the collection. There were thousands of
records in the collection.
However, today when I attempted to read the records, many of them
appeared missing -- e.g. records that I expected to be there were not,
although some of the records were available. Now, there appear to be only
250 records. =
I copied the database files and did a repair of the (copied)
database, which ran fine apparently, but only the 250 records appeared, and
cannot find the others.
I can see manually that the bSON binary files in the database
directory associated with this database still contain the data that I want
to access (by grepping the contents of the database files, I can see some
unique text strings that are associated just with missing records -- of
course they do, since the files haven't been modified for months).
I cannot think of anything that changed between now and the last time
that I successfully was able to access the records, except that the DB
process crashed once. It was brought back up immediately. Although, as I
say, it doesn't appear that anything modified the files associated with the
database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJCY4%2Bvp8Rzoq1f0eF0mM1f_WO5pVYeYp8MswMETy%2B2KFQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
t***@gmail.com
2015-08-19 22:14:04 UTC
Permalink
Thanks for the suggestion. The logs don't appear to show any removals on
that database since Oct. 2014. At least as far as these logs go, it
doesn't look to me like a remove was done.

Any suggestions for what to do now? I can definitely see traces of the
data I want to get at in the (unmodified) database files. Is there no way
to get at the data in them?

Thanks!
T
Post by Asya Kamsky
The only CRUD operations that would be logged would be those that take
longer than 100ms.
2015-07-14T12:10:34.559-0500 I WRITE [conn4822] remove blackbox.bigcoll
query: { _id: { $gte: 10000.0 } } ndeleted:90000 keyUpdates:0
{ acquireCount: { w: 12331 } } } 249168ms
Of course this was a huge mass remove of giant docs which is why it took
so long. If the number of documents deleted was small (like if they were
deleted one at a time instead of via single command) then it's much less
likely they would have taken longer than 100ms.
grep 'remove <dbname>' *.log.*
where you would replace <dbname> with the name of your actual DB and
replace *.log.* with path to your log files.
Asya
Post by t***@gmail.com
Post by Asya Kamsky
Do you have the logs from around the time the database crashed? When
was that relative to when you noticed the records missing?
Yes I have all the logs going back for a year. I noticed the records
missing the day that I wrote the help question originally (several days
ago). The crash happened several weeks ago.
Since you had journaling on, and you were able to restart mongod without
Post by Asya Kamsky
any errors, I would rule out the crash *causing* records to disappear,
which leaves you with the possibility that they were deleted. If they
were deleted and you don't have any backups from *before* they were
deleted, then there's not much that can be done to recover the data - sure,
it's possible to write code to literally look through the deleted list, but
if any data was inserted after the deletes then that space would be reused
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But wouldn't there be
some record in the logs if there was a deleation?
We haven't written any records since the deletes. (The files haven't
changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm using
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of records shown
-- not 250, like I said before, that was a mistake.)
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1),
(u'uploadDate', -1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used to manage a
gridfs instance.
Also I think I have some additional information. I copied the
database files to a new location and ran a full repair on the whole
database. IN the new repaired version, there were many fewer files .ns
files. As is as if some records were removed using e.g.
gridfs.GridFS.remove ... so in the new (compacted) version the data was
finally deleted. Maybe its somehow possible that the records were removed
from the gridfs collection by someone running a remove operation? I don't
see any evidence of such an operation in the logs but perhaps it's
possible? I've looked carefully at the logs and am not seeing anything
obvious, but maybe dont' know what to look for.
Of course, that I guess does not modify the underlying files on the
original collection where the (hypothetical) remove would have been run.
If this indeed the case, is there some way to get the data back? After
all, I do have the original .ns files.
Thanks!
Tob
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the investigation of
1) Describe your MongoDB deployment (standalone, replica set, or
sharded cluster)
2) What version of MongoDB are you running? (You can check it with
db.version()
<http://docs.mongodb.org/manual/reference/method/db.version> in the
mongo shell)
3) Which query did you use to find or count the documents? Can you
please run that query with explain()
<http://docs.mongodb.org/manual/reference/method/cursor.explain/> and
send us the output?
4) Output of db.collection.getIndexes()
<http://docs.mongodb.org/manual/reference/method/db.collection.getIndexes/>
for the collection where documents appear to be missing.
Regards,
ankit
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what is the
structure of the docs, are there indexes on this collection?
Post by t***@gmail.com
I have a mongodb database containing a collection that has remained
unchanged for a period of time -- the underlying .ns and .0, .1 etc files
have not been modified for months. Up until a few weeks ago, I had no
problem reading records from the collection. There were thousands of
records in the collection.
However, today when I attempted to read the records, many of them
appeared missing -- e.g. records that I expected to be there were not,
although some of the records were available. Now, there appear to be only
250 records. =
I copied the database files and did a repair of the (copied)
database, which ran fine apparently, but only the 250 records appeared, and
cannot find the others.
I can see manually that the bSON binary files in the database
directory associated with this database still contain the data that I want
to access (by grepping the contents of the database files, I can see some
unique text strings that are associated just with missing records -- of
course they do, since the files haven't been modified for months).
I cannot think of anything that changed between now and the last
time that I successfully was able to access the records, except that the DB
process crashed once. It was brought back up immediately. Although, as I
say, it doesn't appear that anything modified the files associated with the
database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-08-21 13:49:21 UTC
Permalink
Some of the records are in a free list and haven't been overwritten yet.
It may be possible to get them back.

Since you're using version 2.4.x I believe that there is code that will
walk through the DB files and dump out every document it finds there.

Let me dig around for it - obviously there is (a) no guarantee that this
will restore all of them and (b) did you say these were GridFS chunks - if
so then you won't get back a valid file unless all chunks from that file
are restored (along with corresponding files record).

Asya
Post by t***@gmail.com
Thanks for the suggestion. The logs don't appear to show any removals on
that database since Oct. 2014. At least as far as these logs go, it
doesn't look to me like a remove was done.
Any suggestions for what to do now? I can definitely see traces of the
data I want to get at in the (unmodified) database files. Is there no way
to get at the data in them?
Thanks!
T
Post by Asya Kamsky
The only CRUD operations that would be logged would be those that take
longer than 100ms.
2015-07-14T12:10:34.559-0500 I WRITE [conn4822] remove
blackbox.bigcoll query: { _id: { $gte: 10000.0 } } ndeleted:90000
keyUpdates:0 writeConflicts:0 numYields:12330 locks:{ Global: {
12331 } }, Collection: { acquireCount: { w: 12331 } } } 249168ms
Of course this was a huge mass remove of giant docs which is why it took
so long. If the number of documents deleted was small (like if they were
deleted one at a time instead of via single command) then it's much less
likely they would have taken longer than 100ms.
grep 'remove <dbname>' *.log.*
where you would replace <dbname> with the name of your actual DB and
replace *.log.* with path to your log files.
Asya
Post by t***@gmail.com
Post by Asya Kamsky
Do you have the logs from around the time the database crashed? When
was that relative to when you noticed the records missing?
Yes I have all the logs going back for a year. I noticed the records
missing the day that I wrote the help question originally (several days
ago). The crash happened several weeks ago.
Since you had journaling on, and you were able to restart mongod without
Post by Asya Kamsky
any errors, I would rule out the crash *causing* records to disappear,
which leaves you with the possibility that they were deleted. If they
were deleted and you don't have any backups from *before* they were
deleted, then there's not much that can be done to recover the data - sure,
it's possible to write code to literally look through the deleted list, but
if any data was inserted after the deletes then that space would be reused
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But wouldn't there be
some record in the logs if there was a deleation?
We haven't written any records since the deletes. (The files haven't
changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm using
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of records
shown -- not 250, like I said before, that was a mistake.)
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1),
(u'uploadDate', -1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used to manage a
gridfs instance.
Also I think I have some additional information. I copied the
database files to a new location and ran a full repair on the whole
database. IN the new repaired version, there were many fewer files .ns
files. As is as if some records were removed using e.g.
gridfs.GridFS.remove ... so in the new (compacted) version the data was
finally deleted. Maybe its somehow possible that the records were removed
from the gridfs collection by someone running a remove operation? I don't
see any evidence of such an operation in the logs but perhaps it's
possible? I've looked carefully at the logs and am not seeing anything
obvious, but maybe dont' know what to look for.
Of course, that I guess does not modify the underlying files on the
original collection where the (hypothetical) remove would have been run.
If this indeed the case, is there some way to get the data back? After
all, I do have the original .ns files.
Thanks!
Tob
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the investigation of
1) Describe your MongoDB deployment (standalone, replica set, or
sharded cluster)
2) What version of MongoDB are you running? (You can check it with
db.version()
<http://docs.mongodb.org/manual/reference/method/db.version> in the
mongo shell)
3) Which query did you use to find or count the documents? Can you
please run that query with explain()
<http://docs.mongodb.org/manual/reference/method/cursor.explain/> and
send us the output?
4) Output of db.collection.getIndexes()
<http://docs.mongodb.org/manual/reference/method/db.collection.getIndexes/>
for the collection where documents appear to be missing.
Regards,
ankit
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what is the
structure of the docs, are there indexes on this collection?
Post by t***@gmail.com
I have a mongodb database containing a collection that has remained
unchanged for a period of time -- the underlying .ns and .0, .1 etc files
have not been modified for months. Up until a few weeks ago, I had no
problem reading records from the collection. There were thousands of
records in the collection.
However, today when I attempted to read the records, many of them
appeared missing -- e.g. records that I expected to be there were not,
although some of the records were available. Now, there appear to be only
250 records. =
I copied the database files and did a repair of the (copied)
database, which ran fine apparently, but only the 250 records appeared, and
cannot find the others.
I can see manually that the bSON binary files in the database
directory associated with this database still contain the data that I want
to access (by grepping the contents of the database files, I can see some
unique text strings that are associated just with missing records -- of
course they do, since the files haven't been modified for months).
I cannot think of anything that changed between now and the last
time that I successfully was able to access the records, except that the DB
process crashed once. It was brought back up immediately. Although, as I
say, it doesn't appear that anything modified the files associated with the
database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJAPPXWWya3yZLBz2V0FuVOD3%2B8cFYEK9-5XQ6RKTDjhbg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
t***@gmail.com
2015-08-21 15:33:02 UTC
Permalink
Great thanks so much for the help!
T
Post by Asya Kamsky
Some of the records are in a free list and haven't been overwritten yet.
It may be possible to get them back.
Since you're using version 2.4.x I believe that there is code that will
walk through the DB files and dump out every document it finds there.
Let me dig around for it - obviously there is (a) no guarantee that this
will restore all of them and (b) did you say these were GridFS chunks - if
so then you won't get back a valid file unless all chunks from that file
are restored (along with corresponding files record).
Asya
Post by t***@gmail.com
Thanks for the suggestion. The logs don't appear to show any removals
on that database since Oct. 2014. At least as far as these logs go, it
doesn't look to me like a remove was done.
Any suggestions for what to do now? I can definitely see traces of the
data I want to get at in the (unmodified) database files. Is there no way
to get at the data in them?
Thanks!
T
Post by Asya Kamsky
The only CRUD operations that would be logged would be those that take
longer than 100ms.
2015-07-14T12:10:34.559-0500 I WRITE [conn4822] remove
blackbox.bigcoll query: { _id: { $gte: 10000.0 } } ndeleted:90000
keyUpdates:0 writeConflicts:0 numYields:12330 locks:{ Global: {
12331 } }, Collection: { acquireCount: { w: 12331 } } } 249168ms
Of course this was a huge mass remove of giant docs which is why it took
so long. If the number of documents deleted was small (like if they were
deleted one at a time instead of via single command) then it's much less
likely they would have taken longer than 100ms.
grep 'remove <dbname>' *.log.*
where you would replace <dbname> with the name of your actual DB and
replace *.log.* with path to your log files.
Asya
Post by t***@gmail.com
Post by Asya Kamsky
Do you have the logs from around the time the database crashed? When
was that relative to when you noticed the records missing?
Yes I have all the logs going back for a year. I noticed the records
missing the day that I wrote the help question originally (several days
ago). The crash happened several weeks ago.
Since you had journaling on, and you were able to restart mongod
Post by Asya Kamsky
without any errors, I would rule out the crash *causing* records to
disappear, which leaves you with the possibility that they were deleted.
If they were deleted and you don't have any backups from *before* they were
deleted, then there's not much that can be done to recover the data - sure,
it's possible to write code to literally look through the deleted list, but
if any data was inserted after the deletes then that space would be reused
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But wouldn't there be
some record in the logs if there was a deleation?
We haven't written any records since the deletes. (The files haven't
changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm using
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of records
shown -- not 250, like I said before, that was a mistake.)
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1),
(u'uploadDate', -1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used to manage a
gridfs instance.
Also I think I have some additional information. I copied the
database files to a new location and ran a full repair on the whole
database. IN the new repaired version, there were many fewer files .ns
files. As is as if some records were removed using e.g.
gridfs.GridFS.remove ... so in the new (compacted) version the data was
finally deleted. Maybe its somehow possible that the records were removed
from the gridfs collection by someone running a remove operation? I don't
see any evidence of such an operation in the logs but perhaps it's
possible? I've looked carefully at the logs and am not seeing anything
obvious, but maybe dont' know what to look for.
Of course, that I guess does not modify the underlying files on the
original collection where the (hypothetical) remove would have been run.
If this indeed the case, is there some way to get the data back? After
all, I do have the original .ns files.
Thanks!
Tob
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the investigation of
1) Describe your MongoDB deployment (standalone, replica set, or
sharded cluster)
2) What version of MongoDB are you running? (You can check it with
db.version()
<http://docs.mongodb.org/manual/reference/method/db.version> in the
mongo shell)
3) Which query did you use to find or count the documents? Can you
please run that query with explain()
<http://docs.mongodb.org/manual/reference/method/cursor.explain/> and
send us the output?
4) Output of db.collection.getIndexes()
<http://docs.mongodb.org/manual/reference/method/db.collection.getIndexes/>
for the collection where documents appear to be missing.
Regards,
ankit
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what is the
structure of the docs, are there indexes on this collection?
Post by t***@gmail.com
I have a mongodb database containing a collection that has
remained unchanged for a period of time -- the underlying .ns and .0, .1
etc files have not been modified for months. Up until a few weeks ago, I
had no problem reading records from the collection. There were thousands of
records in the collection.
However, today when I attempted to read the records, many of them
appeared missing -- e.g. records that I expected to be there were not,
although some of the records were available. Now, there appear to be only
250 records. =
I copied the database files and did a repair of the (copied)
database, which ran fine apparently, but only the 250 records appeared, and
cannot find the others.
I can see manually that the bSON binary files in the database
directory associated with this database still contain the data that I want
to access (by grepping the contents of the database files, I can see some
unique text strings that are associated just with missing records -- of
course they do, since the files haven't been modified for months).
I cannot think of anything that changed between now and the last
time that I successfully was able to access the records, except that the DB
process crashed once. It was brought back up immediately. Although, as I
say, it doesn't appear that anything modified the files associated with the
database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-08-21 18:02:06 UTC
Permalink
I'm assuming that you start up mongod with the original files (ones you did
not repair but saved when you first noticed that something was amiss):

Okay, first, it would be good to know which collections are missing
documents:

You'd mentioned having 279 documents in files collection - how many are in
chunks collection? Do they "cross-verify"?

You can see what each has here:
http://docs.mongodb.org/manual/reference/gridfs/

So you can tell for each file how many chunk documents there should be and
for each chunk document you can tell which file it belongs to and in what
order.

Some possible ways to verify this:
db.fs.chunks.aggregate({$sort:{files_id:1, num:1}},
{$group:{_id:"$files_id",count:{$sum:1}, chunks:{$push:"$num"}}}).

If you have 279 files in your fs.files collection, then you should get back
279 documents from this aggregation, each corresponding to a "file" in
gridFS.

How many files do you expect to have?

You can now run this command (very slow):

db.fs.files.validate(true)


and
db.fs.chunks.validate(true)

This will give output including fields described here:
http://docs.mongodb.org/manual/reference/command/validate/#output

You are interested in the deleted records (
http://docs.mongodb.org/manual/reference/command/validate/#validate.deletedCount
and the next field, deletedSize).
Basically these are the records that can be potentially recovered - can you
provide output to above experiments so we don't waste time trying to
recover data that's not still there?

Asya
Post by t***@gmail.com
Great thanks so much for the help!
T
Post by Asya Kamsky
Some of the records are in a free list and haven't been overwritten yet.
It may be possible to get them back.
Since you're using version 2.4.x I believe that there is code that will
walk through the DB files and dump out every document it finds there.
Let me dig around for it - obviously there is (a) no guarantee that this
will restore all of them and (b) did you say these were GridFS chunks - if
so then you won't get back a valid file unless all chunks from that file
are restored (along with corresponding files record).
Asya
Post by t***@gmail.com
Thanks for the suggestion. The logs don't appear to show any removals
on that database since Oct. 2014. At least as far as these logs go, it
doesn't look to me like a remove was done.
Any suggestions for what to do now? I can definitely see traces of the
data I want to get at in the (unmodified) database files. Is there no way
to get at the data in them?
Thanks!
T
Post by Asya Kamsky
The only CRUD operations that would be logged would be those that take
longer than 100ms.
2015-07-14T12:10:34.559-0500 I WRITE [conn4822] remove
blackbox.bigcoll query: { _id: { $gte: 10000.0 } } ndeleted:90000
keyUpdates:0 writeConflicts:0 numYields:12330 locks:{ Global: {
12331 } }, Collection: { acquireCount: { w: 12331 } } } 249168ms
Of course this was a huge mass remove of giant docs which is why it
took so long. If the number of documents deleted was small (like if they
were deleted one at a time instead of via single command) then it's much
less likely they would have taken longer than 100ms.
grep 'remove <dbname>' *.log.*
where you would replace <dbname> with the name of your actual DB and
replace *.log.* with path to your log files.
Asya
Post by t***@gmail.com
Post by Asya Kamsky
Do you have the logs from around the time the database crashed?
When was that relative to when you noticed the records missing?
Yes I have all the logs going back for a year. I noticed the records
missing the day that I wrote the help question originally (several days
ago). The crash happened several weeks ago.
Since you had journaling on, and you were able to restart mongod
Post by Asya Kamsky
without any errors, I would rule out the crash *causing* records to
disappear, which leaves you with the possibility that they were deleted.
If they were deleted and you don't have any backups from *before* they were
deleted, then there's not much that can be done to recover the data - sure,
it's possible to write code to literally look through the deleted list, but
if any data was inserted after the deletes then that space would be reused
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But wouldn't there be
some record in the logs if there was a deleation?
We haven't written any records since the deletes. (The files haven't
changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm using
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of records
shown -- not 250, like I said before, that was a mistake.)
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1),
(u'uploadDate', -1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used to manage a
gridfs instance.
Also I think I have some additional information. I copied the
database files to a new location and ran a full repair on the whole
database. IN the new repaired version, there were many fewer files .ns
files. As is as if some records were removed using e.g.
gridfs.GridFS.remove ... so in the new (compacted) version the data was
finally deleted. Maybe its somehow possible that the records were removed
from the gridfs collection by someone running a remove operation? I don't
see any evidence of such an operation in the logs but perhaps it's
possible? I've looked carefully at the logs and am not seeing anything
obvious, but maybe dont' know what to look for.
Of course, that I guess does not modify the underlying files on the
original collection where the (hypothetical) remove would have been run.
If this indeed the case, is there some way to get the data back? After
all, I do have the original .ns files.
Thanks!
Tob
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the investigation of
1) Describe your MongoDB deployment (standalone, replica set, or
sharded cluster)
2) What version of MongoDB are you running? (You can check it with
db.version()
<http://docs.mongodb.org/manual/reference/method/db.version> in
the mongo shell)
3) Which query did you use to find or count the documents? Can you
please run that query with explain()
<http://docs.mongodb.org/manual/reference/method/cursor.explain/> and
send us the output?
4) Output of db.collection.getIndexes()
<http://docs.mongodb.org/manual/reference/method/db.collection.getIndexes/>
for the collection where documents appear to be missing.
Regards,
ankit
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what is the
structure of the docs, are there indexes on this collection?
Post by t***@gmail.com
I have a mongodb database containing a collection that has
remained unchanged for a period of time -- the underlying .ns and .0, .1
etc files have not been modified for months. Up until a few weeks ago, I
had no problem reading records from the collection. There were thousands of
records in the collection.
However, today when I attempted to read the records, many of them
appeared missing -- e.g. records that I expected to be there were not,
although some of the records were available. Now, there appear to be only
250 records. =
I copied the database files and did a repair of the (copied)
database, which ran fine apparently, but only the 250 records appeared, and
cannot find the others.
I can see manually that the bSON binary files in the database
directory associated with this database still contain the data that I want
to access (by grepping the contents of the database files, I can see some
unique text strings that are associated just with missing records -- of
course they do, since the files haven't been modified for months).
I cannot think of anything that changed between now and the last
time that I successfully was able to access the records, except that the DB
process crashed once. It was brought back up immediately. Although, as I
say, it doesn't appear that anything modified the files associated with the
database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJC369fjTi%3DgjGK0ygtP-ra%3DGv%3DjP3Hzg%3DRfwRKHD7UKfA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
t***@gmail.com
2015-08-22 01:30:55 UTC
Permalink
Asya,

Thanks for the help. Ok, here are the answers to your questions:

1) The aggregation of the chunk collection produces 279 documents. I
don't recall how many there were before the ones I want went missing.
However, I guess there more than 279. I don't know exactly how many more,
but I expect on the order of several hundred at least (but again, I'm not
sure). The collection does verify, in that all the chunks in the chunks
collection are accounted for by a single file in the files collection. I
computed the expected number of chunks for each file by dividing length by
chunk size for each file record; that number exactly equals the count of
chunks for that file as produced by the aggregation in the chunk collection.

2) Ok, here's the info from the validate commands:

db.files.validate(true) produced:

{
"ns" : "mydb.myfs.files",
"firstExtent" : "0:19000 ns:mydb.myfs.files",
"lastExtent" : "7:67757000 ns:mydb.myfs.files",
"extentCount" : 5,

<snip "extents">

"datasize" : 24075568,
"nrecords" : 279,
"lastExtentSize" : 12902400,
"padding" : 1,

<snip some other stuff>

"objectsFound" : 279,
"invalidObjects" : 0,
"bytesWithHeaders" : 24080032,
"bytesWithoutHeaders" : 24075568,
"deletedCount" : 3,
"deletedSize" : 12008944,
"nIndexes" : 3,
"keysPerIndex" : {
"mydb.myfs.files.$_id_" : 279,
"mydb.myfs.files.$filename_1_uploadDate_-1" : 279,
"mydb.myfs.files.$timestamp_1" : 279
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}

db.chunks.validate(true) produced:

db.chunks.validate(true)
{
"ns" : "mydb.myfs.chunks",
"firstExtent" : "0:5000 ns:mydb.myfs.chunks",
"lastExtent" : "28:2000 ns:mydb.myfs.chunks",
"extentCount" : 43,

<snip "extents">

"datasize" : 49854830688,
"nrecords" : 190427,
"lastExtentSize" : 2146426864,
"padding" : 1,
"firstExtentDetails" : {
"loc" : "0:5000",
"xnext" : "0:162000",
"xprev" : "null",
"nsdiag" : "mydb.myfs.chunks",
"size" : 8192,
"firstRecord" : "0:50b0",
"lastRecord" : "0:6f28"
},
"lastExtentDetails" : {
"loc" : "28:2000",
"xnext" : "null",
"xprev" : "27:2000",
"nsdiag" : "mydb.myfs.chunks",
"size" : 2146426864,
"firstRecord" : "28:20b0",
"lastRecord" : "28:189c20b0"
},
"objectsFound" : 190427,
"invalidObjects" : 0,
"bytesWithHeaders" : 49857877520,
"bytesWithoutHeaders" : 49854830688,
"deletedCount" : 20,
"deletedSize" : 1733626640,
"nIndexes" : 2,
"keysPerIndex" : {
"mydb.myfs.chunks.$_id_" : 190427,
"mydb.myfs.chunks.$files_id_1_n_1" : 190427
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}

I'm not sure whether "deletedCount" of "3" in the files collection would
make up for all the deleted records that I expect ... if I interpret it
correctly. It would also not make up for the huge compaction that
occurred in the repaired version of the collection (which is much smaller
on disk, but still contains 279 records).

Any thoughts on what to do next? Thanks again for your help!

T
Post by Asya Kamsky
I'm assuming that you start up mongod with the original files (ones you
Okay, first, it would be good to know which collections are missing
You'd mentioned having 279 documents in files collection - how many are in
chunks collection? Do they "cross-verify"?
http://docs.mongodb.org/manual/reference/gridfs/
So you can tell for each file how many chunk documents there should be and
for each chunk document you can tell which file it belongs to and in what
order.
db.fs.chunks.aggregate({$sort:{files_id:1, num:1}},
{$group:{_id:"$files_id",count:{$sum:1}, chunks:{$push:"$num"}}}).
If you have 279 files in your fs.files collection, then you should get
back 279 documents from this aggregation, each corresponding to a "file" in
gridFS.
How many files do you expect to have?
db.fs.files.validate(true)
and
db.fs.chunks.validate(true)
http://docs.mongodb.org/manual/reference/command/validate/#output
You are interested in the deleted records (
http://docs.mongodb.org/manual/reference/command/validate/#validate.deletedCount
and the next field, deletedSize).
Basically these are the records that can be potentially recovered - can
you provide output to above experiments so we don't waste time trying to
recover data that's not still there?
Asya
Post by t***@gmail.com
Great thanks so much for the help!
T
Post by Asya Kamsky
Some of the records are in a free list and haven't been overwritten
yet. It may be possible to get them back.
Since you're using version 2.4.x I believe that there is code that will
walk through the DB files and dump out every document it finds there.
Let me dig around for it - obviously there is (a) no guarantee that this
will restore all of them and (b) did you say these were GridFS chunks - if
so then you won't get back a valid file unless all chunks from that file
are restored (along with corresponding files record).
Asya
Post by t***@gmail.com
Thanks for the suggestion. The logs don't appear to show any removals
on that database since Oct. 2014. At least as far as these logs go, it
doesn't look to me like a remove was done.
Any suggestions for what to do now? I can definitely see traces of
the data I want to get at in the (unmodified) database files. Is there no
way to get at the data in them?
Thanks!
T
Post by Asya Kamsky
The only CRUD operations that would be logged would be those that take
longer than 100ms.
2015-07-14T12:10:34.559-0500 I WRITE [conn4822] remove
blackbox.bigcoll query: { _id: { $gte: 10000.0 } } ndeleted:90000
keyUpdates:0 writeConflicts:0 numYields:12330 locks:{ Global: {
12331 } }, Collection: { acquireCount: { w: 12331 } } } 249168ms
Of course this was a huge mass remove of giant docs which is why it
took so long. If the number of documents deleted was small (like if they
were deleted one at a time instead of via single command) then it's much
less likely they would have taken longer than 100ms.
grep 'remove <dbname>' *.log.*
where you would replace <dbname> with the name of your actual DB and
replace *.log.* with path to your log files.
Asya
Post by t***@gmail.com
Post by Asya Kamsky
Do you have the logs from around the time the database crashed?
When was that relative to when you noticed the records missing?
Yes I have all the logs going back for a year. I noticed the
records missing the day that I wrote the help question originally (several
days ago). The crash happened several weeks ago.
Since you had journaling on, and you were able to restart mongod
Post by Asya Kamsky
without any errors, I would rule out the crash *causing* records to
disappear, which leaves you with the possibility that they were deleted.
If they were deleted and you don't have any backups from *before* they were
deleted, then there's not much that can be done to recover the data - sure,
it's possible to write code to literally look through the deleted list, but
if any data was inserted after the deletes then that space would be reused
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But wouldn't there
be some record in the logs if there was a deleation?
We haven't written any records since the deletes. (The files
haven't changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm using
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of records
shown -- not 250, like I said before, that was a mistake.)
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1),
(u'uploadDate', -1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used to manage a
gridfs instance.
Also I think I have some additional information. I copied the
database files to a new location and ran a full repair on the whole
database. IN the new repaired version, there were many fewer files .ns
files. As is as if some records were removed using e.g.
gridfs.GridFS.remove ... so in the new (compacted) version the data was
finally deleted. Maybe its somehow possible that the records were removed
from the gridfs collection by someone running a remove operation? I don't
see any evidence of such an operation in the logs but perhaps it's
possible? I've looked carefully at the logs and am not seeing anything
obvious, but maybe dont' know what to look for.
Of course, that I guess does not modify the underlying files on the
original collection where the (hypothetical) remove would have been run.
If this indeed the case, is there some way to get the data back? After
all, I do have the original .ns files.
Thanks!
Tob
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the investigation
1) Describe your MongoDB deployment (standalone, replica set, or
sharded cluster)
2) What version of MongoDB are you running? (You can check it with
db.version()
<http://docs.mongodb.org/manual/reference/method/db.version> in
the mongo shell)
3) Which query did you use to find or count the documents? Can you
please run that query with explain()
<http://docs.mongodb.org/manual/reference/method/cursor.explain/> and
send us the output?
4) Output of db.collection.getIndexes()
<http://docs.mongodb.org/manual/reference/method/db.collection.getIndexes/>
for the collection where documents appear to be missing.
Regards,
ankit
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what is the
structure of the docs, are there indexes on this collection?
Post by t***@gmail.com
I have a mongodb database containing a collection that has
remained unchanged for a period of time -- the underlying .ns and .0, .1
etc files have not been modified for months. Up until a few weeks ago, I
had no problem reading records from the collection. There were thousands of
records in the collection.
However, today when I attempted to read the records, many of
them appeared missing -- e.g. records that I expected to be there were not,
although some of the records were available. Now, there appear to be only
250 records. =
I copied the database files and did a repair of the (copied)
database, which ran fine apparently, but only the 250 records appeared, and
cannot find the others.
I can see manually that the bSON binary files in the database
directory associated with this database still contain the data that I want
to access (by grepping the contents of the database files, I can see some
unique text strings that are associated just with missing records -- of
course they do, since the files haven't been modified for months).
I cannot think of anything that changed between now and the last
time that I successfully was able to access the records, except that the DB
process crashed once. It was brought back up immediately. Although, as I
say, it doesn't appear that anything modified the files associated with the
database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/624bfdbb-2e6d-4640-9976-5741c6a4724d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-08-22 23:00:42 UTC
Permalink
This is somewhat surprising. Are there other collections in this database?

You're correct, the deleted number *is* the number of deleted records and
that's pretty much the upper bound on what you could conceivably recover
out of these files.

This is the original files, right? I'm a bit surprised at relatively
large number of extents in the files collection relative to number of
deleted records. Also normally there would be an array of deleted records
as they are kept in "groups" - I guess that's for regular collections, in
gridFS the sizes of records would be more similar...

Are you *sure* there are files missing? I don't mean by count but by
content... You mentioned you were about to find some strings matching
something you expected to find in the DB but didn't?

How many dbname.ns files are there and what are their sizes?

Asya
Post by t***@gmail.com
Asya,
1) The aggregation of the chunk collection produces 279 documents. I
don't recall how many there were before the ones I want went missing.
However, I guess there more than 279. I don't know exactly how many more,
but I expect on the order of several hundred at least (but again, I'm not
sure). The collection does verify, in that all the chunks in the chunks
collection are accounted for by a single file in the files collection. I
computed the expected number of chunks for each file by dividing length by
chunk size for each file record; that number exactly equals the count of
chunks for that file as produced by the aggregation in the chunk collection.
{
"ns" : "mydb.myfs.files",
"firstExtent" : "0:19000 ns:mydb.myfs.files",
"lastExtent" : "7:67757000 ns:mydb.myfs.files",
"extentCount" : 5,
<snip "extents">
"datasize" : 24075568,
"nrecords" : 279,
"lastExtentSize" : 12902400,
"padding" : 1,
<snip some other stuff>
"objectsFound" : 279,
"invalidObjects" : 0,
"bytesWithHeaders" : 24080032,
"bytesWithoutHeaders" : 24075568,
"deletedCount" : 3,
"deletedSize" : 12008944,
"nIndexes" : 3,
"keysPerIndex" : {
"mydb.myfs.files.$_id_" : 279,
"mydb.myfs.files.$filename_1_uploadDate_-1" : 279,
"mydb.myfs.files.$timestamp_1" : 279
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
db.chunks.validate(true)
{
"ns" : "mydb.myfs.chunks",
"firstExtent" : "0:5000 ns:mydb.myfs.chunks",
"lastExtent" : "28:2000 ns:mydb.myfs.chunks",
"extentCount" : 43,
<snip "extents">
"datasize" : 49854830688,
"nrecords" : 190427,
"lastExtentSize" : 2146426864,
"padding" : 1,
"firstExtentDetails" : {
"loc" : "0:5000",
"xnext" : "0:162000",
"xprev" : "null",
"nsdiag" : "mydb.myfs.chunks",
"size" : 8192,
"firstRecord" : "0:50b0",
"lastRecord" : "0:6f28"
},
"lastExtentDetails" : {
"loc" : "28:2000",
"xnext" : "null",
"xprev" : "27:2000",
"nsdiag" : "mydb.myfs.chunks",
"size" : 2146426864,
"firstRecord" : "28:20b0",
"lastRecord" : "28:189c20b0"
},
"objectsFound" : 190427,
"invalidObjects" : 0,
"bytesWithHeaders" : 49857877520,
"bytesWithoutHeaders" : 49854830688,
"deletedCount" : 20,
"deletedSize" : 1733626640,
"nIndexes" : 2,
"keysPerIndex" : {
"mydb.myfs.chunks.$_id_" : 190427,
"mydb.myfs.chunks.$files_id_1_n_1" : 190427
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
I'm not sure whether "deletedCount" of "3" in the files collection would
make up for all the deleted records that I expect ... if I interpret it
correctly. It would also not make up for the huge compaction that
occurred in the repaired version of the collection (which is much smaller
on disk, but still contains 279 records).
Any thoughts on what to do next? Thanks again for your help!
T
Post by Asya Kamsky
I'm assuming that you start up mongod with the original files (ones you
Okay, first, it would be good to know which collections are missing
You'd mentioned having 279 documents in files collection - how many are
in chunks collection? Do they "cross-verify"?
http://docs.mongodb.org/manual/reference/gridfs/
So you can tell for each file how many chunk documents there should be
and for each chunk document you can tell which file it belongs to and in
what order.
db.fs.chunks.aggregate({$sort:{files_id:1, num:1}},
{$group:{_id:"$files_id",count:{$sum:1}, chunks:{$push:"$num"}}}).
If you have 279 files in your fs.files collection, then you should get
back 279 documents from this aggregation, each corresponding to a "file" in
gridFS.
How many files do you expect to have?
db.fs.files.validate(true)
and
db.fs.chunks.validate(true)
http://docs.mongodb.org/manual/reference/command/validate/#output
You are interested in the deleted records (
http://docs.mongodb.org/manual/reference/command/validate/#validate.deletedCount
and the next field, deletedSize).
Basically these are the records that can be potentially recovered - can
you provide output to above experiments so we don't waste time trying to
recover data that's not still there?
Asya
Post by t***@gmail.com
Great thanks so much for the help!
T
Post by Asya Kamsky
Some of the records are in a free list and haven't been overwritten
yet. It may be possible to get them back.
Since you're using version 2.4.x I believe that there is code that will
walk through the DB files and dump out every document it finds there.
Let me dig around for it - obviously there is (a) no guarantee that
this will restore all of them and (b) did you say these were GridFS chunks
- if so then you won't get back a valid file unless all chunks from that
file are restored (along with corresponding files record).
Asya
Post by t***@gmail.com
Thanks for the suggestion. The logs don't appear to show any
removals on that database since Oct. 2014. At least as far as these logs
go, it doesn't look to me like a remove was done.
Any suggestions for what to do now? I can definitely see traces of
the data I want to get at in the (unmodified) database files. Is there no
way to get at the data in them?
Thanks!
T
Post by Asya Kamsky
The only CRUD operations that would be logged would be those that
take longer than 100ms.
2015-07-14T12:10:34.559-0500 I WRITE [conn4822] remove
blackbox.bigcoll query: { _id: { $gte: 10000.0 } } ndeleted:90000
keyUpdates:0 writeConflicts:0 numYields:12330 locks:{ Global: {
12331 } }, Collection: { acquireCount: { w: 12331 } } } 249168ms
Of course this was a huge mass remove of giant docs which is why it
took so long. If the number of documents deleted was small (like if they
were deleted one at a time instead of via single command) then it's much
less likely they would have taken longer than 100ms.
grep 'remove <dbname>' *.log.*
where you would replace <dbname> with the name of your actual DB and
replace *.log.* with path to your log files.
Asya
Post by t***@gmail.com
Post by Asya Kamsky
Do you have the logs from around the time the database crashed?
When was that relative to when you noticed the records missing?
Yes I have all the logs going back for a year. I noticed the
records missing the day that I wrote the help question originally (several
days ago). The crash happened several weeks ago.
Since you had journaling on, and you were able to restart mongod
Post by Asya Kamsky
without any errors, I would rule out the crash *causing* records to
disappear, which leaves you with the possibility that they were deleted.
If they were deleted and you don't have any backups from *before* they were
deleted, then there's not much that can be done to recover the data - sure,
it's possible to write code to literally look through the deleted list, but
if any data was inserted after the deletes then that space would be reused
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But wouldn't there
be some record in the logs if there was a deleation?
We haven't written any records since the deletes. (The files
haven't changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm using
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of records
shown -- not 250, like I said before, that was a mistake.)
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1),
(u'uploadDate', -1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used to manage
a gridfs instance.
Also I think I have some additional information. I copied the
database files to a new location and ran a full repair on the whole
database. IN the new repaired version, there were many fewer files .ns
files. As is as if some records were removed using e.g.
gridfs.GridFS.remove ... so in the new (compacted) version the data was
finally deleted. Maybe its somehow possible that the records were removed
from the gridfs collection by someone running a remove operation? I don't
see any evidence of such an operation in the logs but perhaps it's
possible? I've looked carefully at the logs and am not seeing anything
obvious, but maybe dont' know what to look for.
Of course, that I guess does not modify the underlying files on
the original collection where the (hypothetical) remove would have been
run. If this indeed the case, is there some way to get the data back?
After all, I do have the original .ns files.
Thanks!
Tob
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the investigation
1) Describe your MongoDB deployment (standalone, replica set, or
sharded cluster)
2) What version of MongoDB are you running? (You can check it
with db.version()
<http://docs.mongodb.org/manual/reference/method/db.version> in
the mongo shell)
3) Which query did you use to find or count the documents? Can
you please run that query with explain()
<http://docs.mongodb.org/manual/reference/method/cursor.explain/> and
send us the output?
4) Output of db.collection.getIndexes()
<http://docs.mongodb.org/manual/reference/method/db.collection.getIndexes/>
for the collection where documents appear to be missing.
Regards,
ankit
On Monday, August 17, 2015 at 12:56:20 PM UTC+5:30, Chris De
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what is
the structure of the docs, are there indexes on this collection?
On Sunday, August 16, 2015 at 12:33:16 AM UTC+2,
Post by t***@gmail.com
I have a mongodb database containing a collection that has
remained unchanged for a period of time -- the underlying .ns and .0, .1
etc files have not been modified for months. Up until a few weeks ago, I
had no problem reading records from the collection. There were thousands of
records in the collection.
However, today when I attempted to read the records, many of
them appeared missing -- e.g. records that I expected to be there were not,
although some of the records were available. Now, there appear to be only
250 records. =
I copied the database files and did a repair of the (copied)
database, which ran fine apparently, but only the 250 records appeared, and
cannot find the others.
I can see manually that the bSON binary files in the database
directory associated with this database still contain the data that I want
to access (by grepping the contents of the database files, I can see some
unique text strings that are associated just with missing records -- of
course they do, since the files haven't been modified for months).
I cannot think of anything that changed between now and the
last time that I successfully was able to access the records, except that
the DB process crashed once. It was brought back up immediately. Although,
as I say, it doesn't appear that anything modified the files associated
with the database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/624bfdbb-2e6d-4640-9976-5741c6a4724d%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/624bfdbb-2e6d-4640-9976-5741c6a4724d%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJAaVRxpb-0%3DnHwOO9XWYwHdwWdv03g6y9at_X5jnhjhig%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
t***@gmail.com
2015-08-24 13:48:01 UTC
Permalink
Thanks for the response -- sorry for the delay in reply (was away from
computer).
Post by Asya Kamsky
This is somewhat surprising. Are there other collections in this database?
Well, not really, here is the result of ".collection_names()":

[u'system.indexes', u'myfs.chunks', u'myfs.files']
Post by Asya Kamsky
This is the original files, right?
Yes.
Post by Asya Kamsky
Are you *sure* there are files missing? I don't mean by count but by
content... You mentioned you were about to find some strings matching
something you expected to find in the DB but didn't?
Definitely 100% completely sure. I can use "grep" on the binary files of
this database and I see at least parts of the missing records, e.g. some
keys that are unique and are exactly what the missing records would have.
Moreover, I (and several others) were happily using this collection as the
source a stable restful API for months ... at least, until this problem
occurred ...
Post by Asya Kamsky
How many dbname.ns files are there and what are their sizes?
In the original collection there are 90 .ns files, most of which are the
standard size "2146435072", but some of which (the early ones of course)
are smaller.

In the repaired collection there are 35 .ns files, same size pattern.

Thanks again for all the help --

T
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Asya,
1) The aggregation of the chunk collection produces 279 documents. I
don't recall how many there were before the ones I want went missing.
However, I guess there more than 279. I don't know exactly how many more,
but I expect on the order of several hundred at least (but again, I'm not
sure). The collection does verify, in that all the chunks in the chunks
collection are accounted for by a single file in the files collection. I
computed the expected number of chunks for each file by dividing length by
chunk size for each file record; that number exactly equals the count of
chunks for that file as produced by the aggregation in the chunk collection.
{
"ns" : "mydb.myfs.files",
"firstExtent" : "0:19000 ns:mydb.myfs.files",
"lastExtent" : "7:67757000 ns:mydb.myfs.files",
"extentCount" : 5,
<snip "extents">
"datasize" : 24075568,
"nrecords" : 279,
"lastExtentSize" : 12902400,
"padding" : 1,
<snip some other stuff>
"objectsFound" : 279,
"invalidObjects" : 0,
"bytesWithHeaders" : 24080032,
"bytesWithoutHeaders" : 24075568,
"deletedCount" : 3,
"deletedSize" : 12008944,
"nIndexes" : 3,
"keysPerIndex" : {
"mydb.myfs.files.$_id_" : 279,
"mydb.myfs.files.$filename_1_uploadDate_-1" : 279,
"mydb.myfs.files.$timestamp_1" : 279
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
db.chunks.validate(true)
{
"ns" : "mydb.myfs.chunks",
"firstExtent" : "0:5000 ns:mydb.myfs.chunks",
"lastExtent" : "28:2000 ns:mydb.myfs.chunks",
"extentCount" : 43,
<snip "extents">
"datasize" : 49854830688,
"nrecords" : 190427,
"lastExtentSize" : 2146426864,
"padding" : 1,
"firstExtentDetails" : {
"loc" : "0:5000",
"xnext" : "0:162000",
"xprev" : "null",
"nsdiag" : "mydb.myfs.chunks",
"size" : 8192,
"firstRecord" : "0:50b0",
"lastRecord" : "0:6f28"
},
"lastExtentDetails" : {
"loc" : "28:2000",
"xnext" : "null",
"xprev" : "27:2000",
"nsdiag" : "mydb.myfs.chunks",
"size" : 2146426864,
"firstRecord" : "28:20b0",
"lastRecord" : "28:189c20b0"
},
"objectsFound" : 190427,
"invalidObjects" : 0,
"bytesWithHeaders" : 49857877520,
"bytesWithoutHeaders" : 49854830688,
"deletedCount" : 20,
"deletedSize" : 1733626640,
"nIndexes" : 2,
"keysPerIndex" : {
"mydb.myfs.chunks.$_id_" : 190427,
"mydb.myfs.chunks.$files_id_1_n_1" : 190427
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
I'm not sure whether "deletedCount" of "3" in the files collection would
make up for all the deleted records that I expect ... if I interpret it
correctly. It would also not make up for the huge compaction that
occurred in the repaired version of the collection (which is much smaller
on disk, but still contains 279 records).
Any thoughts on what to do next? Thanks again for your help!
T
Post by Asya Kamsky
I'm assuming that you start up mongod with the original files (ones you
Okay, first, it would be good to know which collections are missing
You'd mentioned having 279 documents in files collection - how many are
in chunks collection? Do they "cross-verify"?
http://docs.mongodb.org/manual/reference/gridfs/
So you can tell for each file how many chunk documents there should be
and for each chunk document you can tell which file it belongs to and in
what order.
db.fs.chunks.aggregate({$sort:{files_id:1, num:1}},
{$group:{_id:"$files_id",count:{$sum:1}, chunks:{$push:"$num"}}}).
If you have 279 files in your fs.files collection, then you should get
back 279 documents from this aggregation, each corresponding to a "file" in
gridFS.
How many files do you expect to have?
db.fs.files.validate(true)
and
db.fs.chunks.validate(true)
http://docs.mongodb.org/manual/reference/command/validate/#output
You are interested in the deleted records (
http://docs.mongodb.org/manual/reference/command/validate/#validate.deletedCount
and the next field, deletedSize).
Basically these are the records that can be potentially recovered - can
you provide output to above experiments so we don't waste time trying to
recover data that's not still there?
Asya
Post by t***@gmail.com
Great thanks so much for the help!
T
Post by Asya Kamsky
Some of the records are in a free list and haven't been overwritten
yet. It may be possible to get them back.
Since you're using version 2.4.x I believe that there is code that
will walk through the DB files and dump out every document it finds there.
Let me dig around for it - obviously there is (a) no guarantee that
this will restore all of them and (b) did you say these were GridFS chunks
- if so then you won't get back a valid file unless all chunks from that
file are restored (along with corresponding files record).
Asya
Post by t***@gmail.com
Thanks for the suggestion. The logs don't appear to show any
removals on that database since Oct. 2014. At least as far as these logs
go, it doesn't look to me like a remove was done.
Any suggestions for what to do now? I can definitely see traces of
the data I want to get at in the (unmodified) database files. Is there no
way to get at the data in them?
Thanks!
T
Post by Asya Kamsky
The only CRUD operations that would be logged would be those that
take longer than 100ms.
2015-07-14T12:10:34.559-0500 I WRITE [conn4822] remove
blackbox.bigcoll query: { _id: { $gte: 10000.0 } } ndeleted:90000
keyUpdates:0 writeConflicts:0 numYields:12330 locks:{ Global: {
12331 } }, Collection: { acquireCount: { w: 12331 } } } 249168ms
Of course this was a huge mass remove of giant docs which is why it
took so long. If the number of documents deleted was small (like if they
were deleted one at a time instead of via single command) then it's much
less likely they would have taken longer than 100ms.
grep 'remove <dbname>' *.log.*
where you would replace <dbname> with the name of your actual DB and
replace *.log.* with path to your log files.
Asya
Post by t***@gmail.com
Post by Asya Kamsky
Do you have the logs from around the time the database crashed?
When was that relative to when you noticed the records missing?
Yes I have all the logs going back for a year. I noticed the
records missing the day that I wrote the help question originally (several
days ago). The crash happened several weeks ago.
Since you had journaling on, and you were able to restart mongod
Post by Asya Kamsky
without any errors, I would rule out the crash *causing* records to
disappear, which leaves you with the possibility that they were deleted.
If they were deleted and you don't have any backups from *before* they were
deleted, then there's not much that can be done to recover the data - sure,
it's possible to write code to literally look through the deleted list, but
if any data was inserted after the deletes then that space would be reused
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But wouldn't there
be some record in the logs if there was a deleation?
We haven't written any records since the deletes. (The files
haven't changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm using
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of records
shown -- not 250, like I said before, that was a mistake.)
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1),
(u'uploadDate', -1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used to manage
a gridfs instance.
Also I think I have some additional information. I copied the
database files to a new location and ran a full repair on the whole
database. IN the new repaired version, there were many fewer files .ns
files. As is as if some records were removed using e.g.
gridfs.GridFS.remove ... so in the new (compacted) version the data was
finally deleted. Maybe its somehow possible that the records were removed
from the gridfs collection by someone running a remove operation? I don't
see any evidence of such an operation in the logs but perhaps it's
possible? I've looked carefully at the logs and am not seeing anything
obvious, but maybe dont' know what to look for.
Of course, that I guess does not modify the underlying files on
the original collection where the (hypothetical) remove would have been
run. If this indeed the case, is there some way to get the data back?
After all, I do have the original .ns files.
Thanks!
Tob
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the investigation
1) Describe your MongoDB deployment (standalone, replica set, or
sharded cluster)
2) What version of MongoDB are you running? (You can check it
with db.version()
<http://docs.mongodb.org/manual/reference/method/db.version> in
the mongo shell)
3) Which query did you use to find or count the documents? Can
you please run that query with explain()
<http://docs.mongodb.org/manual/reference/method/cursor.explain/> and
send us the output?
4) Output of db.collection.getIndexes()
<http://docs.mongodb.org/manual/reference/method/db.collection.getIndexes/>
for the collection where documents appear to be missing.
Regards,
ankit
On Monday, August 17, 2015 at 12:56:20 PM UTC+5:30, Chris De
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what is
the structure of the docs, are there indexes on this collection?
On Sunday, August 16, 2015 at 12:33:16 AM UTC+2,
Post by t***@gmail.com
I have a mongodb database containing a collection that has
remained unchanged for a period of time -- the underlying .ns and .0, .1
etc files have not been modified for months. Up until a few weeks ago, I
had no problem reading records from the collection. There were thousands of
records in the collection.
However, today when I attempted to read the records, many of
them appeared missing -- e.g. records that I expected to be there were not,
although some of the records were available. Now, there appear to be only
250 records. =
I copied the database files and did a repair of the (copied)
database, which ran fine apparently, but only the 250 records appeared, and
cannot find the others.
I can see manually that the bSON binary files in the database
directory associated with this database still contain the data that I want
to access (by grepping the contents of the database files, I can see some
unique text strings that are associated just with missing records -- of
course they do, since the files haven't been modified for months).
I cannot think of anything that changed between now and the
last time that I successfully was able to access the records, except that
the DB process crashed once. It was brought back up immediately. Although,
as I say, it doesn't appear that anything modified the files associated
with the database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the
Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the
Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/624bfdbb-2e6d-4640-9976-5741c6a4724d%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/624bfdbb-2e6d-4640-9976-5741c6a4724d%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/1e0efcf0-c0e3-4cd5-a9fd-7a21844fe44f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-08-24 15:18:47 UTC
Permalink
Okay, well, I guess it can't hurt so you should try to see if anything
dumped out with something that reads the raw files ends up helping at all:

Check out this github repo:

https://github.com/chergert/mdb

This was mostly an exercise in dumping out on-disk format (*note* to anyone
reading, this had never been updated since 2.4 so I'm only suggesting it
specifically because OP is using 2.4 - using this on later versions without
changes will simply give errors).

To build this you will need libbson version from when this was written
which I believe is 1.0. The executable you want to run is called mdbundo,
here's its usage.

usage: mdbundo DBPATH DBNAME COLNAME

what it will do is walk through the DB files looking for deleted records
and then try to restore them. I can't remember for sure but I think it
will send them to stdout so you can redirect the output of this to a file.
That file should have bson in them so you can use bsondump to view them
(which is pointless for chunks but useful for "files") and you can run
mongorestore on the output to load it into a new collection so you can then
figure out if anything in there is valuable.

Obviously you run it twice on the same DB files, once for myfs.files, once
for myfs.chunks.

Again, given the output to validate I don't really expect it to find
anything more than a handful of records at most, possibly unusable (since
for every "file" every chunk belonging to it must be present to have the
full usable file back.

Good luck and let us know what you uncover,
Asya
Post by t***@gmail.com
Thanks for the response -- sorry for the delay in reply (was away from
computer).
Post by Asya Kamsky
This is somewhat surprising. Are there other collections in this database?
[u'system.indexes', u'myfs.chunks', u'myfs.files']
Post by Asya Kamsky
This is the original files, right?
Yes.
Post by Asya Kamsky
Are you *sure* there are files missing? I don't mean by count but by
content... You mentioned you were about to find some strings matching
something you expected to find in the DB but didn't?
Definitely 100% completely sure. I can use "grep" on the binary files
of this database and I see at least parts of the missing records, e.g. some
keys that are unique and are exactly what the missing records would have.
Moreover, I (and several others) were happily using this collection as the
source a stable restful API for months ... at least, until this problem
occurred ...
Post by Asya Kamsky
How many dbname.ns files are there and what are their sizes?
In the original collection there are 90 .ns files, most of which are the
standard size "2146435072", but some of which (the early ones of course)
are smaller.
In the repaired collection there are 35 .ns files, same size pattern.
Thanks again for all the help --
T
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Asya,
1) The aggregation of the chunk collection produces 279 documents. I
don't recall how many there were before the ones I want went missing.
However, I guess there more than 279. I don't know exactly how many more,
but I expect on the order of several hundred at least (but again, I'm not
sure). The collection does verify, in that all the chunks in the chunks
collection are accounted for by a single file in the files collection. I
computed the expected number of chunks for each file by dividing length by
chunk size for each file record; that number exactly equals the count of
chunks for that file as produced by the aggregation in the chunk collection.
{
"ns" : "mydb.myfs.files",
"firstExtent" : "0:19000 ns:mydb.myfs.files",
"lastExtent" : "7:67757000 ns:mydb.myfs.files",
"extentCount" : 5,
<snip "extents">
"datasize" : 24075568,
"nrecords" : 279,
"lastExtentSize" : 12902400,
"padding" : 1,
<snip some other stuff>
"objectsFound" : 279,
"invalidObjects" : 0,
"bytesWithHeaders" : 24080032,
"bytesWithoutHeaders" : 24075568,
"deletedCount" : 3,
"deletedSize" : 12008944,
"nIndexes" : 3,
"keysPerIndex" : {
"mydb.myfs.files.$_id_" : 279,
"mydb.myfs.files.$filename_1_uploadDate_-1" : 279,
"mydb.myfs.files.$timestamp_1" : 279
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
db.chunks.validate(true)
{
"ns" : "mydb.myfs.chunks",
"firstExtent" : "0:5000 ns:mydb.myfs.chunks",
"lastExtent" : "28:2000 ns:mydb.myfs.chunks",
"extentCount" : 43,
<snip "extents">
"datasize" : 49854830688,
"nrecords" : 190427,
"lastExtentSize" : 2146426864,
"padding" : 1,
"firstExtentDetails" : {
"loc" : "0:5000",
"xnext" : "0:162000",
"xprev" : "null",
"nsdiag" : "mydb.myfs.chunks",
"size" : 8192,
"firstRecord" : "0:50b0",
"lastRecord" : "0:6f28"
},
"lastExtentDetails" : {
"loc" : "28:2000",
"xnext" : "null",
"xprev" : "27:2000",
"nsdiag" : "mydb.myfs.chunks",
"size" : 2146426864,
"firstRecord" : "28:20b0",
"lastRecord" : "28:189c20b0"
},
"objectsFound" : 190427,
"invalidObjects" : 0,
"bytesWithHeaders" : 49857877520,
"bytesWithoutHeaders" : 49854830688,
"deletedCount" : 20,
"deletedSize" : 1733626640,
"nIndexes" : 2,
"keysPerIndex" : {
"mydb.myfs.chunks.$_id_" : 190427,
"mydb.myfs.chunks.$files_id_1_n_1" : 190427
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
I'm not sure whether "deletedCount" of "3" in the files collection would
make up for all the deleted records that I expect ... if I interpret it
correctly. It would also not make up for the huge compaction that
occurred in the repaired version of the collection (which is much smaller
on disk, but still contains 279 records).
Any thoughts on what to do next? Thanks again for your help!
T
Post by Asya Kamsky
I'm assuming that you start up mongod with the original files (ones you
Okay, first, it would be good to know which collections are missing
You'd mentioned having 279 documents in files collection - how many are
in chunks collection? Do they "cross-verify"?
http://docs.mongodb.org/manual/reference/gridfs/
So you can tell for each file how many chunk documents there should be
and for each chunk document you can tell which file it belongs to and in
what order.
db.fs.chunks.aggregate({$sort:{files_id:1, num:1}},
{$group:{_id:"$files_id",count:{$sum:1}, chunks:{$push:"$num"}}}).
If you have 279 files in your fs.files collection, then you should get
back 279 documents from this aggregation, each corresponding to a "file" in
gridFS.
How many files do you expect to have?
db.fs.files.validate(true)
and
db.fs.chunks.validate(true)
http://docs.mongodb.org/manual/reference/command/validate/#output
You are interested in the deleted records (
http://docs.mongodb.org/manual/reference/command/validate/#validate.deletedCount
and the next field, deletedSize).
Basically these are the records that can be potentially recovered - can
you provide output to above experiments so we don't waste time trying to
recover data that's not still there?
Asya
Post by t***@gmail.com
Great thanks so much for the help!
T
Post by Asya Kamsky
Some of the records are in a free list and haven't been overwritten
yet. It may be possible to get them back.
Since you're using version 2.4.x I believe that there is code that
will walk through the DB files and dump out every document it finds there.
Let me dig around for it - obviously there is (a) no guarantee that
this will restore all of them and (b) did you say these were GridFS chunks
- if so then you won't get back a valid file unless all chunks from that
file are restored (along with corresponding files record).
Asya
Post by t***@gmail.com
Thanks for the suggestion. The logs don't appear to show any
removals on that database since Oct. 2014. At least as far as these logs
go, it doesn't look to me like a remove was done.
Any suggestions for what to do now? I can definitely see traces of
the data I want to get at in the (unmodified) database files. Is there no
way to get at the data in them?
Thanks!
T
Post by Asya Kamsky
The only CRUD operations that would be logged would be those that
take longer than 100ms.
2015-07-14T12:10:34.559-0500 I WRITE [conn4822] remove
blackbox.bigcoll query: { _id: { $gte: 10000.0 } } ndeleted:90000
keyUpdates:0 writeConflicts:0 numYields:12330 locks:{ Global: {
12331 } }, Collection: { acquireCount: { w: 12331 } } } 249168ms
Of course this was a huge mass remove of giant docs which is why it
took so long. If the number of documents deleted was small (like if they
were deleted one at a time instead of via single command) then it's much
less likely they would have taken longer than 100ms.
grep 'remove <dbname>' *.log.*
where you would replace <dbname> with the name of your actual DB
and replace *.log.* with path to your log files.
Asya
Post by t***@gmail.com
Post by Asya Kamsky
Do you have the logs from around the time the database crashed?
When was that relative to when you noticed the records missing?
Yes I have all the logs going back for a year. I noticed the
records missing the day that I wrote the help question originally (several
days ago). The crash happened several weeks ago.
Since you had journaling on, and you were able to restart mongod
Post by Asya Kamsky
without any errors, I would rule out the crash *causing* records to
disappear, which leaves you with the possibility that they were deleted.
If they were deleted and you don't have any backups from *before* they were
deleted, then there's not much that can be done to recover the data - sure,
it's possible to write code to literally look through the deleted list, but
if any data was inserted after the deletes then that space would be reused
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But wouldn't
there be some record in the logs if there was a deleation?
We haven't written any records since the deletes. (The files
haven't changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm using
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of
records shown -- not 250, like I said before, that was a mistake.)
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1),
(u'uploadDate', -1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used to
manage a gridfs instance.
Also I think I have some additional information. I copied the
database files to a new location and ran a full repair on the whole
database. IN the new repaired version, there were many fewer files .ns
files. As is as if some records were removed using e.g.
gridfs.GridFS.remove ... so in the new (compacted) version the data was
finally deleted. Maybe its somehow possible that the records were removed
from the gridfs collection by someone running a remove operation? I don't
see any evidence of such an operation in the logs but perhaps it's
possible? I've looked carefully at the logs and am not seeing anything
obvious, but maybe dont' know what to look for.
Of course, that I guess does not modify the underlying files on
the original collection where the (hypothetical) remove would have been
run. If this indeed the case, is there some way to get the data back?
After all, I do have the original .ns files.
Thanks!
Tob
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the
1) Describe your MongoDB deployment (standalone, replica set,
or sharded cluster)
2) What version of MongoDB are you running? (You can check it
with db.version()
<http://docs.mongodb.org/manual/reference/method/db.version>
in the mongo shell)
3) Which query did you use to find or count the documents? Can
you please run that query with explain()
<http://docs.mongodb.org/manual/reference/method/cursor.explain/> and
send us the output?
4) Output of db.collection.getIndexes()
<http://docs.mongodb.org/manual/reference/method/db.collection.getIndexes/>
for the collection where documents appear to be missing.
Regards,
ankit
On Monday, August 17, 2015 at 12:56:20 PM UTC+5:30, Chris De
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what is
the structure of the docs, are there indexes on this collection?
On Sunday, August 16, 2015 at 12:33:16 AM UTC+2,
Post by t***@gmail.com
I have a mongodb database containing a collection that has
remained unchanged for a period of time -- the underlying .ns and .0, .1
etc files have not been modified for months. Up until a few weeks ago, I
had no problem reading records from the collection. There were thousands of
records in the collection.
However, today when I attempted to read the records, many of
them appeared missing -- e.g. records that I expected to be there were not,
although some of the records were available. Now, there appear to be only
250 records. =
I copied the database files and did a repair of the (copied)
database, which ran fine apparently, but only the 250 records appeared, and
cannot find the others.
I can see manually that the bSON binary files in the database
directory associated with this database still contain the data that I want
to access (by grepping the contents of the database files, I can see some
unique text strings that are associated just with missing records -- of
course they do, since the files haven't been modified for months).
I cannot think of anything that changed between now and the
last time that I successfully was able to access the records, except that
the DB process crashed once. It was brought back up immediately. Although,
as I say, it doesn't appear that anything modified the files associated
with the database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the
Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the
Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/624bfdbb-2e6d-4640-9976-5741c6a4724d%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/624bfdbb-2e6d-4640-9976-5741c6a4724d%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/1e0efcf0-c0e3-4cd5-a9fd-7a21844fe44f%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/1e0efcf0-c0e3-4cd5-a9fd-7a21844fe44f%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJAkrZjN5iCfONbEPqGB9CctH%2BqL4%3DgDxr0gii%2Brb7nGwg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
t***@gmail.com
2015-08-24 22:43:50 UTC
Permalink
Ok great -- thanks, will try this.

In terms of getting the right version of libbson, I guess that getting a
commit from

https://github.com/mongodb/libbson/commits/

here is the the way to go? Which commit do you suggest I use?

Thanks!
T
Post by Asya Kamsky
Okay, well, I guess it can't hurt so you should try to see if anything
https://github.com/chergert/mdb
This was mostly an exercise in dumping out on-disk format (*note* to
anyone reading, this had never been updated since 2.4 so I'm only
suggesting it specifically because OP is using 2.4 - using this on later
versions without changes will simply give errors).
To build this you will need libbson version from when this was written
which I believe is 1.0. The executable you want to run is called mdbundo,
here's its usage.
usage: mdbundo DBPATH DBNAME COLNAME
what it will do is walk through the DB files looking for deleted records
and then try to restore them. I can't remember for sure but I think it
will send them to stdout so you can redirect the output of this to a file.
That file should have bson in them so you can use bsondump to view them
(which is pointless for chunks but useful for "files") and you can run
mongorestore on the output to load it into a new collection so you can then
figure out if anything in there is valuable.
Obviously you run it twice on the same DB files, once for myfs.files, once
for myfs.chunks.
Again, given the output to validate I don't really expect it to find
anything more than a handful of records at most, possibly unusable (since
for every "file" every chunk belonging to it must be present to have the
full usable file back.
Good luck and let us know what you uncover,
Asya
Post by t***@gmail.com
Thanks for the response -- sorry for the delay in reply (was away from
computer).
Post by Asya Kamsky
This is somewhat surprising. Are there other collections in this database?
[u'system.indexes', u'myfs.chunks', u'myfs.files']
Post by Asya Kamsky
This is the original files, right?
Yes.
Post by Asya Kamsky
Are you *sure* there are files missing? I don't mean by count but by
content... You mentioned you were about to find some strings matching
something you expected to find in the DB but didn't?
Definitely 100% completely sure. I can use "grep" on the binary files
of this database and I see at least parts of the missing records, e.g. some
keys that are unique and are exactly what the missing records would have.
Moreover, I (and several others) were happily using this collection as the
source a stable restful API for months ... at least, until this problem
occurred ...
Post by Asya Kamsky
How many dbname.ns files are there and what are their sizes?
In the original collection there are 90 .ns files, most of which are the
standard size "2146435072", but some of which (the early ones of course)
are smaller.
In the repaired collection there are 35 .ns files, same size pattern.
Thanks again for all the help --
T
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Asya,
1) The aggregation of the chunk collection produces 279 documents. I
don't recall how many there were before the ones I want went missing.
However, I guess there more than 279. I don't know exactly how many more,
but I expect on the order of several hundred at least (but again, I'm not
sure). The collection does verify, in that all the chunks in the chunks
collection are accounted for by a single file in the files collection. I
computed the expected number of chunks for each file by dividing length by
chunk size for each file record; that number exactly equals the count of
chunks for that file as produced by the aggregation in the chunk collection.
{
"ns" : "mydb.myfs.files",
"firstExtent" : "0:19000 ns:mydb.myfs.files",
"lastExtent" : "7:67757000 ns:mydb.myfs.files",
"extentCount" : 5,
<snip "extents">
"datasize" : 24075568,
"nrecords" : 279,
"lastExtentSize" : 12902400,
"padding" : 1,
<snip some other stuff>
"objectsFound" : 279,
"invalidObjects" : 0,
"bytesWithHeaders" : 24080032,
"bytesWithoutHeaders" : 24075568,
"deletedCount" : 3,
"deletedSize" : 12008944,
"nIndexes" : 3,
"keysPerIndex" : {
"mydb.myfs.files.$_id_" : 279,
"mydb.myfs.files.$filename_1_uploadDate_-1" : 279,
"mydb.myfs.files.$timestamp_1" : 279
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
db.chunks.validate(true)
{
"ns" : "mydb.myfs.chunks",
"firstExtent" : "0:5000 ns:mydb.myfs.chunks",
"lastExtent" : "28:2000 ns:mydb.myfs.chunks",
"extentCount" : 43,
<snip "extents">
"datasize" : 49854830688,
"nrecords" : 190427,
"lastExtentSize" : 2146426864,
"padding" : 1,
"firstExtentDetails" : {
"loc" : "0:5000",
"xnext" : "0:162000",
"xprev" : "null",
"nsdiag" : "mydb.myfs.chunks",
"size" : 8192,
"firstRecord" : "0:50b0",
"lastRecord" : "0:6f28"
},
"lastExtentDetails" : {
"loc" : "28:2000",
"xnext" : "null",
"xprev" : "27:2000",
"nsdiag" : "mydb.myfs.chunks",
"size" : 2146426864,
"firstRecord" : "28:20b0",
"lastRecord" : "28:189c20b0"
},
"objectsFound" : 190427,
"invalidObjects" : 0,
"bytesWithHeaders" : 49857877520,
"bytesWithoutHeaders" : 49854830688,
"deletedCount" : 20,
"deletedSize" : 1733626640,
"nIndexes" : 2,
"keysPerIndex" : {
"mydb.myfs.chunks.$_id_" : 190427,
"mydb.myfs.chunks.$files_id_1_n_1" : 190427
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
I'm not sure whether "deletedCount" of "3" in the files collection
would make up for all the deleted records that I expect ... if I interpret
it correctly. It would also not make up for the huge compaction that
occurred in the repaired version of the collection (which is much smaller
on disk, but still contains 279 records).
Any thoughts on what to do next? Thanks again for your help!
T
Post by Asya Kamsky
I'm assuming that you start up mongod with the original files (ones
you did not repair but saved when you first noticed that something was
Okay, first, it would be good to know which collections are missing
You'd mentioned having 279 documents in files collection - how many
are in chunks collection? Do they "cross-verify"?
http://docs.mongodb.org/manual/reference/gridfs/
So you can tell for each file how many chunk documents there should be
and for each chunk document you can tell which file it belongs to and in
what order.
db.fs.chunks.aggregate({$sort:{files_id:1, num:1}},
{$group:{_id:"$files_id",count:{$sum:1}, chunks:{$push:"$num"}}}).
If you have 279 files in your fs.files collection, then you should get
back 279 documents from this aggregation, each corresponding to a "file" in
gridFS.
How many files do you expect to have?
db.fs.files.validate(true)
and
db.fs.chunks.validate(true)
http://docs.mongodb.org/manual/reference/command/validate/#output
You are interested in the deleted records (
http://docs.mongodb.org/manual/reference/command/validate/#validate.deletedCount
and the next field, deletedSize).
Basically these are the records that can be potentially recovered -
can you provide output to above experiments so we don't waste time trying
to recover data that's not still there?
Asya
Post by t***@gmail.com
Great thanks so much for the help!
T
Post by Asya Kamsky
Some of the records are in a free list and haven't been overwritten
yet. It may be possible to get them back.
Since you're using version 2.4.x I believe that there is code that
will walk through the DB files and dump out every document it finds there.
Let me dig around for it - obviously there is (a) no guarantee that
this will restore all of them and (b) did you say these were GridFS chunks
- if so then you won't get back a valid file unless all chunks from that
file are restored (along with corresponding files record).
Asya
Post by t***@gmail.com
Thanks for the suggestion. The logs don't appear to show any
removals on that database since Oct. 2014. At least as far as these logs
go, it doesn't look to me like a remove was done.
Any suggestions for what to do now? I can definitely see traces
of the data I want to get at in the (unmodified) database files. Is there
no way to get at the data in them?
Thanks!
T
Post by Asya Kamsky
The only CRUD operations that would be logged would be those that
take longer than 100ms.
2015-07-14T12:10:34.559-0500 I WRITE [conn4822] remove
blackbox.bigcoll query: { _id: { $gte: 10000.0 } } ndeleted:90000
keyUpdates:0 writeConflicts:0 numYields:12330 locks:{ Global: {
12331 } }, Collection: { acquireCount: { w: 12331 } } } 249168ms
Of course this was a huge mass remove of giant docs which is why
it took so long. If the number of documents deleted was small (like if
they were deleted one at a time instead of via single command) then it's
much less likely they would have taken longer than 100ms.
grep 'remove <dbname>' *.log.*
where you would replace <dbname> with the name of your actual DB
and replace *.log.* with path to your log files.
Asya
Post by t***@gmail.com
Post by Asya Kamsky
Do you have the logs from around the time the database crashed?
When was that relative to when you noticed the records missing?
Yes I have all the logs going back for a year. I noticed the
records missing the day that I wrote the help question originally (several
days ago). The crash happened several weeks ago.
Since you had journaling on, and you were able to restart mongod
Post by Asya Kamsky
without any errors, I would rule out the crash *causing* records to
disappear, which leaves you with the possibility that they were deleted.
If they were deleted and you don't have any backups from *before* they were
deleted, then there's not much that can be done to recover the data - sure,
it's possible to write code to literally look through the deleted list, but
if any data was inserted after the deletes then that space would be reused
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But wouldn't
there be some record in the logs if there was a deleation?
We haven't written any records since the deletes. (The files
haven't changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm using
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of
records shown -- not 250, like I said before, that was a mistake.)
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1),
(u'uploadDate', -1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used to
manage a gridfs instance.
Also I think I have some additional information. I copied the
database files to a new location and ran a full repair on the whole
database. IN the new repaired version, there were many fewer files .ns
files. As is as if some records were removed using e.g.
gridfs.GridFS.remove ... so in the new (compacted) version the data was
finally deleted. Maybe its somehow possible that the records were removed
from the gridfs collection by someone running a remove operation? I don't
see any evidence of such an operation in the logs but perhaps it's
possible? I've looked carefully at the logs and am not seeing anything
obvious, but maybe dont' know what to look for.
Of course, that I guess does not modify the underlying files on
the original collection where the (hypothetical) remove would have been
run. If this indeed the case, is there some way to get the data back?
After all, I do have the original .ns files.
Thanks!
Tob
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the
1) Describe your MongoDB deployment (standalone, replica set,
or sharded cluster)
2) What version of MongoDB are you running? (You can check it
with db.version()
<http://docs.mongodb.org/manual/reference/method/db.version>
in the mongo shell)
3) Which query did you use to find or count the documents? Can
you please run that query with explain()
<http://docs.mongodb.org/manual/reference/method/cursor.explain/> and
send us the output?
4) Output of db.collection.getIndexes()
<http://docs.mongodb.org/manual/reference/method/db.collection.getIndexes/>
for the collection where documents appear to be missing.
Regards,
ankit
On Monday, August 17, 2015 at 12:56:20 PM UTC+5:30, Chris De
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what is
the structure of the docs, are there indexes on this collection?
On Sunday, August 16, 2015 at 12:33:16 AM UTC+2,
Post by t***@gmail.com
I have a mongodb database containing a collection that has
remained unchanged for a period of time -- the underlying .ns and .0, .1
etc files have not been modified for months. Up until a few weeks ago, I
had no problem reading records from the collection. There were thousands of
records in the collection.
However, today when I attempted to read the records, many of
them appeared missing -- e.g. records that I expected to be there were not,
although some of the records were available. Now, there appear to be only
250 records. =
I copied the database files and did a repair of the (copied)
database, which ran fine apparently, but only the 250 records appeared, and
cannot find the others.
I can see manually that the bSON binary files in the
database directory associated with this database still contain the data
that I want to access (by grepping the contents of the database files, I
can see some unique text strings that are associated just with missing
records -- of course they do, since the files haven't been modified for
months).
I cannot think of anything that changed between now and the
last time that I successfully was able to access the records, except that
the DB process crashed once. It was brought back up immediately. Although,
as I say, it doesn't appear that anything modified the files associated
with the database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the
Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the
Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from
.
Visit this group at http://groups.google.com/group/mongodb-user
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the
Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the
Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/624bfdbb-2e6d-4640-9976-5741c6a4724d%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/624bfdbb-2e6d-4640-9976-5741c6a4724d%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
<javascript:>.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/1e0efcf0-c0e3-4cd5-a9fd-7a21844fe44f%40googlegroups.com
<https://groups.google.com/d/msgid/mongodb-user/1e0efcf0-c0e3-4cd5-a9fd-7a21844fe44f%40googlegroups.com?utm_medium=email&utm_source=footer>
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/f19c3639-e220-4e27-a0bb-f1b808df823b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-08-25 07:04:03 UTC
Permalink
I'd suggest 1.0 or whichever version is tagged around the time of last
commit on mdb repo.

If you get the wrong version you'll find out during build.

Asya
Post by t***@gmail.com
Ok great -- thanks, will try this.
In terms of getting the right version of libbson, I guess that getting a
commit from
https://github.com/mongodb/libbson/commits/
here is the the way to go? Which commit do you suggest I use?
Thanks!
T
Post by Asya Kamsky
Okay, well, I guess it can't hurt so you should try to see if anything
https://github.com/chergert/mdb
This was mostly an exercise in dumping out on-disk format (*note* to
anyone reading, this had never been updated since 2.4 so I'm only suggesting
it specifically because OP is using 2.4 - using this on later versions
without changes will simply give errors).
To build this you will need libbson version from when this was written
which I believe is 1.0. The executable you want to run is called mdbundo,
here's its usage.
usage: mdbundo DBPATH DBNAME COLNAME
what it will do is walk through the DB files looking for deleted records
and then try to restore them. I can't remember for sure but I think it
will send them to stdout so you can redirect the output of this to a file.
That file should have bson in them so you can use bsondump to view them
(which is pointless for chunks but useful for "files") and you can run
mongorestore on the output to load it into a new collection so you can then
figure out if anything in there is valuable.
Obviously you run it twice on the same DB files, once for myfs.files, once
for myfs.chunks.
Again, given the output to validate I don't really expect it to find
anything more than a handful of records at most, possibly unusable (since
for every "file" every chunk belonging to it must be present to have the
full usable file back.
Good luck and let us know what you uncover,
Asya
Post by t***@gmail.com
Thanks for the response -- sorry for the delay in reply (was away from
computer).
Post by Asya Kamsky
This is somewhat surprising. Are there other collections in this database?
[u'system.indexes', u'myfs.chunks', u'myfs.files']
Post by Asya Kamsky
This is the original files, right?
Yes.
Post by Asya Kamsky
Are you *sure* there are files missing? I don't mean by count but by
content... You mentioned you were about to find some strings matching
something you expected to find in the DB but didn't?
Definitely 100% completely sure. I can use "grep" on the binary files
of this database and I see at least parts of the missing records, e.g. some
keys that are unique and are exactly what the missing records would have.
Moreover, I (and several others) were happily using this collection as the
source a stable restful API for months ... at least, until this problem
occurred ...
Post by Asya Kamsky
How many dbname.ns files are there and what are their sizes?
In the original collection there are 90 .ns files, most of which are the
standard size "2146435072", but some of which (the early ones of course) are
smaller.
In the repaired collection there are 35 .ns files, same size pattern.
Thanks again for all the help --
T
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Asya,
1) The aggregation of the chunk collection produces 279 documents. I
don't recall how many there were before the ones I want went missing.
However, I guess there more than 279. I don't know exactly how many more,
but I expect on the order of several hundred at least (but again, I'm not
sure). The collection does verify, in that all the chunks in the chunks
collection are accounted for by a single file in the files collection. I
computed the expected number of chunks for each file by dividing length by
chunk size for each file record; that number exactly equals the count of
chunks for that file as produced by the aggregation in the chunk collection.
{
"ns" : "mydb.myfs.files",
"firstExtent" : "0:19000 ns:mydb.myfs.files",
"lastExtent" : "7:67757000 ns:mydb.myfs.files",
"extentCount" : 5,
<snip "extents">
"datasize" : 24075568,
"nrecords" : 279,
"lastExtentSize" : 12902400,
"padding" : 1,
<snip some other stuff>
"objectsFound" : 279,
"invalidObjects" : 0,
"bytesWithHeaders" : 24080032,
"bytesWithoutHeaders" : 24075568,
"deletedCount" : 3,
"deletedSize" : 12008944,
"nIndexes" : 3,
"keysPerIndex" : {
"mydb.myfs.files.$_id_" : 279,
"mydb.myfs.files.$filename_1_uploadDate_-1" : 279,
"mydb.myfs.files.$timestamp_1" : 279
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
db.chunks.validate(true)
{
"ns" : "mydb.myfs.chunks",
"firstExtent" : "0:5000 ns:mydb.myfs.chunks",
"lastExtent" : "28:2000 ns:mydb.myfs.chunks",
"extentCount" : 43,
<snip "extents">
"datasize" : 49854830688,
"nrecords" : 190427,
"lastExtentSize" : 2146426864,
"padding" : 1,
"firstExtentDetails" : {
"loc" : "0:5000",
"xnext" : "0:162000",
"xprev" : "null",
"nsdiag" : "mydb.myfs.chunks",
"size" : 8192,
"firstRecord" : "0:50b0",
"lastRecord" : "0:6f28"
},
"lastExtentDetails" : {
"loc" : "28:2000",
"xnext" : "null",
"xprev" : "27:2000",
"nsdiag" : "mydb.myfs.chunks",
"size" : 2146426864,
"firstRecord" : "28:20b0",
"lastRecord" : "28:189c20b0"
},
"objectsFound" : 190427,
"invalidObjects" : 0,
"bytesWithHeaders" : 49857877520,
"bytesWithoutHeaders" : 49854830688,
"deletedCount" : 20,
"deletedSize" : 1733626640,
"nIndexes" : 2,
"keysPerIndex" : {
"mydb.myfs.chunks.$_id_" : 190427,
"mydb.myfs.chunks.$files_id_1_n_1" : 190427
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
I'm not sure whether "deletedCount" of "3" in the files collection
would make up for all the deleted records that I expect ... if I interpret
it correctly. It would also not make up for the huge compaction that
occurred in the repaired version of the collection (which is much smaller on
disk, but still contains 279 records).
Any thoughts on what to do next? Thanks again for your help!
T
Post by Asya Kamsky
I'm assuming that you start up mongod with the original files (ones
you did not repair but saved when you first noticed that something was
Okay, first, it would be good to know which collections are missing
You'd mentioned having 279 documents in files collection - how many
are in chunks collection? Do they "cross-verify"?
http://docs.mongodb.org/manual/reference/gridfs/
So you can tell for each file how many chunk documents there should be
and for each chunk document you can tell which file it belongs to and in
what order.
db.fs.chunks.aggregate({$sort:{files_id:1, num:1}},
{$group:{_id:"$files_id",count:{$sum:1}, chunks:{$push:"$num"}}}).
If you have 279 files in your fs.files collection, then you should get
back 279 documents from this aggregation, each corresponding to a "file" in
gridFS.
How many files do you expect to have?
db.fs.files.validate(true)
and
db.fs.chunks.validate(true)
http://docs.mongodb.org/manual/reference/command/validate/#output
You are interested in the deleted records
(http://docs.mongodb.org/manual/reference/command/validate/#validate.deletedCount
and the next field, deletedSize).
Basically these are the records that can be potentially recovered -
can you provide output to above experiments so we don't waste time trying to
recover data that's not still there?
Asya
Post by t***@gmail.com
Great thanks so much for the help!
T
Post by Asya Kamsky
Some of the records are in a free list and haven't been overwritten
yet. It may be possible to get them back.
Since you're using version 2.4.x I believe that there is code that
will walk through the DB files and dump out every document it finds there.
Let me dig around for it - obviously there is (a) no guarantee that
this will restore all of them and (b) did you say these were GridFS chunks -
if so then you won't get back a valid file unless all chunks from that file
are restored (along with corresponding files record).
Asya
Post by t***@gmail.com
Thanks for the suggestion. The logs don't appear to show any
removals on that database since Oct. 2014. At least as far as these logs
go, it doesn't look to me like a remove was done.
Any suggestions for what to do now? I can definitely see traces
of the data I want to get at in the (unmodified) database files. Is there
no way to get at the data in them?
Thanks!
T
Post by Asya Kamsky
The only CRUD operations that would be logged would be those that
take longer than 100ms.
2015-07-14T12:10:34.559-0500 I WRITE [conn4822] remove
blackbox.bigcoll query: { _id: { $gte: 10000.0 } } ndeleted:90000
keyUpdates:0 writeConflicts:0 numYields:12330 locks:{ Global: {
acquireCount: { r: 12331, w: 12331 } }, Database: { acquireCount: { w: 12331
} }, Collection: { acquireCount: { w: 12331 } } } 249168ms
Of course this was a huge mass remove of giant docs which is why
it took so long. If the number of documents deleted was small (like if
they were deleted one at a time instead of via single command) then it's
much less likely they would have taken longer than 100ms.
grep 'remove <dbname>' *.log.*
where you would replace <dbname> with the name of your actual DB
and replace *.log.* with path to your log files.
Asya
Post by t***@gmail.com
Post by Asya Kamsky
Do you have the logs from around the time the database crashed?
When was that relative to when you noticed the records missing?
Yes I have all the logs going back for a year. I noticed the
records missing the day that I wrote the help question originally (several
days ago). The crash happened several weeks ago.
Post by Asya Kamsky
Since you had journaling on, and you were able to restart mongod
without any errors, I would rule out the crash *causing* records to
disappear, which leaves you with the possibility that they were deleted.
If they were deleted and you don't have any backups from *before* they were
deleted, then there's not much that can be done to recover the data - sure,
it's possible to write code to literally look through the deleted list, but
if any data was inserted after the deletes then that space would be reused
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But wouldn't
there be some record in the logs if there was a deleation?
We haven't written any records since the deletes. (The files
haven't changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm using
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of
records shown -- not 250, like I said before, that was a mistake.)
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1),
(u'uploadDate', -1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used to
manage a gridfs instance.
Also I think I have some additional information. I copied the
database files to a new location and ran a full repair on the whole
database. IN the new repaired version, there were many fewer files .ns
files. As is as if some records were removed using e.g.
gridfs.GridFS.remove ... so in the new (compacted) version the data was
finally deleted. Maybe its somehow possible that the records were removed
from the gridfs collection by someone running a remove operation? I don't
see any evidence of such an operation in the logs but perhaps it's possible?
I've looked carefully at the logs and am not seeing anything obvious, but
maybe dont' know what to look for.
Of course, that I guess does not modify the underlying files on
the original collection where the (hypothetical) remove would have been run.
If this indeed the case, is there some way to get the data back? After all,
I do have the original .ns files.
Thanks!
Tob
On Monday, August 17, 2015 at 12:47:08 PM UTC-4, Ankit Kakkar
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the
1) Describe your MongoDB deployment (standalone, replica set,
or sharded cluster)
2) What version of MongoDB are you running? (You can check it
with db.version() in the mongo shell)
3) Which query did you use to find or count the documents? Can
you please run that query with explain() and send us the output?
4) Output of db.collection.getIndexes() for the collection
where documents appear to be missing.
Regards,
ankit
On Monday, August 17, 2015 at 12:56:20 PM UTC+5:30, Chris De
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what is
the structure of the docs, are there indexes on this collection?
On Sunday, August 16, 2015 at 12:33:16 AM UTC+2,
Post by t***@gmail.com
I have a mongodb database containing a collection that has
remained unchanged for a period of time -- the underlying .ns and .0, .1 etc
files have not been modified for months. Up until a few weeks ago, I had no
problem reading records from the collection. There were thousands of records
in the collection.
However, today when I attempted to read the records, many of
them appeared missing -- e.g. records that I expected to be there were not,
although some of the records were available. Now, there appear to be only
250 records. =
I copied the database files and did a repair of the (copied)
database, which ran fine apparently, but only the 250 records appeared, and
cannot find the others.
I can see manually that the bSON binary files in the
database directory associated with this database still contain the data that
I want to access (by grepping the contents of the database files, I can see
some unique text strings that are associated just with missing records -- of
course they do, since the files haven't been modified for months).
I cannot think of anything that changed between now and the
last time that I successfully was able to access the records, except that
the DB process crashed once. It was brought back up immediately. Although,
as I say, it doesn't appear that anything modified the files associated with
the database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the
Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the
Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from
To post to this group, send email to
Visit this group at
http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the
Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the
Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/624bfdbb-2e6d-4640-9976-5741c6a4724d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/1e0efcf0-c0e3-4cd5-a9fd-7a21844fe44f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/f19c3639-e220-4e27-a0bb-f1b808df823b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJBqY4MFrRkRVUvmmjaP2HcESkYpQG_yqpbUJ8eqaQW-Dg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
t***@gmail.com
2015-08-25 15:51:17 UTC
Permalink
Hi --

Having trouble compiling mdb. I'm at
commit e7d018cad20f2a6aa62aea5db088a8e7b05a190d of libbson. Is that too
early/late?

To get the make to work at all, I included the following line in all the
compilation steps:
-I /usr/local/include/libbson-1.0

Is this right?

Even so, here's the compilation error:

/tmp/ccanf01B.o: In function `db_init':
/home/brejicz/mdb/mdb.c:177: undefined reference to `bson_strdup_printf'
/home/brejicz/mdb/mdb.c:179: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:182: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:189: undefined reference to `bson_strdup_printf'
/home/brejicz/mdb/mdb.c:191: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:195: undefined reference to `bson_realloc'
/home/brejicz/mdb/mdb.c:197: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:218: undefined reference to `bson_free'
/tmp/ccanf01B.o: In function `record_bson':
/home/brejicz/mdb/mdb.c:608: undefined reference to `bson_init_static'
/tmp/ccvrjSqo.o: In function `main':
/home/brejicz/mdb/mdbdump.c:70: undefined reference to `bson_as_json'
/home/brejicz/mdb/mdbdump.c:73: undefined reference to `bson_free'
collect2: ld returned 1 exit status
make: *** [mdbdump] Error 1


Sorry for the detailed questions, but any help would be great.

Thanks!,
T
Post by Asya Kamsky
I'd suggest 1.0 or whichever version is tagged around the time of last
commit on mdb repo.
If you get the wrong version you'll find out during build.
Asya
Post by t***@gmail.com
Ok great -- thanks, will try this.
In terms of getting the right version of libbson, I guess that getting a
commit from
https://github.com/mongodb/libbson/commits/
here is the the way to go? Which commit do you suggest I use?
Thanks!
T
Post by Asya Kamsky
Okay, well, I guess it can't hurt so you should try to see if anything
dumped out with something that reads the raw files ends up helping at
https://github.com/chergert/mdb
This was mostly an exercise in dumping out on-disk format (*note* to
anyone reading, this had never been updated since 2.4 so I'm only
suggesting
Post by t***@gmail.com
Post by Asya Kamsky
it specifically because OP is using 2.4 - using this on later versions
without changes will simply give errors).
To build this you will need libbson version from when this was written
which I believe is 1.0. The executable you want to run is called
mdbundo,
Post by t***@gmail.com
Post by Asya Kamsky
here's its usage.
usage: mdbundo DBPATH DBNAME COLNAME
what it will do is walk through the DB files looking for deleted
records
Post by t***@gmail.com
Post by Asya Kamsky
and then try to restore them. I can't remember for sure but I think
it
Post by t***@gmail.com
Post by Asya Kamsky
will send them to stdout so you can redirect the output of this to a
file.
Post by t***@gmail.com
Post by Asya Kamsky
That file should have bson in them so you can use bsondump to view them
(which is pointless for chunks but useful for "files") and you can run
mongorestore on the output to load it into a new collection so you can
then
Post by t***@gmail.com
Post by Asya Kamsky
figure out if anything in there is valuable.
Obviously you run it twice on the same DB files, once for myfs.files,
once
Post by t***@gmail.com
Post by Asya Kamsky
for myfs.chunks.
Again, given the output to validate I don't really expect it to find
anything more than a handful of records at most, possibly unusable
(since
Post by t***@gmail.com
Post by Asya Kamsky
for every "file" every chunk belonging to it must be present to have
the
Post by t***@gmail.com
Post by Asya Kamsky
full usable file back.
Good luck and let us know what you uncover,
Asya
Post by t***@gmail.com
Thanks for the response -- sorry for the delay in reply (was away from
computer).
Post by Asya Kamsky
This is somewhat surprising. Are there other collections in this database?
[u'system.indexes', u'myfs.chunks', u'myfs.files']
Post by Asya Kamsky
This is the original files, right?
Yes.
Post by Asya Kamsky
Are you *sure* there are files missing? I don't mean by count but by
content... You mentioned you were about to find some strings matching
something you expected to find in the DB but didn't?
Definitely 100% completely sure. I can use "grep" on the binary
files
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
of this database and I see at least parts of the missing records, e.g.
some
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
keys that are unique and are exactly what the missing records would
have.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Moreover, I (and several others) were happily using this collection as
the
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
source a stable restful API for months ... at least, until this
problem
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
occurred ...
Post by Asya Kamsky
How many dbname.ns files are there and what are their sizes?
In the original collection there are 90 .ns files, most of which are
the
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
standard size "2146435072", but some of which (the early ones of
course) are
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
smaller.
In the repaired collection there are 35 .ns files, same size pattern.
Thanks again for all the help --
T
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Asya,
1) The aggregation of the chunk collection produces 279 documents.
I
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
don't recall how many there were before the ones I want went
missing.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
However, I guess there more than 279. I don't know exactly how many
more,
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
but I expect on the order of several hundred at least (but again,
I'm not
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
sure). The collection does verify, in that all the chunks in the
chunks
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
collection are accounted for by a single file in the files
collection. I
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
computed the expected number of chunks for each file by dividing
length by
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
chunk size for each file record; that number exactly equals the
count of
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
chunks for that file as produced by the aggregation in the chunk
collection.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
{
"ns" : "mydb.myfs.files",
"firstExtent" : "0:19000 ns:mydb.myfs.files",
"lastExtent" : "7:67757000 ns:mydb.myfs.files",
"extentCount" : 5,
<snip "extents">
"datasize" : 24075568,
"nrecords" : 279,
"lastExtentSize" : 12902400,
"padding" : 1,
<snip some other stuff>
"objectsFound" : 279,
"invalidObjects" : 0,
"bytesWithHeaders" : 24080032,
"bytesWithoutHeaders" : 24075568,
"deletedCount" : 3,
"deletedSize" : 12008944,
"nIndexes" : 3,
"keysPerIndex" : {
"mydb.myfs.files.$_id_" : 279,
"mydb.myfs.files.$filename_1_uploadDate_-1" : 279,
"mydb.myfs.files.$timestamp_1" : 279
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
db.chunks.validate(true)
{
"ns" : "mydb.myfs.chunks",
"firstExtent" : "0:5000 ns:mydb.myfs.chunks",
"lastExtent" : "28:2000 ns:mydb.myfs.chunks",
"extentCount" : 43,
<snip "extents">
"datasize" : 49854830688,
"nrecords" : 190427,
"lastExtentSize" : 2146426864,
"padding" : 1,
"firstExtentDetails" : {
"loc" : "0:5000",
"xnext" : "0:162000",
"xprev" : "null",
"nsdiag" : "mydb.myfs.chunks",
"size" : 8192,
"firstRecord" : "0:50b0",
"lastRecord" : "0:6f28"
},
"lastExtentDetails" : {
"loc" : "28:2000",
"xnext" : "null",
"xprev" : "27:2000",
"nsdiag" : "mydb.myfs.chunks",
"size" : 2146426864,
"firstRecord" : "28:20b0",
"lastRecord" : "28:189c20b0"
},
"objectsFound" : 190427,
"invalidObjects" : 0,
"bytesWithHeaders" : 49857877520,
"bytesWithoutHeaders" : 49854830688,
"deletedCount" : 20,
"deletedSize" : 1733626640,
"nIndexes" : 2,
"keysPerIndex" : {
"mydb.myfs.chunks.$_id_" : 190427,
"mydb.myfs.chunks.$files_id_1_n_1" : 190427
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
I'm not sure whether "deletedCount" of "3" in the files collection
would make up for all the deleted records that I expect ... if I
interpret
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
it correctly. It would also not make up for the huge compaction
that
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
occurred in the repaired version of the collection (which is much
smaller on
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
disk, but still contains 279 records).
Any thoughts on what to do next? Thanks again for your help!
T
Post by Asya Kamsky
I'm assuming that you start up mongod with the original files (ones
you did not repair but saved when you first noticed that something
was
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Okay, first, it would be good to know which collections are missing
You'd mentioned having 279 documents in files collection - how many
are in chunks collection? Do they "cross-verify"?
http://docs.mongodb.org/manual/reference/gridfs/
So you can tell for each file how many chunk documents there should
be
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
and for each chunk document you can tell which file it belongs to
and in
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
what order.
db.fs.chunks.aggregate({$sort:{files_id:1, num:1}},
{$group:{_id:"$files_id",count:{$sum:1}, chunks:{$push:"$num"}}}).
If you have 279 files in your fs.files collection, then you should
get
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
back 279 documents from this aggregation, each corresponding to a
"file" in
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
gridFS.
How many files do you expect to have?
db.fs.files.validate(true)
and
db.fs.chunks.validate(true)
http://docs.mongodb.org/manual/reference/command/validate/#output
You are interested in the deleted records
(
http://docs.mongodb.org/manual/reference/command/validate/#validate.deletedCount
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
and the next field, deletedSize).
Basically these are the records that can be potentially recovered -
can you provide output to above experiments so we don't waste time
trying to
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
recover data that's not still there?
Asya
Post by t***@gmail.com
Great thanks so much for the help!
T
Post by Asya Kamsky
Some of the records are in a free list and haven't been
overwritten
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
yet. It may be possible to get them back.
Since you're using version 2.4.x I believe that there is code
that
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
will walk through the DB files and dump out every document it
finds there.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Let me dig around for it - obviously there is (a) no guarantee
that
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
this will restore all of them and (b) did you say these were
GridFS chunks -
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
if so then you won't get back a valid file unless all chunks from
that file
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
are restored (along with corresponding files record).
Asya
Post by t***@gmail.com
Thanks for the suggestion. The logs don't appear to show any
removals on that database since Oct. 2014. At least as far as
these logs
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
go, it doesn't look to me like a remove was done.
Any suggestions for what to do now? I can definitely see
traces
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
of the data I want to get at in the (unmodified) database files.
Is there
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
no way to get at the data in them?
Thanks!
T
Post by Asya Kamsky
The only CRUD operations that would be logged would be those
that
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
take longer than 100ms.
2015-07-14T12:10:34.559-0500 I WRITE [conn4822] remove
blackbox.bigcoll query: { _id: { $gte: 10000.0 } }
ndeleted:90000
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
keyUpdates:0 writeConflicts:0 numYields:12330 locks:{ Global: {
acquireCount: { r: 12331, w: 12331 } }, Database: {
acquireCount: { w: 12331
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
} }, Collection: { acquireCount: { w: 12331 } } } 249168ms
Of course this was a huge mass remove of giant docs which is
why
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
it took so long. If the number of documents deleted was small
(like if
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
they were deleted one at a time instead of via single command)
then it's
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
much less likely they would have taken longer than 100ms.
I would suggest searching through the logs with equivalent
command
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
grep 'remove <dbname>' *.log.*
where you would replace <dbname> with the name of your actual
DB
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
and replace *.log.* with path to your log files.
Asya
Post by t***@gmail.com
Post by Asya Kamsky
Do you have the logs from around the time the database
crashed?
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
When was that relative to when you noticed the records
missing?
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Yes I have all the logs going back for a year. I noticed the
records missing the day that I wrote the help question
originally (several
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
days ago). The crash happened several weeks ago.
Post by Asya Kamsky
Since you had journaling on, and you were able to restart
mongod
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
without any errors, I would rule out the crash *causing*
records to
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
disappear, which leaves you with the possibility that they
were deleted.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
If they were deleted and you don't have any backups from
*before* they were
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
deleted, then there's not much that can be done to recover
the data - sure,
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
it's possible to write code to literally look through the
deleted list, but
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
if any data was inserted after the deletes then that space
would be reused
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But wouldn't
there be some record in the logs if there was a deleation?
We haven't written any records since the deletes. (The files
haven't changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm
using
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of
records shown -- not 250, like I said before, that was a
mistake.)
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1),
(u'uploadDate', -1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used to
manage a gridfs instance.
Also I think I have some additional information. I copied
the
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
database files to a new location and ran a full repair on
the whole
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
database. IN the new repaired version, there were many
fewer files .ns
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
files. As is as if some records were removed using e.g.
gridfs.GridFS.remove ... so in the new (compacted) version
the data was
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
finally deleted. Maybe its somehow possible that the
records were removed
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
from the gridfs collection by someone running a remove
operation? I don't
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
see any evidence of such an operation in the logs but
perhaps it's possible?
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
I've looked carefully at the logs and am not seeing anything
obvious, but
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
maybe dont' know what to look for.
Of course, that I guess does not modify the underlying files
on
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
the original collection where the (hypothetical) remove
would have been run.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
If this indeed the case, is there some way to get the data
back? After all,
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
I do have the original .ns files.
Thanks!
Tob
On Monday, August 17, 2015 at 12:47:08 PM UTC-4, Ankit
Kakkar
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the
investigation of this case, please provide us with
1) Describe your MongoDB deployment (standalone, replica
set,
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
or sharded cluster)
2) What version of MongoDB are you running? (You can check
it
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
with db.version() in the mongo shell)
3) Which query did you use to find or count the documents?
Can
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
you please run that query with explain() and send us the
output?
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
4) Output of db.collection.getIndexes() for the collection
where documents appear to be missing.
Regards,
ankit
On Monday, August 17, 2015 at 12:56:20 PM UTC+5:30, Chris
De
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what
is
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
the structure of the docs, are there indexes on this
collection?
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
On Sunday, August 16, 2015 at 12:33:16 AM UTC+2,
Post by t***@gmail.com
I have a mongodb database containing a collection that
has
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
remained unchanged for a period of time -- the underlying
.ns and .0, .1 etc
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
files have not been modified for months. Up until a few
weeks ago, I had no
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
problem reading records from the collection. There were
thousands of records
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
in the collection.
However, today when I attempted to read the records, many
of
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
them appeared missing -- e.g. records that I expected to
be there were not,
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
although some of the records were available. Now, there
appear to be only
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
250 records. =
I copied the database files and did a repair of the
(copied)
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
database, which ran fine apparently, but only the 250
records appeared, and
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
cannot find the others.
I can see manually that the bSON binary files in the
database directory associated with this database still
contain the data that
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
I want to access (by grepping the contents of the
database files, I can see
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
some unique text strings that are associated just with
missing records -- of
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
course they do, since the files haven't been modified for
months).
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
I cannot think of anything that changed between now and
the
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
last time that I successfully was able to access the
records, except that
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
the DB process crashed once. It was brought back up
immediately. Although,
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
as I say, it doesn't appear that anything modified the
files associated with
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
the database where these records are.
What am I doing wrong, and how can I get my data back?
Thanks,
Tobjan
--
You received this message because you are subscribed to the
Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the
Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails
from
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
To post to this group, send email to
Visit this group at
http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/f7bfdcca-09cb-4d77-95fd-6028b31c4edb%40googlegroups.com.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the
Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the
Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from
it,
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
To post to this group, send email to
Visit this group at
http://groups.google.com/group/mongodb-user.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/aeb293c7-4c45-4134-b437-fcb0fe6aba88%40googlegroups.com.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the
Google
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the
Google
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from
it,
<javascript:>.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/e97e3f3e-7a54-4a56-8238-8551bd7e2985%40googlegroups.com.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/16893425-2e36-42a3-902a-70a203592200%40googlegroups.com.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it,
send
<javascript:>.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/624bfdbb-2e6d-4640-9976-5741c6a4724d%40googlegroups.com.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
an
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/1e0efcf0-c0e3-4cd5-a9fd-7a21844fe44f%40googlegroups.com.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups
Post by t***@gmail.com
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups
Post by t***@gmail.com
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
an
<javascript:>.
Post by t***@gmail.com
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/f19c3639-e220-4e27-a0bb-f1b808df823b%40googlegroups.com.
Post by t***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/512099b8-45dd-4908-9560-d1f85cfda6b0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
t***@gmail.com
2015-08-25 16:03:36 UTC
Permalink
I should add that the relevant libbson files with those symbols seem to be
properly installed, e.g. running

pkg-config --cflags --libs --libs libbson-1.0

returns

-I/usr/local/include/libbson-1.0 -L/usr/local/lib -lbson-1.0

and inside /usr/local/include/libbson-1.0, running "grep -R bson_free ."

returns:

./bson.h: * Returns: A newly allocated string that should be freed with
bson_free().
./bson-memory.h:void bson_free (void *mem);
Post by t***@gmail.com
Hi --
Having trouble compiling mdb. I'm at
commit e7d018cad20f2a6aa62aea5db088a8e7b05a190d of libbson. Is that too
early/late?
To get the make to work at all, I included the following line in all the
-I /usr/local/include/libbson-1.0
Is this right?
/home/brejicz/mdb/mdb.c:177: undefined reference to `bson_strdup_printf'
/home/brejicz/mdb/mdb.c:179: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:182: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:189: undefined reference to `bson_strdup_printf'
/home/brejicz/mdb/mdb.c:191: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:195: undefined reference to `bson_realloc'
/home/brejicz/mdb/mdb.c:197: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:218: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:608: undefined reference to `bson_init_static'
/home/brejicz/mdb/mdbdump.c:70: undefined reference to `bson_as_json'
/home/brejicz/mdb/mdbdump.c:73: undefined reference to `bson_free'
collect2: ld returned 1 exit status
make: *** [mdbdump] Error 1
Sorry for the detailed questions, but any help would be great.
Thanks!,
T
I'd suggest 1.0 or whichever version is tagged around the time of last
commit on mdb repo.
If you get the wrong version you'll find out during build.
Asya
Post by t***@gmail.com
Ok great -- thanks, will try this.
In terms of getting the right version of libbson, I guess that getting a
commit from
https://github.com/mongodb/libbson/commits/
here is the the way to go? Which commit do you suggest I use?
Thanks!
T
Post by Asya Kamsky
Okay, well, I guess it can't hurt so you should try to see if anything
dumped out with something that reads the raw files ends up helping at
https://github.com/chergert/mdb
This was mostly an exercise in dumping out on-disk format (*note* to
anyone reading, this had never been updated since 2.4 so I'm only
suggesting
Post by t***@gmail.com
Post by Asya Kamsky
it specifically because OP is using 2.4 - using this on later versions
without changes will simply give errors).
To build this you will need libbson version from when this was written
which I believe is 1.0. The executable you want to run is called
mdbundo,
Post by t***@gmail.com
Post by Asya Kamsky
here's its usage.
usage: mdbundo DBPATH DBNAME COLNAME
what it will do is walk through the DB files looking for deleted
records
Post by t***@gmail.com
Post by Asya Kamsky
and then try to restore them. I can't remember for sure but I think
it
Post by t***@gmail.com
Post by Asya Kamsky
will send them to stdout so you can redirect the output of this to a
file.
Post by t***@gmail.com
Post by Asya Kamsky
That file should have bson in them so you can use bsondump to view them
(which is pointless for chunks but useful for "files") and you can run
mongorestore on the output to load it into a new collection so you can
then
Post by t***@gmail.com
Post by Asya Kamsky
figure out if anything in there is valuable.
Obviously you run it twice on the same DB files, once for myfs.files,
once
Post by t***@gmail.com
Post by Asya Kamsky
for myfs.chunks.
Again, given the output to validate I don't really expect it to find
anything more than a handful of records at most, possibly unusable
(since
Post by t***@gmail.com
Post by Asya Kamsky
for every "file" every chunk belonging to it must be present to have
the
Post by t***@gmail.com
Post by Asya Kamsky
full usable file back.
Good luck and let us know what you uncover,
Asya
Post by t***@gmail.com
Thanks for the response -- sorry for the delay in reply (was away from
computer).
Post by Asya Kamsky
This is somewhat surprising. Are there other collections in this database?
[u'system.indexes', u'myfs.chunks', u'myfs.files']
Post by Asya Kamsky
This is the original files, right?
Yes.
Post by Asya Kamsky
Are you *sure* there are files missing? I don't mean by count but by
content... You mentioned you were about to find some strings matching
something you expected to find in the DB but didn't?
Definitely 100% completely sure. I can use "grep" on the binary
files
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
of this database and I see at least parts of the missing records, e.g.
some
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
keys that are unique and are exactly what the missing records would
have.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Moreover, I (and several others) were happily using this collection as
the
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
source a stable restful API for months ... at least, until this
problem
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
occurred ...
Post by Asya Kamsky
How many dbname.ns files are there and what are their sizes?
In the original collection there are 90 .ns files, most of which are
the
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
standard size "2146435072", but some of which (the early ones of
course) are
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
smaller.
In the repaired collection there are 35 .ns files, same size pattern.
Thanks again for all the help --
T
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Asya,
1) The aggregation of the chunk collection produces 279 documents.
I
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
don't recall how many there were before the ones I want went
missing.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
However, I guess there more than 279. I don't know exactly how many
more,
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
but I expect on the order of several hundred at least (but again,
I'm not
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
sure). The collection does verify, in that all the chunks in the
chunks
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
collection are accounted for by a single file in the files
collection. I
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
computed the expected number of chunks for each file by dividing
length by
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
chunk size for each file record; that number exactly equals the
count of
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
chunks for that file as produced by the aggregation in the chunk
collection.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
{
"ns" : "mydb.myfs.files",
"firstExtent" : "0:19000 ns:mydb.myfs.files",
"lastExtent" : "7:67757000 ns:mydb.myfs.files",
"extentCount" : 5,
<snip "extents">
"datasize" : 24075568,
"nrecords" : 279,
"lastExtentSize" : 12902400,
"padding" : 1,
<snip some other stuff>
"objectsFound" : 279,
"invalidObjects" : 0,
"bytesWithHeaders" : 24080032,
"bytesWithoutHeaders" : 24075568,
"deletedCount" : 3,
"deletedSize" : 12008944,
"nIndexes" : 3,
"keysPerIndex" : {
"mydb.myfs.files.$_id_" : 279,
"mydb.myfs.files.$filename_1_uploadDate_-1" : 279,
"mydb.myfs.files.$timestamp_1" : 279
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
db.chunks.validate(true)
{
"ns" : "mydb.myfs.chunks",
"firstExtent" : "0:5000 ns:mydb.myfs.chunks",
"lastExtent" : "28:2000 ns:mydb.myfs.chunks",
"extentCount" : 43,
<snip "extents">
"datasize" : 49854830688,
"nrecords" : 190427,
"lastExtentSize" : 2146426864,
"padding" : 1,
"firstExtentDetails" : {
"loc" : "0:5000",
"xnext" : "0:162000",
"xprev" : "null",
"nsdiag" : "mydb.myfs.chunks",
"size" : 8192,
"firstRecord" : "0:50b0",
"lastRecord" : "0:6f28"
},
"lastExtentDetails" : {
"loc" : "28:2000",
"xnext" : "null",
"xprev" : "27:2000",
"nsdiag" : "mydb.myfs.chunks",
"size" : 2146426864,
"firstRecord" : "28:20b0",
"lastRecord" : "28:189c20b0"
},
"objectsFound" : 190427,
"invalidObjects" : 0,
"bytesWithHeaders" : 49857877520,
"bytesWithoutHeaders" : 49854830688,
"deletedCount" : 20,
"deletedSize" : 1733626640,
"nIndexes" : 2,
"keysPerIndex" : {
"mydb.myfs.chunks.$_id_" : 190427,
"mydb.myfs.chunks.$files_id_1_n_1" : 190427
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
I'm not sure whether "deletedCount" of "3" in the files collection
would make up for all the deleted records that I expect ... if I
interpret
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
it correctly. It would also not make up for the huge compaction
that
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
occurred in the repaired version of the collection (which is much
smaller on
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
disk, but still contains 279 records).
Any thoughts on what to do next? Thanks again for your help!
T
Post by Asya Kamsky
I'm assuming that you start up mongod with the original files (ones
you did not repair but saved when you first noticed that something
was
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Okay, first, it would be good to know which collections are missing
You'd mentioned having 279 documents in files collection - how many
are in chunks collection? Do they "cross-verify"?
http://docs.mongodb.org/manual/reference/gridfs/
So you can tell for each file how many chunk documents there should
be
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
and for each chunk document you can tell which file it belongs to
and in
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
what order.
db.fs.chunks.aggregate({$sort:{files_id:1, num:1}},
{$group:{_id:"$files_id",count:{$sum:1}, chunks:{$push:"$num"}}}).
If you have 279 files in your fs.files collection, then you should
get
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
back 279 documents from this aggregation, each corresponding to a
"file" in
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
gridFS.
How many files do you expect to have?
db.fs.files.validate(true)
and
db.fs.chunks.validate(true)
http://docs.mongodb.org/manual/reference/command/validate/#output
You are interested in the deleted records
(
http://docs.mongodb.org/manual/reference/command/validate/#validate.deletedCount
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
and the next field, deletedSize).
Basically these are the records that can be potentially recovered -
can you provide output to above experiments so we don't waste time
trying to
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
recover data that's not still there?
Asya
Post by t***@gmail.com
Great thanks so much for the help!
T
Post by Asya Kamsky
Some of the records are in a free list and haven't been
overwritten
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
yet. It may be possible to get them back.
Since you're using version 2.4.x I believe that there is code
that
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
will walk through the DB files and dump out every document it
finds there.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Let me dig around for it - obviously there is (a) no guarantee
that
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
this will restore all of them and (b) did you say these were
GridFS chunks -
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
if so then you won't get back a valid file unless all chunks from
that file
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
are restored (along with corresponding files record).
Asya
Post by t***@gmail.com
Thanks for the suggestion. The logs don't appear to show any
removals on that database since Oct. 2014. At least as far as
these logs
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
go, it doesn't look to me like a remove was done.
Any suggestions for what to do now? I can definitely see
traces
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
of the data I want to get at in the (unmodified) database files.
Is there
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
no way to get at the data in them?
Thanks!
T
Post by Asya Kamsky
The only CRUD operations that would be logged would be those
that
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
take longer than 100ms.
2015-07-14T12:10:34.559-0500 I WRITE [conn4822] remove
blackbox.bigcoll query: { _id: { $gte: 10000.0 } }
ndeleted:90000
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
keyUpdates:0 writeConflicts:0 numYields:12330 locks:{ Global: {
acquireCount: { r: 12331, w: 12331 } }, Database: {
acquireCount: { w: 12331
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
} }, Collection: { acquireCount: { w: 12331 } } } 249168ms
Of course this was a huge mass remove of giant docs which is
why
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
it took so long. If the number of documents deleted was small
(like if
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
they were deleted one at a time instead of via single command)
then it's
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
much less likely they would have taken longer than 100ms.
I would suggest searching through the logs with equivalent
command
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
grep 'remove <dbname>' *.log.*
where you would replace <dbname> with the name of your actual
DB
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
and replace *.log.* with path to your log files.
Asya
Post by t***@gmail.com
Post by Asya Kamsky
Do you have the logs from around the time the database
crashed?
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
When was that relative to when you noticed the records
missing?
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Yes I have all the logs going back for a year. I noticed the
records missing the day that I wrote the help question
originally (several
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
days ago). The crash happened several weeks ago.
Post by Asya Kamsky
Since you had journaling on, and you were able to restart
mongod
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
without any errors, I would rule out the crash *causing*
records to
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
disappear, which leaves you with the possibility that they
were deleted.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
If they were deleted and you don't have any backups from
*before* they were
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
deleted, then there's not much that can be done to recover
the data - sure,
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
it's possible to write code to literally look through the
deleted list, but
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
if any data was inserted after the deletes then that space
would be reused
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But wouldn't
there be some record in the logs if there was a deleation?
We haven't written any records since the deletes. (The files
haven't changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm
using
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of
records shown -- not 250, like I said before, that was a
mistake.)
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1),
(u'uploadDate', -1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used to
manage a gridfs instance.
Also I think I have some additional information. I copied
the
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
database files to a new location and ran a full repair on
the whole
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
database. IN the new repaired version, there were many
fewer files .ns
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
files. As is as if some records were removed using e.g.
gridfs.GridFS.remove ... so in the new (compacted) version
the data was
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
finally deleted. Maybe its somehow possible that the
records were removed
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
from the gridfs collection by someone running a remove
operation? I don't
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
see any evidence of such an operation in the logs but
perhaps it's possible?
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
I've looked carefully at the logs and am not seeing anything
obvious, but
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
maybe dont' know what to look for.
Of course, that I guess does not modify the underlying files
on
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
the original collection where the (hypothetical) remove
would have been run.
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
If this indeed the case, is there some way to get the data
back? After all,
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
I do have the original .ns files.
Thanks!
Tob
On Monday, August 17, 2015 at 12:47:08 PM UTC-4, Ankit
Kakkar
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the
investigation of this case, please provide us with
1) Describe your MongoDB deployment (standalone, replica
set,
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
or sharded cluster)
2) What version of MongoDB are you running? (You can check
it
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
with db.version() in the mongo shell)
3) Which query did you use to find or count the documents?
Can
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
you please run that query with explain() and send us the
output?
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
4) Output of db.collection.getIndexes() for the collection
where documents appear to be missing.
Regards,
ankit
On Monday, August 17, 2015 at 12:56:20 PM UTC+5:30, Chris
De
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what
is
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
the structure of the docs, are there indexes on this
collection?
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
On Sunday, August 16, 2015 at 12:33:16 AM UTC+2,
Post by t***@gmail.com
I have a mongodb database containing a collection that
has
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
remained unchanged for a period of time -- the underlying
.ns and .0, .1 etc
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
files have not been modified for months. Up until a few
weeks ago, I had no
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
problem reading records from the collection. There were
thousands of records
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
in the collection.
However, today when I attempted to read the records, many
of
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
them appeared missing -- e.g. records that I expected to
be there were not,
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
although some of the records were available. Now, there
appear to be only
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
250 records. =
I copied the database files and did a repair of the
(copied)
...
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/ea02d25c-7c17-47f1-a443-c80956fbc5cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-08-31 01:02:15 UTC
Permalink
Here is what I know works.

Download 1.0.0 of libbson. You can get it from this url:
https://github.com/mongodb/libbson/archive/1.0.0.zip
Unzip it, cd into it, assuming you have all dependencies installed
(compiler, etc) run
./autoconf.sh
make
sudo make install

Then download mdb the same way from
https://github.com/chergert/mdb/archive/master.zip
unzip and cd into it
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
make mdbundo

Now you should have an executable called ./mdbundo
You may need to do this to run it:
export LD_LIBRARY_PATH=/usr/local/lib

./mdbundo --help
usage: mdbundo DBPATH DBNAME COLNAME

DBPATH is the path to the original files, DBNAME is myfs then you
would do this twice once for files and once for chunks.
If memory serves, the recovered bson will go to stderr so you want to
run it like

./mdbundo /path/to/my/dbfilesdir myfs files > myfs.files.bson
./mdbundo /path/to/my/dbfilesdir myfs chunks > myfs.chunks.bson

Let us know how it goes!

Asya
Post by t***@gmail.com
I should add that the relevant libbson files with those symbols seem to be
properly installed, e.g. running
pkg-config --cflags --libs --libs libbson-1.0
returns
-I/usr/local/include/libbson-1.0 -L/usr/local/lib -lbson-1.0
and inside /usr/local/include/libbson-1.0, running "grep -R bson_free ."
./bson.h: * Returns: A newly allocated string that should be freed with
bson_free().
./bson-memory.h:void bson_free (void *mem);
Hi --
Having trouble compiling mdb. I'm at commit
e7d018cad20f2a6aa62aea5db088a8e7b05a190d of libbson. Is that too
early/late?
To get the make to work at all, I included the following line in all the
-I /usr/local/include/libbson-1.0
Is this right?
/home/brejicz/mdb/mdb.c:177: undefined reference to `bson_strdup_printf'
/home/brejicz/mdb/mdb.c:179: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:182: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:189: undefined reference to `bson_strdup_printf'
/home/brejicz/mdb/mdb.c:191: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:195: undefined reference to `bson_realloc'
/home/brejicz/mdb/mdb.c:197: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:218: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:608: undefined reference to `bson_init_static'
/home/brejicz/mdb/mdbdump.c:70: undefined reference to `bson_as_json'
/home/brejicz/mdb/mdbdump.c:73: undefined reference to `bson_free'
collect2: ld returned 1 exit status
make: *** [mdbdump] Error 1
Sorry for the detailed questions, but any help would be great.
Thanks!,
T
I'd suggest 1.0 or whichever version is tagged around the time of last
commit on mdb repo.
If you get the wrong version you'll find out during build.
Asya
Post by t***@gmail.com
Ok great -- thanks, will try this.
In terms of getting the right version of libbson, I guess that getting a
commit from
https://github.com/mongodb/libbson/commits/
here is the the way to go? Which commit do you suggest I use?
Thanks!
T
Post by Asya Kamsky
Okay, well, I guess it can't hurt so you should try to see if anything
https://github.com/chergert/mdb
This was mostly an exercise in dumping out on-disk format (*note* to
anyone reading, this had never been updated since 2.4 so I'm only suggesting
it specifically because OP is using 2.4 - using this on later versions
without changes will simply give errors).
To build this you will need libbson version from when this was written
which I believe is 1.0. The executable you want to run is called mdbundo,
here's its usage.
usage: mdbundo DBPATH DBNAME COLNAME
what it will do is walk through the DB files looking for deleted records
and then try to restore them. I can't remember for sure but I think it
will send them to stdout so you can redirect the output of this to a file.
That file should have bson in them so you can use bsondump to view them
(which is pointless for chunks but useful for "files") and you can run
mongorestore on the output to load it into a new collection so you can then
figure out if anything in there is valuable.
Obviously you run it twice on the same DB files, once for myfs.files, once
for myfs.chunks.
Again, given the output to validate I don't really expect it to find
anything more than a handful of records at most, possibly unusable (since
for every "file" every chunk belonging to it must be present to have the
full usable file back.
Good luck and let us know what you uncover,
Asya
Post by t***@gmail.com
Thanks for the response -- sorry for the delay in reply (was away from
computer).
Post by Asya Kamsky
This is somewhat surprising. Are there other collections in this database?
[u'system.indexes', u'myfs.chunks', u'myfs.files']
Post by Asya Kamsky
This is the original files, right?
Yes.
Post by Asya Kamsky
Are you *sure* there are files missing? I don't mean by count but by
content... You mentioned you were about to find some strings matching
something you expected to find in the DB but didn't?
Definitely 100% completely sure. I can use "grep" on the binary files
of this database and I see at least parts of the missing records, e.g. some
keys that are unique and are exactly what the missing records would have.
Moreover, I (and several others) were happily using this collection as the
source a stable restful API for months ... at least, until this problem
occurred ...
Post by Asya Kamsky
How many dbname.ns files are there and what are their sizes?
In the original collection there are 90 .ns files, most of which are the
standard size "2146435072", but some of which (the early ones of course) are
smaller.
In the repaired collection there are 35 .ns files, same size pattern.
Thanks again for all the help --
T
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Asya,
1) The aggregation of the chunk collection produces 279 documents.
I
don't recall how many there were before the ones I want went missing.
However, I guess there more than 279. I don't know exactly how many more,
but I expect on the order of several hundred at least (but again, I'm not
sure). The collection does verify, in that all the chunks in the chunks
collection are accounted for by a single file in the files collection. I
computed the expected number of chunks for each file by dividing length by
chunk size for each file record; that number exactly equals the count of
chunks for that file as produced by the aggregation in the chunk collection.
{
"ns" : "mydb.myfs.files",
"firstExtent" : "0:19000 ns:mydb.myfs.files",
"lastExtent" : "7:67757000 ns:mydb.myfs.files",
"extentCount" : 5,
<snip "extents">
"datasize" : 24075568,
"nrecords" : 279,
"lastExtentSize" : 12902400,
"padding" : 1,
<snip some other stuff>
"objectsFound" : 279,
"invalidObjects" : 0,
"bytesWithHeaders" : 24080032,
"bytesWithoutHeaders" : 24075568,
"deletedCount" : 3,
"deletedSize" : 12008944,
"nIndexes" : 3,
"keysPerIndex" : {
"mydb.myfs.files.$_id_" : 279,
"mydb.myfs.files.$filename_1_uploadDate_-1" : 279,
"mydb.myfs.files.$timestamp_1" : 279
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
db.chunks.validate(true)
{
"ns" : "mydb.myfs.chunks",
"firstExtent" : "0:5000 ns:mydb.myfs.chunks",
"lastExtent" : "28:2000 ns:mydb.myfs.chunks",
"extentCount" : 43,
<snip "extents">
"datasize" : 49854830688,
"nrecords" : 190427,
"lastExtentSize" : 2146426864,
"padding" : 1,
"firstExtentDetails" : {
"loc" : "0:5000",
"xnext" : "0:162000",
"xprev" : "null",
"nsdiag" : "mydb.myfs.chunks",
"size" : 8192,
"firstRecord" : "0:50b0",
"lastRecord" : "0:6f28"
},
"lastExtentDetails" : {
"loc" : "28:2000",
"xnext" : "null",
"xprev" : "27:2000",
"nsdiag" : "mydb.myfs.chunks",
"size" : 2146426864,
"firstRecord" : "28:20b0",
"lastRecord" : "28:189c20b0"
},
"objectsFound" : 190427,
"invalidObjects" : 0,
"bytesWithHeaders" : 49857877520,
"bytesWithoutHeaders" : 49854830688,
"deletedCount" : 20,
"deletedSize" : 1733626640,
"nIndexes" : 2,
"keysPerIndex" : {
"mydb.myfs.chunks.$_id_" : 190427,
"mydb.myfs.chunks.$files_id_1_n_1" : 190427
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
I'm not sure whether "deletedCount" of "3" in the files collection
would make up for all the deleted records that I expect ... if I interpret
it correctly. It would also not make up for the huge compaction that
occurred in the repaired version of the collection (which is much smaller on
disk, but still contains 279 records).
Any thoughts on what to do next? Thanks again for your help!
T
Post by Asya Kamsky
I'm assuming that you start up mongod with the original files (ones
you did not repair but saved when you first noticed that something was
Okay, first, it would be good to know which collections are missing
You'd mentioned having 279 documents in files collection - how many
are in chunks collection? Do they "cross-verify"?
http://docs.mongodb.org/manual/reference/gridfs/
So you can tell for each file how many chunk documents there should be
and for each chunk document you can tell which file it belongs to and in
what order.
db.fs.chunks.aggregate({$sort:{files_id:1, num:1}},
{$group:{_id:"$files_id",count:{$sum:1}, chunks:{$push:"$num"}}}).
If you have 279 files in your fs.files collection, then you should get
back 279 documents from this aggregation, each corresponding to a "file" in
gridFS.
How many files do you expect to have?
db.fs.files.validate(true)
and
db.fs.chunks.validate(true)
http://docs.mongodb.org/manual/reference/command/validate/#output
You are interested in the deleted records
(http://docs.mongodb.org/manual/reference/command/validate/#validate.deletedCount
and the next field, deletedSize).
Basically these are the records that can be potentially recovered -
can you provide output to above experiments so we don't waste time
trying to
recover data that's not still there?
Asya
Post by t***@gmail.com
Great thanks so much for the help!
T
Post by Asya Kamsky
Some of the records are in a free list and haven't been overwritten
yet. It may be possible to get them back.
Since you're using version 2.4.x I believe that there is code that
will walk through the DB files and dump out every document it
finds there.
Let me dig around for it - obviously there is (a) no guarantee that
this will restore all of them and (b) did you say these were
GridFS chunks -
if so then you won't get back a valid file unless all chunks from
that file
are restored (along with corresponding files record).
Asya
Post by t***@gmail.com
Thanks for the suggestion. The logs don't appear to show any
removals on that database since Oct. 2014. At least as far as
these logs
go, it doesn't look to me like a remove was done.
Any suggestions for what to do now? I can definitely see traces
of the data I want to get at in the (unmodified) database files.
Is there
no way to get at the data in them?
Thanks!
T
Post by Asya Kamsky
The only CRUD operations that would be logged would be those that
take longer than 100ms.
2015-07-14T12:10:34.559-0500 I WRITE [conn4822] remove
blackbox.bigcoll query: { _id: { $gte: 10000.0 } }
ndeleted:90000
keyUpdates:0 writeConflicts:0 numYields:12330 locks:{ Global: {
acquireCount: { r: 12331, w: 12331 } }, Database: {
acquireCount: { w: 12331
} }, Collection: { acquireCount: { w: 12331 } } } 249168ms
Of course this was a huge mass remove of giant docs which is why
it took so long. If the number of documents deleted was small
(like if
they were deleted one at a time instead of via single command)
then it's
much less likely they would have taken longer than 100ms.
I would suggest searching through the logs with equivalent
command
grep 'remove <dbname>' *.log.*
where you would replace <dbname> with the name of your actual DB
and replace *.log.* with path to your log files.
Asya
On Tuesday, August 18, 2015 at 1:53:13 PM UTC-4, Asya Kamsky
Post by Asya Kamsky
Do you have the logs from around the time the database crashed?
When was that relative to when you noticed the records missing?
Yes I have all the logs going back for a year. I noticed the
records missing the day that I wrote the help question
originally (several
days ago). The crash happened several weeks ago.
Post by Asya Kamsky
Since you had journaling on, and you were able to restart
mongod
without any errors, I would rule out the crash *causing*
records to
disappear, which leaves you with the possibility that they
were deleted.
If they were deleted and you don't have any backups from
*before* they were
deleted, then there's not much that can be done to recover
the data - sure,
it's possible to write code to literally look through the
deleted list, but
if any data was inserted after the deletes then that space
would be reused
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But wouldn't
there be some record in the logs if there was a deleation?
We haven't written any records since the deletes. (The files
haven't changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm
using
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of
records shown -- not 250, like I said before, that was a
mistake.)
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1),
(u'uploadDate', -1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used to
manage a gridfs instance.
Also I think I have some additional information. I copied
the
database files to a new location and ran a full repair on
the whole
database. IN the new repaired version, there were many
fewer files .ns
files. As is as if some records were removed using e.g.
gridfs.GridFS.remove ... so in the new (compacted) version
the data was
finally deleted. Maybe its somehow possible that the
records were removed
from the gridfs collection by someone running a remove
operation? I don't
see any evidence of such an operation in the logs but
perhaps it's possible?
I've looked carefully at the logs and am not seeing anything
obvious, but
maybe dont' know what to look for.
Of course, that I guess does not modify the underlying files
on
the original collection where the (hypothetical) remove
would have been run.
If this indeed the case, is there some way to get the data
back? After all,
I do have the original .ns files.
Thanks!
Tob
On Monday, August 17, 2015 at 12:47:08 PM UTC-4, Ankit Kakkar
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the
investigation of this case, please provide us with
1) Describe your MongoDB deployment (standalone, replica
set,
or sharded cluster)
2) What version of MongoDB are you running? (You can check
it
with db.version() in the mongo shell)
3) Which query did you use to find or count the documents?
Can
you please run that query with explain() and send us the
output?
4) Output of db.collection.getIndexes() for the collection
where documents appear to be missing.
Regards,
ankit
On Monday, August 17, 2015 at 12:56:20 PM UTC+5:30, Chris
De
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents, what
is
the structure of the docs, are there indexes on this
collection?
On Sunday, August 16, 2015 at 12:33:16 AM UTC+2,
Post by t***@gmail.com
I have a mongodb database containing a collection that
has
remained unchanged for a period of time -- the underlying
.ns and .0, .1 etc
files have not been modified for months. Up until a few
weeks ago, I had no
problem reading records from the collection. There were
thousands of records
in the collection.
However, today when I attempted to read the records, many
of
them appeared missing -- e.g. records that I expected to
be there were not,
although some of the records were available. Now, there
appear to be only
250 records. =
I copied the database files and did a repair of the
(copied)
...
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/ea02d25c-7c17-47f1-a443-c80956fbc5cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJAtVac-ezkKNhTbdY%2BrsNp_cxqiwoY1pm%3DLqJ1ePQetwA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
t***@gmail.com
2015-08-31 13:32:59 UTC
Permalink
Thanks -- helpful. Still getting stuck on the compilation of mdbundo.

I follow your exact steps (including getting libbson from the source you
specified). But at "make mdbundo" I get a compilation error (see below).
Any thoughts?

Thanks!
T

cc -o mdbundo -Wall -Werror -O0 -ggdb -I/usr/local/include/libbson-1.0
-L/usr/local/lib -lbson-1.0 mdb.c mdb.h mdbundo.c
/tmp/cc1rVoR4.o: In function `db_init':
/home/brejicz/src/mdb-master/mdb.c:177: undefined reference to
`bson_strdup_printf'
/home/brejicz/src/mdb-master/mdb.c:179: undefined reference to `bson_free'
/home/brejicz/src/mdb-master/mdb.c:182: undefined reference to `bson_free'
/home/brejicz/src/mdb-master/mdb.c:189: undefined reference to
`bson_strdup_printf'
/home/brejicz/src/mdb-master/mdb.c:191: undefined reference to `bson_free'
/home/brejicz/src/mdb-master/mdb.c:195: undefined reference to
`bson_realloc'
/home/brejicz/src/mdb-master/mdb.c:197: undefined reference to `bson_free'
/home/brejicz/src/mdb-master/mdb.c:218: undefined reference to `bson_free'
/tmp/cc1rVoR4.o: In function `record_bson':
/home/brejicz/src/mdb-master/mdb.c:608: undefined reference to
`bson_init_static'
/tmp/ccvogrxm.o: In function `fixup_bson':
/home/brejicz/src/mdb-master/mdbundo.c:28: undefined reference to
`bson_init_static'
/home/brejicz/src/mdb-master/mdbundo.c:32: undefined reference to
`bson_iter_init'
/home/brejicz/src/mdb-master/mdbundo.c:36: undefined reference to
`bson_iter_next'
/tmp/ccvogrxm.o: In function `get_bson_at_loc':
/home/brejicz/src/mdb-master/mdbundo.c:85: undefined reference to
`bson_init_static'
/home/brejicz/src/mdb-master/mdbundo.c:89: undefined reference to
`bson_validate'
/tmp/ccvogrxm.o: In function `mdbundo':
/home/brejicz/src/mdb-master/mdbundo.c:123: undefined reference to
`bson_get_data'
collect2: ld returned 1 exit status
make: *** [mdbundo] Error 1
Post by Asya Kamsky
Here is what I know works.
https://github.com/mongodb/libbson/archive/1.0.0.zip
Unzip it, cd into it, assuming you have all dependencies installed
(compiler, etc) run
./autoconf.sh
make
sudo make install
Then download mdb the same way from
https://github.com/chergert/mdb/archive/master.zip
unzip and cd into it
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
make mdbundo
Now you should have an executable called ./mdbundo
export LD_LIBRARY_PATH=/usr/local/lib
./mdbundo --help
usage: mdbundo DBPATH DBNAME COLNAME
DBPATH is the path to the original files, DBNAME is myfs then you
would do this twice once for files and once for chunks.
If memory serves, the recovered bson will go to stderr so you want to
run it like
./mdbundo /path/to/my/dbfilesdir myfs files > myfs.files.bson
./mdbundo /path/to/my/dbfilesdir myfs chunks > myfs.chunks.bson
Let us know how it goes!
Asya
Post by t***@gmail.com
I should add that the relevant libbson files with those symbols seem to
be
Post by t***@gmail.com
properly installed, e.g. running
pkg-config --cflags --libs --libs libbson-1.0
returns
-I/usr/local/include/libbson-1.0 -L/usr/local/lib -lbson-1.0
and inside /usr/local/include/libbson-1.0, running "grep -R bson_free ."
./bson.h: * Returns: A newly allocated string that should be freed
with
Post by t***@gmail.com
bson_free().
./bson-memory.h:void bson_free (void *mem);
Hi --
Having trouble compiling mdb. I'm at commit
e7d018cad20f2a6aa62aea5db088a8e7b05a190d of libbson. Is that too
early/late?
To get the make to work at all, I included the following line in all
the
Post by t***@gmail.com
-I /usr/local/include/libbson-1.0
Is this right?
/home/brejicz/mdb/mdb.c:177: undefined reference to
`bson_strdup_printf'
Post by t***@gmail.com
/home/brejicz/mdb/mdb.c:179: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:182: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:189: undefined reference to
`bson_strdup_printf'
Post by t***@gmail.com
/home/brejicz/mdb/mdb.c:191: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:195: undefined reference to `bson_realloc'
/home/brejicz/mdb/mdb.c:197: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:218: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:608: undefined reference to `bson_init_static'
/home/brejicz/mdb/mdbdump.c:70: undefined reference to `bson_as_json'
/home/brejicz/mdb/mdbdump.c:73: undefined reference to `bson_free'
collect2: ld returned 1 exit status
make: *** [mdbdump] Error 1
Sorry for the detailed questions, but any help would be great.
Thanks!,
T
I'd suggest 1.0 or whichever version is tagged around the time of last
commit on mdb repo.
If you get the wrong version you'll find out during build.
Asya
Post by t***@gmail.com
Ok great -- thanks, will try this.
In terms of getting the right version of libbson, I guess that
getting a
Post by t***@gmail.com
Post by t***@gmail.com
commit from
https://github.com/mongodb/libbson/commits/
here is the the way to go? Which commit do you suggest I use?
Thanks!
T
Post by Asya Kamsky
Okay, well, I guess it can't hurt so you should try to see if
anything
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
dumped out with something that reads the raw files ends up helping
at
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
https://github.com/chergert/mdb
This was mostly an exercise in dumping out on-disk format (*note* to
anyone reading, this had never been updated since 2.4 so I'm only suggesting
it specifically because OP is using 2.4 - using this on later
versions
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
without changes will simply give errors).
To build this you will need libbson version from when this was
written
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
which I believe is 1.0. The executable you want to run is called mdbundo,
here's its usage.
usage: mdbundo DBPATH DBNAME COLNAME
what it will do is walk through the DB files looking for deleted records
and then try to restore them. I can't remember for sure but I
think
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
it
will send them to stdout so you can redirect the output of this to a file.
That file should have bson in them so you can use bsondump to view
them
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
(which is pointless for chunks but useful for "files") and you can
run
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
mongorestore on the output to load it into a new collection so you
can
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
then
figure out if anything in there is valuable.
Obviously you run it twice on the same DB files, once for
myfs.files,
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
once
for myfs.chunks.
Again, given the output to validate I don't really expect it to find
anything more than a handful of records at most, possibly unusable (since
for every "file" every chunk belonging to it must be present to have the
full usable file back.
Good luck and let us know what you uncover,
Asya
Post by t***@gmail.com
Thanks for the response -- sorry for the delay in reply (was away
from
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
computer).
On Saturday, August 22, 2015 at 7:00:54 PM UTC-4, Asya Kamsky
Post by Asya Kamsky
This is somewhat surprising. Are there other collections in this database?
[u'system.indexes', u'myfs.chunks', u'myfs.files']
Post by Asya Kamsky
This is the original files, right?
Yes.
Post by Asya Kamsky
Are you *sure* there are files missing? I don't mean by count but
by
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
content... You mentioned you were about to find some strings
matching
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
something you expected to find in the DB but didn't?
Definitely 100% completely sure. I can use "grep" on the binary files
of this database and I see at least parts of the missing records,
e.g.
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
some
keys that are unique and are exactly what the missing records would have.
Moreover, I (and several others) were happily using this collection
as
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
the
source a stable restful API for months ... at least, until this problem
occurred ...
Post by Asya Kamsky
How many dbname.ns files are there and what are their sizes?
In the original collection there are 90 .ns files, most of which
are
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
the
standard size "2146435072", but some of which (the early ones of course) are
smaller.
In the repaired collection there are 35 .ns files, same size
pattern.
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Thanks again for all the help --
T
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Asya,
1) The aggregation of the chunk collection produces 279
documents.
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
I
don't recall how many there were before the ones I want went missing.
However, I guess there more than 279. I don't know exactly how
many
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
more,
but I expect on the order of several hundred at least (but again, I'm not
sure). The collection does verify, in that all the chunks in
the
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
chunks
collection are accounted for by a single file in the files collection. I
computed the expected number of chunks for each file by dividing
length by
chunk size for each file record; that number exactly equals the count of
chunks for that file as produced by the aggregation in the chunk
collection.
{
"ns" : "mydb.myfs.files",
"firstExtent" : "0:19000 ns:mydb.myfs.files",
"lastExtent" : "7:67757000 ns:mydb.myfs.files",
"extentCount" : 5,
<snip "extents">
"datasize" : 24075568,
"nrecords" : 279,
"lastExtentSize" : 12902400,
"padding" : 1,
<snip some other stuff>
"objectsFound" : 279,
"invalidObjects" : 0,
"bytesWithHeaders" : 24080032,
"bytesWithoutHeaders" : 24075568,
"deletedCount" : 3,
"deletedSize" : 12008944,
"nIndexes" : 3,
"keysPerIndex" : {
"mydb.myfs.files.$_id_" : 279,
279,
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
"mydb.myfs.files.$timestamp_1" : 279
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
db.chunks.validate(true)
{
"ns" : "mydb.myfs.chunks",
"firstExtent" : "0:5000 ns:mydb.myfs.chunks",
"lastExtent" : "28:2000 ns:mydb.myfs.chunks",
"extentCount" : 43,
<snip "extents">
"datasize" : 49854830688,
"nrecords" : 190427,
"lastExtentSize" : 2146426864,
"padding" : 1,
"firstExtentDetails" : {
"loc" : "0:5000",
"xnext" : "0:162000",
"xprev" : "null",
"nsdiag" : "mydb.myfs.chunks",
"size" : 8192,
"firstRecord" : "0:50b0",
"lastRecord" : "0:6f28"
},
"lastExtentDetails" : {
"loc" : "28:2000",
"xnext" : "null",
"xprev" : "27:2000",
"nsdiag" : "mydb.myfs.chunks",
"size" : 2146426864,
"firstRecord" : "28:20b0",
"lastRecord" : "28:189c20b0"
},
"objectsFound" : 190427,
"invalidObjects" : 0,
"bytesWithHeaders" : 49857877520,
"bytesWithoutHeaders" : 49854830688,
"deletedCount" : 20,
"deletedSize" : 1733626640,
"nIndexes" : 2,
"keysPerIndex" : {
"mydb.myfs.chunks.$_id_" : 190427,
"mydb.myfs.chunks.$files_id_1_n_1" : 190427
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
I'm not sure whether "deletedCount" of "3" in the files
collection
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
would make up for all the deleted records that I expect ... if I
interpret
it correctly. It would also not make up for the huge compaction that
occurred in the repaired version of the collection (which is much
smaller on
disk, but still contains 279 records).
Any thoughts on what to do next? Thanks again for your help!
T
On Friday, August 21, 2015 at 2:02:27 PM UTC-4, Asya Kamsky
Post by Asya Kamsky
I'm assuming that you start up mongod with the original files
(ones
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
you did not repair but saved when you first noticed that
something
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
was
Okay, first, it would be good to know which collections are
missing
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
You'd mentioned having 279 documents in files collection - how
many
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
are in chunks collection? Do they "cross-verify"?
http://docs.mongodb.org/manual/reference/gridfs/
So you can tell for each file how many chunk documents there
should
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
be
and for each chunk document you can tell which file it belongs
to
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
and in
what order.
db.fs.chunks.aggregate({$sort:{files_id:1, num:1}},
{$group:{_id:"$files_id",count:{$sum:1},
chunks:{$push:"$num"}}}).
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
If you have 279 files in your fs.files collection, then you
should
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
get
back 279 documents from this aggregation, each corresponding to
a
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
"file" in
gridFS.
How many files do you expect to have?
db.fs.files.validate(true)
and
db.fs.chunks.validate(true)
http://docs.mongodb.org/manual/reference/command/validate/#output
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
You are interested in the deleted records
(
http://docs.mongodb.org/manual/reference/command/validate/#validate.deletedCount
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
and the next field, deletedSize).
Basically these are the records that can be potentially
recovered -
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
can you provide output to above experiments so we don't waste
time
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
trying to
recover data that's not still there?
Asya
Post by t***@gmail.com
Great thanks so much for the help!
T
On Friday, August 21, 2015 at 9:49:36 AM UTC-4, Asya Kamsky
Post by Asya Kamsky
Some of the records are in a free list and haven't been overwritten
yet. It may be possible to get them back.
Since you're using version 2.4.x I believe that there is code that
will walk through the DB files and dump out every document it
finds there.
Let me dig around for it - obviously there is (a) no guarantee that
this will restore all of them and (b) did you say these were
GridFS chunks -
if so then you won't get back a valid file unless all chunks
from
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
that file
are restored (along with corresponding files record).
Asya
Post by t***@gmail.com
Thanks for the suggestion. The logs don't appear to show
any
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
removals on that database since Oct. 2014. At least as far
as
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
these logs
go, it doesn't look to me like a remove was done.
Any suggestions for what to do now? I can definitely see traces
of the data I want to get at in the (unmodified) database
files.
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Is there
no way to get at the data in them?
Thanks!
T
On Wednesday, August 19, 2015 at 2:53:46 PM UTC-4, Asya
Kamsky
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
The only CRUD operations that would be logged would be those
that
take longer than 100ms.
2015-07-14T12:10:34.559-0500 I WRITE [conn4822] remove
blackbox.bigcoll query: { _id: { $gte: 10000.0 } }
ndeleted:90000
keyUpdates:0 writeConflicts:0 numYields:12330 locks:{
Global: {
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
acquireCount: { r: 12331, w: 12331 } }, Database: {
acquireCount: { w: 12331
} }, Collection: { acquireCount: { w: 12331 } } } 249168ms
Of course this was a huge mass remove of giant docs which is
why
it took so long. If the number of documents deleted was
small
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
(like if
they were deleted one at a time instead of via single
command)
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
then it's
much less likely they would have taken longer than 100ms.
I would suggest searching through the logs with equivalent
command
grep 'remove <dbname>' *.log.*
where you would replace <dbname> with the name of your
actual
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
DB
and replace *.log.* with path to your log files.
Asya
On Tuesday, August 18, 2015 at 1:53:13 PM UTC-4, Asya
Kamsky
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Do you have the logs from around the time the database
crashed?
When was that relative to when you noticed the records
missing?
Yes I have all the logs going back for a year. I noticed
the
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
records missing the day that I wrote the help question
originally (several
days ago). The crash happened several weeks ago.
Post by Asya Kamsky
Since you had journaling on, and you were able to restart
mongod
without any errors, I would rule out the crash *causing*
records to
disappear, which leaves you with the possibility that they
were deleted.
If they were deleted and you don't have any backups from
*before* they were
deleted, then there's not much that can be done to recover
the data - sure,
it's possible to write code to literally look through the
deleted list, but
if any data was inserted after the deletes then that space
would be reused
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But
wouldn't
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
there be some record in the logs if there was a deleation?
We haven't written any records since the deletes. (The
files
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
haven't changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm
using
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of
records shown -- not 250, like I said before, that was a
mistake.)
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1),
(u'uploadDate', -1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used
to
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
manage a gridfs instance.
Also I think I have some additional information. I
copied
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
the
database files to a new location and ran a full repair on
the whole
database. IN the new repaired version, there were many
fewer files .ns
files. As is as if some records were removed using e.g.
gridfs.GridFS.remove ... so in the new (compacted)
version
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
the data was
finally deleted. Maybe its somehow possible that the
records were removed
from the gridfs collection by someone running a remove
operation? I don't
see any evidence of such an operation in the logs but
perhaps it's possible?
I've looked carefully at the logs and am not seeing
anything
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
obvious, but
maybe dont' know what to look for.
Of course, that I guess does not modify the underlying
files
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
on
the original collection where the (hypothetical) remove
would have been run.
If this indeed the case, is there some way to get the
data
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
back? After all,
I do have the original .ns files.
Thanks!
Tob
On Monday, August 17, 2015 at 12:47:08 PM UTC-4, Ankit
Kakkar
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the
investigation of this case, please provide us with
1) Describe your MongoDB deployment (standalone, replica
set,
or sharded cluster)
2) What version of MongoDB are you running? (You can
check
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
it
with db.version() in the mongo shell)
3) Which query did you use to find or count the
documents?
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Can
you please run that query with explain() and send us the
output?
4) Output of db.collection.getIndexes() for the
collection
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
where documents appear to be missing.
Regards,
ankit
On Monday, August 17, 2015 at 12:56:20 PM UTC+5:30,
Chris
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
De
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents,
what
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
is
the structure of the docs, are there indexes on this
collection?
On Sunday, August 16, 2015 at 12:33:16 AM UTC+2,
Post by t***@gmail.com
I have a mongodb database containing a collection that
has
remained unchanged for a period of time -- the
underlying
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
.ns and .0, .1 etc
files have not been modified for months. Up until a
few
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
weeks ago, I had no
problem reading records from the collection. There
were
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
thousands of records
in the collection.
However, today when I attempted to read the records,
many
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
of
them appeared missing -- e.g. records that I expected
to
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
be there were not,
although some of the records were available. Now,
there
Post by t***@gmail.com
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by t***@gmail.com
Post by Asya Kamsky
Post by Asya Kamsky
Post by t***@gmail.com
Post by Ankit Kakkar
Post by Chris De Bruyne
Post by t***@gmail.com
appear to be only
250 records. =
I copied the database files and did a repair of the
(copied)
...
--
You received this message because you are subscribed to the Google
Groups
Post by t***@gmail.com
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google
Groups
Post by t***@gmail.com
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send
an
<javascript:>.
Post by t***@gmail.com
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/ea02d25c-7c17-47f1-a443-c80956fbc5cd%40googlegroups.com.
Post by t***@gmail.com
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/6f2898d9-6d00-4ff7-af8e-4d4abab4b523%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Asya Kamsky
2015-09-01 06:01:35 UTC
Permalink
When I build mdbundo my compile looks like this, right below it is yours:

cc -o mdbundo -Wall -Werror -O0 -ggdb -I/usr/local/include/libbson-1.0
-L/usr/local/lib -lbson-1.0 mdb.c mdb.h mdbundo.c

cc -o mdbundo -Wall -Werror -O0 -ggdb -I/usr/local/include/libbson-1.0
-L/usr/local/lib -lbson-1.0 mdb.c mdb.h mdbundo.c

So that looks the same which means some sort of environment may not
have been set up the same way.
I assume you build libbson without errors, and exported PKG_CONFIG_PATH?

Is it possible you have a different version of libson installed from
your previous attempts?

Asya
Post by t***@gmail.com
Thanks -- helpful. Still getting stuck on the compilation of mdbundo.
I follow your exact steps (including getting libbson from the source you
specified). But at "make mdbundo" I get a compilation error (see below).
Any thoughts?
Thanks!
T
cc -o mdbundo -Wall -Werror -O0 -ggdb -I/usr/local/include/libbson-1.0
-L/usr/local/lib -lbson-1.0 mdb.c mdb.h mdbundo.c
/home/brejicz/src/mdb-master/mdb.c:177: undefined reference to
`bson_strdup_printf'
/home/brejicz/src/mdb-master/mdb.c:179: undefined reference to `bson_free'
/home/brejicz/src/mdb-master/mdb.c:182: undefined reference to `bson_free'
/home/brejicz/src/mdb-master/mdb.c:189: undefined reference to
`bson_strdup_printf'
/home/brejicz/src/mdb-master/mdb.c:191: undefined reference to `bson_free'
/home/brejicz/src/mdb-master/mdb.c:195: undefined reference to
`bson_realloc'
/home/brejicz/src/mdb-master/mdb.c:197: undefined reference to `bson_free'
/home/brejicz/src/mdb-master/mdb.c:218: undefined reference to `bson_free'
/home/brejicz/src/mdb-master/mdb.c:608: undefined reference to
`bson_init_static'
/home/brejicz/src/mdb-master/mdbundo.c:28: undefined reference to
`bson_init_static'
/home/brejicz/src/mdb-master/mdbundo.c:32: undefined reference to
`bson_iter_init'
/home/brejicz/src/mdb-master/mdbundo.c:36: undefined reference to
`bson_iter_next'
/home/brejicz/src/mdb-master/mdbundo.c:85: undefined reference to
`bson_init_static'
/home/brejicz/src/mdb-master/mdbundo.c:89: undefined reference to
`bson_validate'
/home/brejicz/src/mdb-master/mdbundo.c:123: undefined reference to
`bson_get_data'
collect2: ld returned 1 exit status
make: *** [mdbundo] Error 1
Post by Asya Kamsky
Here is what I know works.
https://github.com/mongodb/libbson/archive/1.0.0.zip
Unzip it, cd into it, assuming you have all dependencies installed
(compiler, etc) run
./autoconf.sh
make
sudo make install
Then download mdb the same way from
https://github.com/chergert/mdb/archive/master.zip
unzip and cd into it
export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
make mdbundo
Now you should have an executable called ./mdbundo
export LD_LIBRARY_PATH=/usr/local/lib
./mdbundo --help
usage: mdbundo DBPATH DBNAME COLNAME
DBPATH is the path to the original files, DBNAME is myfs then you
would do this twice once for files and once for chunks.
If memory serves, the recovered bson will go to stderr so you want to
run it like
./mdbundo /path/to/my/dbfilesdir myfs files > myfs.files.bson
./mdbundo /path/to/my/dbfilesdir myfs chunks > myfs.chunks.bson
Let us know how it goes!
Asya
Post by t***@gmail.com
I should add that the relevant libbson files with those symbols seem to be
properly installed, e.g. running
pkg-config --cflags --libs --libs libbson-1.0
returns
-I/usr/local/include/libbson-1.0 -L/usr/local/lib -lbson-1.0
and inside /usr/local/include/libbson-1.0, running "grep -R bson_free ."
./bson.h: * Returns: A newly allocated string that should be freed with
bson_free().
./bson-memory.h:void bson_free (void *mem);
Hi --
Having trouble compiling mdb. I'm at commit
e7d018cad20f2a6aa62aea5db088a8e7b05a190d of libbson. Is that too
early/late?
To get the make to work at all, I included the following line in all the
-I /usr/local/include/libbson-1.0
Is this right?
/home/brejicz/mdb/mdb.c:177: undefined reference to
`bson_strdup_printf'
/home/brejicz/mdb/mdb.c:179: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:182: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:189: undefined reference to
`bson_strdup_printf'
/home/brejicz/mdb/mdb.c:191: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:195: undefined reference to `bson_realloc'
/home/brejicz/mdb/mdb.c:197: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:218: undefined reference to `bson_free'
/home/brejicz/mdb/mdb.c:608: undefined reference to `bson_init_static'
/home/brejicz/mdb/mdbdump.c:70: undefined reference to `bson_as_json'
/home/brejicz/mdb/mdbdump.c:73: undefined reference to `bson_free'
collect2: ld returned 1 exit status
make: *** [mdbdump] Error 1
Sorry for the detailed questions, but any help would be great.
Thanks!,
T
I'd suggest 1.0 or whichever version is tagged around the time of last
commit on mdb repo.
If you get the wrong version you'll find out during build.
Asya
Post by t***@gmail.com
Ok great -- thanks, will try this.
In terms of getting the right version of libbson, I guess that getting a
commit from
https://github.com/mongodb/libbson/commits/
here is the the way to go? Which commit do you suggest I use?
Thanks!
T
Post by Asya Kamsky
Okay, well, I guess it can't hurt so you should try to see if anything
dumped out with something that reads the raw files ends up helping at
https://github.com/chergert/mdb
This was mostly an exercise in dumping out on-disk format (*note* to
anyone reading, this had never been updated since 2.4 so I'm only suggesting
it specifically because OP is using 2.4 - using this on later versions
without changes will simply give errors).
To build this you will need libbson version from when this was written
which I believe is 1.0. The executable you want to run is called mdbundo,
here's its usage.
usage: mdbundo DBPATH DBNAME COLNAME
what it will do is walk through the DB files looking for deleted records
and then try to restore them. I can't remember for sure but I think
it
will send them to stdout so you can redirect the output of this to a file.
That file should have bson in them so you can use bsondump to view them
(which is pointless for chunks but useful for "files") and you can run
mongorestore on the output to load it into a new collection so you can
then
figure out if anything in there is valuable.
Obviously you run it twice on the same DB files, once for myfs.files,
once
for myfs.chunks.
Again, given the output to validate I don't really expect it to find
anything more than a handful of records at most, possibly unusable (since
for every "file" every chunk belonging to it must be present to have the
full usable file back.
Good luck and let us know what you uncover,
Asya
Post by t***@gmail.com
Thanks for the response -- sorry for the delay in reply (was away from
computer).
Post by Asya Kamsky
This is somewhat surprising. Are there other collections in this
database?
[u'system.indexes', u'myfs.chunks', u'myfs.files']
Post by Asya Kamsky
This is the original files, right?
Yes.
Post by Asya Kamsky
Are you *sure* there are files missing? I don't mean by count but by
content... You mentioned you were about to find some strings matching
something you expected to find in the DB but didn't?
Definitely 100% completely sure. I can use "grep" on the binary files
of this database and I see at least parts of the missing records, e.g.
some
keys that are unique and are exactly what the missing records would have.
Moreover, I (and several others) were happily using this collection as
the
source a stable restful API for months ... at least, until this problem
occurred ...
Post by Asya Kamsky
How many dbname.ns files are there and what are their sizes?
In the original collection there are 90 .ns files, most of which are
the
standard size "2146435072", but some of which (the early ones of
course) are
smaller.
In the repaired collection there are 35 .ns files, same size pattern.
Thanks again for all the help --
T
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Asya,
1) The aggregation of the chunk collection produces 279 documents.
I
don't recall how many there were before the ones I want went missing.
However, I guess there more than 279. I don't know exactly how many
more,
but I expect on the order of several hundred at least (but again,
I'm not
sure). The collection does verify, in that all the chunks in the
chunks
collection are accounted for by a single file in the files
collection. I
computed the expected number of chunks for each file by dividing
length by
chunk size for each file record; that number exactly equals the
count of
chunks for that file as produced by the aggregation in the chunk
collection.
{
"ns" : "mydb.myfs.files",
"firstExtent" : "0:19000 ns:mydb.myfs.files",
"lastExtent" : "7:67757000 ns:mydb.myfs.files",
"extentCount" : 5,
<snip "extents">
"datasize" : 24075568,
"nrecords" : 279,
"lastExtentSize" : 12902400,
"padding" : 1,
<snip some other stuff>
"objectsFound" : 279,
"invalidObjects" : 0,
"bytesWithHeaders" : 24080032,
"bytesWithoutHeaders" : 24075568,
"deletedCount" : 3,
"deletedSize" : 12008944,
"nIndexes" : 3,
"keysPerIndex" : {
"mydb.myfs.files.$_id_" : 279,
"mydb.myfs.files.$filename_1_uploadDate_-1" : 279,
"mydb.myfs.files.$timestamp_1" : 279
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
db.chunks.validate(true)
{
"ns" : "mydb.myfs.chunks",
"firstExtent" : "0:5000 ns:mydb.myfs.chunks",
"lastExtent" : "28:2000 ns:mydb.myfs.chunks",
"extentCount" : 43,
<snip "extents">
"datasize" : 49854830688,
"nrecords" : 190427,
"lastExtentSize" : 2146426864,
"padding" : 1,
"firstExtentDetails" : {
"loc" : "0:5000",
"xnext" : "0:162000",
"xprev" : "null",
"nsdiag" : "mydb.myfs.chunks",
"size" : 8192,
"firstRecord" : "0:50b0",
"lastRecord" : "0:6f28"
},
"lastExtentDetails" : {
"loc" : "28:2000",
"xnext" : "null",
"xprev" : "27:2000",
"nsdiag" : "mydb.myfs.chunks",
"size" : 2146426864,
"firstRecord" : "28:20b0",
"lastRecord" : "28:189c20b0"
},
"objectsFound" : 190427,
"invalidObjects" : 0,
"bytesWithHeaders" : 49857877520,
"bytesWithoutHeaders" : 49854830688,
"deletedCount" : 20,
"deletedSize" : 1733626640,
"nIndexes" : 2,
"keysPerIndex" : {
"mydb.myfs.chunks.$_id_" : 190427,
"mydb.myfs.chunks.$files_id_1_n_1" : 190427
},
"valid" : true,
"errors" : [ ],
"ok" : 1
}
I'm not sure whether "deletedCount" of "3" in the files collection
would make up for all the deleted records that I expect ... if I
interpret
it correctly. It would also not make up for the huge compaction that
occurred in the repaired version of the collection (which is much
smaller on
disk, but still contains 279 records).
Any thoughts on what to do next? Thanks again for your help!
T
Post by Asya Kamsky
I'm assuming that you start up mongod with the original files (ones
you did not repair but saved when you first noticed that something
was
Okay, first, it would be good to know which collections are missing
You'd mentioned having 279 documents in files collection - how many
are in chunks collection? Do they "cross-verify"?
http://docs.mongodb.org/manual/reference/gridfs/
So you can tell for each file how many chunk documents there should
be
and for each chunk document you can tell which file it belongs to
and in
what order.
db.fs.chunks.aggregate({$sort:{files_id:1, num:1}},
{$group:{_id:"$files_id",count:{$sum:1},
chunks:{$push:"$num"}}}).
If you have 279 files in your fs.files collection, then you should
get
back 279 documents from this aggregation, each corresponding to a
"file" in
gridFS.
How many files do you expect to have?
db.fs.files.validate(true)
and
db.fs.chunks.validate(true)
http://docs.mongodb.org/manual/reference/command/validate/#output
You are interested in the deleted records
(http://docs.mongodb.org/manual/reference/command/validate/#validate.deletedCount
and the next field, deletedSize).
Basically these are the records that can be potentially recovered -
can you provide output to above experiments so we don't waste time
trying to
recover data that's not still there?
Asya
Post by t***@gmail.com
Great thanks so much for the help!
T
Post by Asya Kamsky
Some of the records are in a free list and haven't been
overwritten
yet. It may be possible to get them back.
Since you're using version 2.4.x I believe that there is code
that
will walk through the DB files and dump out every document it
finds there.
Let me dig around for it - obviously there is (a) no guarantee
that
this will restore all of them and (b) did you say these were
GridFS chunks -
if so then you won't get back a valid file unless all chunks from
that file
are restored (along with corresponding files record).
Asya
Post by t***@gmail.com
Thanks for the suggestion. The logs don't appear to show any
removals on that database since Oct. 2014. At least as far
as
these logs
go, it doesn't look to me like a remove was done.
Any suggestions for what to do now? I can definitely see
traces
of the data I want to get at in the (unmodified) database
files.
Is there
no way to get at the data in them?
Thanks!
T
On Wednesday, August 19, 2015 at 2:53:46 PM UTC-4, Asya
Kamsky
Post by Asya Kamsky
The only CRUD operations that would be logged would be those
that
take longer than 100ms.
2015-07-14T12:10:34.559-0500 I WRITE [conn4822] remove
blackbox.bigcoll query: { _id: { $gte: 10000.0 } }
ndeleted:90000
keyUpdates:0 writeConflicts:0 numYields:12330 locks:{
Global: {
acquireCount: { r: 12331, w: 12331 } }, Database: {
acquireCount: { w: 12331
} }, Collection: { acquireCount: { w: 12331 } } } 249168ms
Of course this was a huge mass remove of giant docs which is
why
it took so long. If the number of documents deleted was
small
(like if
they were deleted one at a time instead of via single
command)
then it's
much less likely they would have taken longer than 100ms.
I would suggest searching through the logs with equivalent
command
grep 'remove <dbname>' *.log.*
where you would replace <dbname> with the name of your actual
DB
and replace *.log.* with path to your log files.
Asya
On Tuesday, August 18, 2015 at 1:53:13 PM UTC-4, Asya
Kamsky
Post by Asya Kamsky
Do you have the logs from around the time the database
crashed?
When was that relative to when you noticed the records
missing?
Yes I have all the logs going back for a year. I noticed
the
records missing the day that I wrote the help question
originally (several
days ago). The crash happened several weeks ago.
Post by Asya Kamsky
Since you had journaling on, and you were able to restart
mongod
without any errors, I would rule out the crash *causing*
records to
disappear, which leaves you with the possibility that they
were deleted.
If they were deleted and you don't have any backups from
*before* they were
deleted, then there's not much that can be done to recover
the data - sure,
it's possible to write code to literally look through the
deleted list, but
if any data was inserted after the deletes then that space
would be reused
and the old data would be overwritten forever. :(
The backups are not from before unfortunately. But
wouldn't
there be some record in the logs if there was a deleation?
We haven't written any records since the deletes. (The
files
haven't changed for months.)
Post by Asya Kamsky
Asya
Post by t***@gmail.com
Hi Ankit --
Thanks so much for your reply -- it's really appreciated.
1) it's a standalone deployment, with no replica set (I'm
using
journaling though)
2) version is 2.4.6
{u'allPlans': [{u'cursor': u'BasicCursor',
u'indexBounds': {},
u'n': 0,
u'nscanned': 279,
u'nscannedObjects': 279}],
u'cursor': u'BasicCursor',
u'indexBounds': {},
u'indexOnly': False,
u'isMultiKey': False,
u'millis': 176,
u'n': 0,
u'nChunkSkips': 0,
u'nYields': 1,
u'nscanned': 279,
u'nscannedAllPlans': 279,
u'nscannedObjects': 279,
u'nscannedObjectsAllPlans': 279,
u'scanAndOrder': False,
(I think is saying 279 here because that is the number of
records shown -- not 250, like I said before, that was a
mistake.)
{u'_id_': {u'key': [(u'_id', 1)], u'v': 1},
u'filename_1_uploadDate_-1': {u'key': [(u'filename', 1),
(u'uploadDate', -1)],
u'v': 1},
u'timestamp_1': {u'key': [(u'timestamp', 1)], u'v': 1}}
As you can see, this is from a ".files" collection used
to
manage a gridfs instance.
Also I think I have some additional information. I
copied
the
database files to a new location and ran a full repair on
the whole
database. IN the new repaired version, there were many
fewer files .ns
files. As is as if some records were removed using e.g.
gridfs.GridFS.remove ... so in the new (compacted)
version
the data was
finally deleted. Maybe its somehow possible that the
records were removed
from the gridfs collection by someone running a remove
operation? I don't
see any evidence of such an operation in the logs but
perhaps it's possible?
I've looked carefully at the logs and am not seeing
anything
obvious, but
maybe dont' know what to look for.
Of course, that I guess does not modify the underlying
files
on
the original collection where the (hypothetical) remove
would have been run.
If this indeed the case, is there some way to get the
data
back? After all,
I do have the original .ns files.
Thanks!
Tob
On Monday, August 17, 2015 at 12:47:08 PM UTC-4, Ankit
Kakkar
Post by Ankit Kakkar
Hello Tobjan,
Thanks for reaching out to us. To assist us in the
investigation of this case, please provide us with
1) Describe your MongoDB deployment (standalone, replica
set,
or sharded cluster)
2) What version of MongoDB are you running? (You can
check
it
with db.version() in the mongo shell)
3) Which query did you use to find or count the
documents?
Can
you please run that query with explain() and send us the
output?
4) Output of db.collection.getIndexes() for the
collection
where documents appear to be missing.
Regards,
ankit
On Monday, August 17, 2015 at 12:56:20 PM UTC+5:30,
Chris
De
Post by Chris De Bruyne
Can you give some more info?
Like which query are you doing to find the documents,
what
is
the structure of the docs, are there indexes on this
collection?
On Sunday, August 16, 2015 at 12:33:16 AM UTC+2,
Post by t***@gmail.com
I have a mongodb database containing a collection that
has
remained unchanged for a period of time -- the
underlying
.ns and .0, .1 etc
files have not been modified for months. Up until a
few
weeks ago, I had no
problem reading records from the collection. There
were
thousands of records
in the collection.
However, today when I attempted to read the records,
many
of
them appeared missing -- e.g. records that I expected
to
be there were not,
although some of the records were available. Now,
there
appear to be only
250 records. =
I copied the database files and did a repair of the
(copied)
...
--
You received this message because you are subscribed to the Google Groups
"mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/ea02d25c-7c17-47f1-a443-c80956fbc5cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.
http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups
"mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit
https://groups.google.com/d/msgid/mongodb-user/6f2898d9-6d00-4ff7-af8e-4d4abab4b523%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/CAOe6dJDTQ7N7sXLnHv8HqcCKkxZ%3DwMBhMVnM34A%3DMfzS9bLYKg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.
Continue reading on narkive:
Loading...