[mongodb-user] MongoDB: Change block_compressor for existing collection

[Tobias Kempkensteffen]

Hi,


I have a server running with wiredTiger storage engine, but without
compression enabled globally.


Now I want to change the "block_compressor" option for a single collection
to use "snappy".

According to the mongoDB documentation this is possible while creating a
new collection using "db.createCollection()", but how can I perform this
step for an existing one? (Maybe for new replica set members before initial
sync)


Since initial sync between non-wiredTiger and wiredTiger members in a
replica set is performed on a per-document level (and not by copying the
files), this should be possible...


http://docs.mongodb.org/manual/reference/method/db.createCollection/#specify-storage-engine-options
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: http://www.mongodb.org/about/support/.
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at http://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/a45bbf4f-272e-4e41-a580-0a109fce684a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tobias Kempkensteffen]

Still having the problem.
Am I the only one with existing collections!? :)

Post by Tobias Kempkensteffen
Hi,
I have a server running with wiredTiger storage engine, but without
compression enabled globally.
Now I want to change the "block_compressor" option for a single collection
to use "snappy".
According to the mongoDB documentation this is possible while creating a
new collection using "db.createCollection()", but how can I perform this
step for an existing one? (Maybe for new replica set members before initial
sync)
Since initial sync between non-wiredTiger and wiredTiger members in a
replica set is performed on a per-document level (and not by copying the
files), this should be possible...
http://docs.mongodb.org/manual/reference/method/db.createCollection/#specify-storage-engine-options

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.org/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/8e642c1e-6873-468c-ac19-95baea9d79e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kevin Adistambha]

Hi Tobias,

Now I want to change the “block_compressor” option for a single collection
to use “snappy”.

It is not possible to change the block_compressor option of an existing
collection.

However, you can create a new collection with the required options and
replace the old collection:

1. Create a new collection with the parameters you need, as described in
the Specify Storage Engine Options
<https://docs.mongodb.org/manual/reference/method/db.createCollection/#specify-storage-engine-options>
page. E.g.: db.createCollection( "coll_snappy", { storageEngine: {
wiredTiger: { configString: 'block_compressor=snappy' }}})
2. Copy all data from the existing collection into the new collection.
*Note:* Please refrain from adding new data into the old collection
during this stage.
3. Drop the old collection and rename the new collection using
db.collection.renameCollection()
<https://docs.mongodb.org/manual/reference/method/db.collection.renameCollection/>,
so that the new collection replaces the old collection.

Since initial sync between non-wiredTiger and wiredTiger members in a
replica set is performed on a per-document level (and not by copying the
files), this should be possible


It is possible to mix different storage engines in a replica set, but it is
currently not possible (as of MongoDB 3.2) to set different storage engine
*options* for a collection in a replica set. I.e., if you specify a
collection to use the snappy compressor, the collection will be using
snappy across all secondaries as well (since collection creation options
are replicated to the secondaries).

Best regards,
Kevin
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.org/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/7e3f4c8b-56cd-4255-a231-e32280b3b049%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tobias Kempkensteffen]

Hey Kevin,

thanks for your reply!
I already thought about creating a new (compressed) collection and to simpy
copy the data, but the collection is about 400GB in size. This would lead
to some days of offline time, which is not possible for us.
My new plan is to add a new replica set member with enabled snappy
compression and instead of copying the big collection, I would copy all
other collections (which are much smaller) to new collections with disabled
compression. Not perfect, since I have to disable compression for every new
collection, but may be a way to handle the problem.

What do you think about this way?

Best regards,
Tobias

Post by Kevin Adistambha
Hi Tobias,
Now I want to change the “block_compressor” option for a single collection
to use “snappy”.
It is not possible to change the block_compressor option of an existing
collection.
However, you can create a new collection with the required options and
1. Create a new collection with the parameters you need, as described
in the Specify Storage Engine Options
<https://docs.mongodb.org/manual/reference/method/db.createCollection/#specify-storage-engine-options>
page. E.g.: db.createCollection( "coll_snappy", { storageEngine: {
wiredTiger: { configString: 'block_compressor=snappy' }}})
2. Copy all data from the existing collection into the new collection.
*Note:* Please refrain from adding new data into the old collection
during this stage.
3. Drop the old collection and rename the new collection using
db.collection.renameCollection()
<https://docs.mongodb.org/manual/reference/method/db.collection.renameCollection/>,
so that the new collection replaces the old collection.
Since initial sync between non-wiredTiger and wiredTiger members in a
replica set is performed on a per-document level (and not by copying the
files), this should be possible

It is possible to mix different storage engines in a replica set, but it
is currently not possible (as of MongoDB 3.2) to set different storage
engine *options* for a collection in a replica set. I.e., if you specify
a collection to use the snappy compressor, the collection will be using
snappy across all secondaries as well (since collection creation options
are replicated to the secondaries).
Best regards,
Kevin
​

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.org/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/89d99a55-483a-461c-8ff7-f03570aa8d17%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Kevin Adistambha]

Hi Tobias,

My new plan is to add a new replica set member with enabled snappy
compression and instead of copying the big collection, I would copy all
other collections (which are much smaller) to new collections with disabled
compression. Not perfect, since I have to disable compression for every new
collection, but may be a way to handle the problem.

May I ask why you want to turn off compression for some of the collections?
Why not just use Snappy compression (the default in WiredTiger engine
<https://docs.mongodb.org/manual/core/wiredtiger/#compression>) for all the
collections instead? Snappy requires relatively low CPU resources, but
provides a reasonable level of compression. Compressed data has the
advantage of lowering disk usage (since there’s less data to transfer) and
more efficient memory use in general. Also, if a document cannot be
compressed, Snappy will store it uncompressed. Please see the Snappy page
<https://google.github.io/snappy/> for more information.

Once compression is enabled in a specific mongod (e.g. Snappy by default,
none, or zlib using --wiredTigerCollectionBlockCompressor
<https://docs.mongodb.org/manual/reference/configuration-options/#storage.wiredTiger.collectionConfig.blockCompressor>),
all new collection created in that particular mongod will use that block
compressor.

This behaviour can be changed by creating a new collection using
db.createCollection with specific storage engine options
<https://docs.mongodb.org/manual/reference/method/db.createCollection/#specify-storage-engine-options>.
However please be aware that any collection-specific setting will override
any mongod setting, and it is not possible to change those settings anymore
unless the collection is recreated.

If your use case allows it, I would recommend to use Snappy compression for
all collections, since the benefits (lower disk usage, more efficient
memory usage) generally outweighs the drawbacks (sightly higher CPU
utilization). Another benefit for your case is that you don’t need to
micromanage collection-level specific settings.

Best regards,
Kevin
​
--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.org/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/e1f479f6-6111-4963-8ab4-d4566dca605b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.