Manually removing empty chunks?

[Derek Chen-Becker]

I've looked at https://jira.mongodb.org/browse/SERVER-2487 and we're in a
similar situation. Part of our sharding key is a timestamp which works fine
for use because the rest of the key makes write balance out across the
cluster. However, we're at a point now where we need to archive off older
data, and our shards are starting to get out of balance in terms of disk
usage because while chunks counts are balanced, some of those chunks are
empty and will never again hold data. I understand why it would be
difficult to automate this, but I'm wondering if there's any harm in
removing chunks manually.

Essentially, can I stop balancing on the cluster, delete some of the empty
chunks from the config DB, then restart balancing? Would I need to restart
mongos on the servers as well? Is this a fantastically dangerous thing to
attempt?

Thanks,

Derek

[Gianfranco]

In general it's no recommended to delete chunks even if they are empty.

As you mentioned there is SERVER ticket on Jira to implement this in a
secure manner but it's not being planned yet.
You can of course vote to increase the chances of being developed first.

Though my suggestion would be to move the data you want to archive into
another database or cluster.
When adding more space the empty chunks will be reused.

Gianfranco

Post by Derek Chen-Becker
I've looked at https://jira.mongodb.org/browse/SERVER-2487 and we're in a
similar situation. Part of our sharding key is a timestamp which works fine
for use because the rest of the key makes write balance out across the
cluster. However, we're at a point now where we need to archive off older
data, and our shards are starting to get out of balance in terms of disk
usage because while chunks counts are balanced, some of those chunks are
empty and will never again hold data. I understand why it would be
difficult to automate this, but I'm wondering if there's any harm in
removing chunks manually.
Essentially, can I stop balancing on the cluster, delete some of the empty
chunks from the config DB, then restart balancing? Would I need to restart
mongos on the servers as well? Is this a fantastically dangerous thing to
attempt?
Thanks,
Derek

[Derek Chen-Becker]

I'm not sure what you mean by moving the data into another DB or cluster.
I'm flat out removing it (as in db.<collection>.remove(...)) and then
compacting the replsets. I have 7 replsets in the cluster, and while
printShardStatus() shows that they all have equal chunk counts (+/- a few),
some replsets are using 85% of disk while others are using more like 55%.
The reason is that while they all have equal chunks, some of the chunks are
empty. Am I missing something here? At least if empty chunks could be
removed, then the config servers would try and rebalance the chunks that
*do* have data in them. Right now I'm looking at a situation where I may
need to add more shards to the system to prevent running out of disk on
specific replsets, even though my overall cluster disk utilization is 50%
of capacity.

Thanks,

Derek

Post by Gianfranco
In general it's no recommended to delete chunks even if they are empty.
As you mentioned there is SERVER ticket on Jira to implement this in a
secure manner but it's not being planned yet.
You can of course vote to increase the chances of being developed first.
Though my suggestion would be to move the data you want to archive into
another database or cluster.
When adding more space the empty chunks will be reused.
Gianfranco

Post by Derek Chen-Becker
I've looked at https://jira.mongodb.org/browse/SERVER-2487 and we're in
a similar situation. Part of our sharding key is a timestamp which works
fine for use because the rest of the key makes write balance out across the
cluster. However, we're at a point now where we need to archive off older
data, and our shards are starting to get out of balance in terms of disk
usage because while chunks counts are balanced, some of those chunks are
empty and will never again hold data. I understand why it would be
difficult to automate this, but I'm wondering if there's any harm in
removing chunks manually.
Essentially, can I stop balancing on the cluster, delete some of the
empty chunks from the config DB, then restart balancing? Would I need to
restart mongos on the servers as well? Is this a fantastically dangerous
thing to attempt?
Thanks,
Derek

[Otis Zein]

Could you elaborate why it's not recommended to remove empty chunks.

We are almost exactly in the same situation, our shard key is based partially on timestamp.  In our case, we delete old data and have no need to archive it.  With our shard key and data, we will never see data hit an empty chunk (ie, one which we deleted data from).


________________________________
From: Gianfranco <gianfranco-***@public.gmane.org>
To: mongodb-user-/***@public.gmane.org
Sent: Thursday, October 11, 2012 6:04 AM
Subject: [mongodb-user] Re: Manually removing empty chunks?


In general it's no recommended to delete chunks even if they are empty.

As you mentioned there is SERVER ticket on Jira to implement this in a secure manner but it's not being planned yet.
You can of course vote to increase the chances of being developed first.

Though my suggestion would be to move the data you want to archive into another database or cluster.
When adding more space the empty chunks will be reused.

Gianfranco

On Monday, October 8, 2012 6:03:57 PM UTC+1, Derek Chen-Becker wrote:
I've looked at https://jira.mongodb.org/ browse/SERVER-2487 and we're in a similar situation. Part of our sharding key is a timestamp which works fine for use because the rest of the key makes write balance out across the cluster. However, we're at a point now where we need to archive off older data, and our shards are starting to get out of balance in terms of disk usage because while chunks counts are balanced, some of those chunks are empty and will never again hold data. I understand why it would be difficult to automate this, but I'm wondering if there's any harm in removing chunks manually.

Essentially, can I stop balancing on the cluster, delete some of the empty chunks from the config DB, then restart balancing? Would I need to restart mongos on the servers as well? Is this a fantastically dangerous thing to attempt?
Thanks,
Derek

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/***@public.gmane.org
To unsubscribe from this group, send email to
mongodb-user+unsubscribe-/***@public.gmane.org
See also the IRC channel -- freenode.net#mongodb

[Otis Zein]

Bump



________________________________
From: Otis Zein <otiszein-/***@public.gmane.org>
To: "mongodb-user-/***@public.gmane.org" <mongodb-user-/***@public.gmane.org>
Sent: Friday, October 12, 2012 2:11 PM
Subject: Re: [mongodb-user] Re: Manually removing empty chunks?


Could you elaborate why it's not recommended to remove empty chunks.

We are almost exactly in the same situation, our shard key is based partially on timestamp.  In our case, we delete old data and have no need to archive it.  With our shard key and data, we will never see data hit an empty chunk (ie, one which we deleted data from).


________________________________
From: Gianfranco <gianfranco-***@public.gmane.org>
To: mongodb-user-/***@public.gmane.org
Sent: Thursday, October 11, 2012 6:04 AM
Subject: [mongodb-user] Re: Manually removing empty chunks?


In general it's no recommended to delete chunks even if they are empty.

As you mentioned there is SERVER ticket on Jira to implement this in a secure manner but it's not being planned yet.
You can of course vote to increase the chances of being developed first.

Though my suggestion would be to move the data you want to archive into another database or cluster.
When adding more space the empty chunks will be reused.

Gianfranco

On Monday, October 8, 2012 6:03:57 PM UTC+1, Derek Chen-Becker wrote:
I've looked at https://jira.mongodb.org/ browse/SERVER-2487 and we're in a similar situation. Part of our sharding key is a timestamp which works fine for use because the rest of the key makes write balance out across the cluster. However, we're at a point now where we need to archive off older data, and our shards are starting to get out of balance in terms of disk usage because while chunks counts are balanced, some of those chunks are empty and will never again hold data. I understand why it would be difficult to automate this, but I'm wondering if there's any harm in removing chunks manually.

Essentially, can I stop balancing on the cluster, delete some of the empty chunks from the config DB, then restart balancing? Would I need to restart mongos on the servers as well? Is this a fantastically dangerous thing to attempt?
Thanks,
Derek

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/***@public.gmane.org
To unsubscribe from this group, send email to
mongodb-user+unsubscribe-/***@public.gmane.org
See also the IRC channel -- freenode.net#mongodb
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/***@public.gmane.org
To unsubscribe from this group, send email to
mongodb-user+unsubscribe-/***@public.gmane.org
See also the IRC channel -- freenode.net#mongodb

[Gianfranco]

Hi Otis,

If you want to keep certain documents for a certain amount of time
depending on a Date attribute, you can use a TTL Collection.
This collection uses a special index and it automatically deletes "expired"
documents.
http://docs.mongodb.org/manual/tutorial/expire-data/

Regards,
Gianfranco

Bump
------------------------------
*Sent:* Friday, October 12, 2012 2:11 PM
*Subject:* Re: [mongodb-user] Re: Manually removing empty chunks?
Could you elaborate why it's not recommended to remove empty chunks.
We are almost exactly in the same situation, our shard key is based
partially on timestamp. In our case, we delete old data and have no need
to archive it. With our shard key and data, we will never see data hit an
empty chunk (ie, one which we deleted data from).
------------------------------
*Sent:* Thursday, October 11, 2012 6:04 AM
*Subject:* [mongodb-user] Re: Manually removing empty chunks?
In general it's no recommended to delete chunks even if they are empty.
As you mentioned there is SERVER ticket on Jira to implement this in a
secure manner but it's not being planned yet.
You can of course vote to increase the chances of being developed first.
Though my suggestion would be to move the data you want to archive into
another database or cluster.
When adding more space the empty chunks will be reused.
Gianfranco
I've looked at https://jira.mongodb.org/ browse/SERVER-2487<https://jira.mongodb.org/browse/SERVER-2487>and we're in a similar situation. Part of our sharding key is a timestamp
which works fine for use because the rest of the key makes write balance
out across the cluster. However, we're at a point now where we need to
archive off older data, and our shards are starting to get out of balance
in terms of disk usage because while chunks counts are balanced, some of
those chunks are empty and will never again hold data. I understand why it
would be difficult to automate this, but I'm wondering if there's any harm
in removing chunks manually.
Essentially, can I stop balancing on the cluster, delete some of the empty
chunks from the config DB, then restart balancing? Would I need to restart
mongos on the servers as well? Is this a fantastically dangerous thing to
attempt?
Thanks,
Derek
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group, send email to
See also the IRC channel -- freenode.net#mongodb
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group, send email to
See also the IRC channel -- freenode.net#mongodb

[Derek Chen-Becker]

Just as a follow up, I manually merged/removed about 1400 chunks last night
and the rebalance is complete now. Disk usage dropped by the expected
amount across our shards and things are much happier :)

Post by Gianfranco
Hi Otis,
If you want to keep certain documents for a certain amount of time
depending on a Date attribute, you can use a TTL Collection.
This collection uses a special index and it automatically deletes
"expired" documents.
http://docs.mongodb.org/manual/tutorial/expire-data/
Regards,
Gianfranco

Bump
------------------------------
*Sent:* Friday, October 12, 2012 2:11 PM
*Subject:* Re: [mongodb-user] Re: Manually removing empty chunks?
Could you elaborate why it's not recommended to remove empty chunks.
We are almost exactly in the same situation, our shard key is based
partially on timestamp. In our case, we delete old data and have no need
to archive it. With our shard key and data, we will never see data hit an
empty chunk (ie, one which we deleted data from).
------------------------------
*Sent:* Thursday, October 11, 2012 6:04 AM
*Subject:* [mongodb-user] Re: Manually removing empty chunks?
In general it's no recommended to delete chunks even if they are empty.
As you mentioned there is SERVER ticket on Jira to implement this in a
secure manner but it's not being planned yet.
You can of course vote to increase the chances of being developed first.
Though my suggestion would be to move the data you want to archive into
another database or cluster.
When adding more space the empty chunks will be reused.
Gianfranco
I've looked at https://jira.mongodb.org/ browse/SERVER-2487<https://jira.mongodb.org/browse/SERVER-2487>and we're in a similar situation. Part of our sharding key is a timestamp
which works fine for use because the rest of the key makes write balance
out across the cluster. However, we're at a point now where we need to
archive off older data, and our shards are starting to get out of balance
in terms of disk usage because while chunks counts are balanced, some of
those chunks are empty and will never again hold data. I understand why it
would be difficult to automate this, but I'm wondering if there's any harm
in removing chunks manually.
Essentially, can I stop balancing on the cluster, delete some of the
empty chunks from the config DB, then restart balancing? Would I need to
restart mongos on the servers as well? Is this a fantastically dangerous
thing to attempt?
Thanks,
Derek
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group, send email to
See also the IRC channel -- freenode.net#mongodb
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group, send email to
See also the IRC channel -- freenode.net#mongodb

[Ted]

Derek - How did you accomplish the merge and remove of chunks?

To remove the chunk, are you just removing it from the config.chunks
collection?

On Thu, Oct 25, 2012 at 11:53 AM, Derek Chen-Becker

Post by Derek Chen-Becker
Just as a follow up, I manually merged/removed about 1400 chunks last
night and the rebalance is complete now. Disk usage dropped by the expected
amount across our shards and things are much happier :)

Post by Gianfranco
Hi Otis,
If you want to keep certain documents for a certain amount of time
depending on a Date attribute, you can use a TTL Collection.
This collection uses a special index and it automatically deletes
"expired" documents.
http://docs.mongodb.org/**manual/tutorial/expire-data/<http://docs.mongodb.org/manual/tutorial/expire-data/>
Regards,
Gianfranco

Bump
------------------------------
*Sent:* Friday, October 12, 2012 2:11 PM
*Subject:* Re: [mongodb-user] Re: Manually removing empty chunks?
Could you elaborate why it's not recommended to remove empty chunks.
We are almost exactly in the same situation, our shard key is based
partially on timestamp. In our case, we delete old data and have no need
to archive it. With our shard key and data, we will never see data hit an
empty chunk (ie, one which we deleted data from).
------------------------------
*Sent:* Thursday, October 11, 2012 6:04 AM
*Subject:* [mongodb-user] Re: Manually removing empty chunks?
In general it's no recommended to delete chunks even if they are empty.
As you mentioned there is SERVER ticket on Jira to implement this in a
secure manner but it's not being planned yet.
You can of course vote to increase the chances of being developed first.
Though my suggestion would be to move the data you want to archive into
another database or cluster.
When adding more space the empty chunks will be reused.
Gianfranco
I've looked at https://jira.mongodb.org/ browse/SERVER-2487<https://jira.mongodb.org/browse/SERVER-2487>and we're in a similar situation. Part of our sharding key is a timestamp
which works fine for use because the rest of the key makes write balance
out across the cluster. However, we're at a point now where we need to
archive off older data, and our shards are starting to get out of balance
in terms of disk usage because while chunks counts are balanced, some of
those chunks are empty and will never again hold data. I understand why it
would be difficult to automate this, but I'm wondering if there's any harm
in removing chunks manually.
Essentially, can I stop balancing on the cluster, delete some of the
empty chunks from the config DB, then restart balancing? Would I need to
restart mongos on the servers as well? Is this a fantastically dangerous
thing to attempt?
Thanks,
Derek
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group, send email to
See also the IRC channel -- freenode.net#mongodb
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group, send email to
See also the IRC channel -- freenode.net#mongodb
--

You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group, send email to
See also the IRC channel -- freenode.net#mongodb

[Derek Chen-Becker]

I should have kept my mouth shut. Shard metadata started corrupting this
morning (chunk min/max keys were reversed on a large number of chunks)
after some chunk splits. After a long battle I ended up having to dump the
data off of each shard, wipe the database from the cluster, recreate the DB
and indices, and I'm in the process now of restoring data. Not something
I'd recommend doing again.

Derek

Post by Ted
Derek - How did you accomplish the merge and remove of chunks?
To remove the chunk, are you just removing it from the config.chunks
collection?

Post by Derek Chen-Becker
Just as a follow up, I manually merged/removed about 1400 chunks last
night and the rebalance is complete now. Disk usage dropped by the expected
amount across our shards and things are much happier :)

Post by Gianfranco
Hi Otis,
If you want to keep certain documents for a certain amount of time
depending on a Date attribute, you can use a TTL Collection.
This collection uses a special index and it automatically deletes
"expired" documents.
http://docs.mongodb.org/**manual/tutorial/expire-data/<http://docs.mongodb.org/manual/tutorial/expire-data/>
Regards,
Gianfranco

Bump
------------------------------
*Sent:* Friday, October 12, 2012 2:11 PM
*Subject:* Re: [mongodb-user] Re: Manually removing empty chunks?
Could you elaborate why it's not recommended to remove empty chunks.
We are almost exactly in the same situation, our shard key is based
partially on timestamp. In our case, we delete old data and have no need
to archive it. With our shard key and data, we will never see data hit an
empty chunk (ie, one which we deleted data from).
------------------------------
*Sent:* Thursday, October 11, 2012 6:04 AM
*Subject:* [mongodb-user] Re: Manually removing empty chunks?
In general it's no recommended to delete chunks even if they are empty.
As you mentioned there is SERVER ticket on Jira to implement this in a
secure manner but it's not being planned yet.
You can of course vote to increase the chances of being developed first.
Though my suggestion would be to move the data you want to archive into
another database or cluster.
When adding more space the empty chunks will be reused.
Gianfranco
I've looked at https://jira.mongodb.org/ browse/SERVER-2487<https://jira.mongodb.org/browse/SERVER-2487>and we're in a similar situation. Part of our sharding key is a timestamp
which works fine for use because the rest of the key makes write balance
out across the cluster. However, we're at a point now where we need to
archive off older data, and our shards are starting to get out of balance
in terms of disk usage because while chunks counts are balanced, some of
those chunks are empty and will never again hold data. I understand why it
would be difficult to automate this, but I'm wondering if there's any harm
in removing chunks manually.
Essentially, can I stop balancing on the cluster, delete some of the
empty chunks from the config DB, then restart balancing? Would I need to
restart mongos on the servers as well? Is this a fantastically dangerous
thing to attempt?
Thanks,
Derek
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group, send email to
See also the IRC channel -- freenode.net#mongodb
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group, send email to
See also the IRC channel -- freenode.net#mongodb
--

You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group, send email to
See also the IRC channel -- freenode.net#mongodb

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group, send email to
See also the IRC channel -- freenode.net#mongodb

[Jim Reitz]

I realize this is a late response, but I'll post it anyway, just in case
someone searches for this issue and finds this conversation.
The reliable way to do this is to:
Disable the balancer
find the contiguous ranges of empty chunks, and move them all to the same
shard (sh.moveChunk())
merge them all together using the mergeChunks command (db.runCommand())
After you've finished merging all the empty chunks into ones that have
data, your done. Turn the balancer back on. Never delete chunks from the
config.chunks collection manually.

Note that the above steps are now documented in the MongoDB documentation,
however it's still a manual process.

Jim

Post by Derek Chen-Becker
I should have kept my mouth shut. Shard metadata started corrupting this
morning (chunk min/max keys were reversed on a large number of chunks)
after some chunk splits. After a long battle I ended up having to dump the
data off of each shard, wipe the database from the cluster, recreate the DB
and indices, and I'm in the process now of restoring data. Not something
I'd recommend doing again.
Derek

Post by Ted
Derek - How did you accomplish the merge and remove of chunks?
To remove the chunk, are you just removing it from the config.chunks
collection?

Post by Derek Chen-Becker
Just as a follow up, I manually merged/removed about 1400 chunks last
night and the rebalance is complete now. Disk usage dropped by the expected
amount across our shards and things are much happier :)

Post by Gianfranco
Hi Otis,
If you want to keep certain documents for a certain amount of time
depending on a Date attribute, you can use a TTL Collection.
This collection uses a special index and it automatically deletes
"expired" documents.
http://docs.mongodb.org/manual/tutorial/expire-data/
Regards,
Gianfranco

Bump
------------------------------
*Sent:* Friday, October 12, 2012 2:11 PM
*Subject:* Re: [mongodb-user] Re: Manually removing empty chunks?
Could you elaborate why it's not recommended to remove empty chunks.
We are almost exactly in the same situation, our shard key is based
partially on timestamp. In our case, we delete old data and have no need
to archive it. With our shard key and data, we will never see data hit an
empty chunk (ie, one which we deleted data from).
------------------------------
*Sent:* Thursday, October 11, 2012 6:04 AM
*Subject:* [mongodb-user] Re: Manually removing empty chunks?
In general it's no recommended to delete chunks even if they are empty.
As you mentioned there is SERVER ticket on Jira to implement this in a
secure manner but it's not being planned yet.
You can of course vote to increase the chances of being developed first.
Though my suggestion would be to move the data you want to archive
into another database or cluster.
When adding more space the empty chunks will be reused.
Gianfranco
I've looked at https://jira.mongodb.org/ browse/SERVER-2487
<https://jira.mongodb.org/browse/SERVER-2487> and we're in a similar
situation. Part of our sharding key is a timestamp which works fine for use
because the rest of the key makes write balance out across the cluster.
However, we're at a point now where we need to archive off older data, and
our shards are starting to get out of balance in terms of disk usage
because while chunks counts are balanced, some of those chunks are empty
and will never again hold data. I understand why it would be difficult to
automate this, but I'm wondering if there's any harm in removing chunks
manually.
Essentially, can I stop balancing on the cluster, delete some of the
empty chunks from the config DB, then restart balancing? Would I need to
restart mongos on the servers as well? Is this a fantastically dangerous
thing to attempt?
Thanks,
Derek
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group, send email to
See also the IRC channel -- freenode.net#mongodb
--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
To unsubscribe from this group, send email to
See also the IRC channel -- freenode.net#mongodb
--

You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
<javascript:>
To unsubscribe from this group, send email to
See also the IRC channel -- freenode.net#mongodb

--
You received this message because you are subscribed to the Google
Groups "mongodb-user" group.
<javascript:>
To unsubscribe from this group, send email to
See also the IRC channel -- freenode.net#mongodb

--
You received this message because you are subscribed to the Google Groups "mongodb-user"
group.

For other MongoDB technical support options, see: https://docs.mongodb.com/manual/support/
---
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mongodb-user+***@googlegroups.com.
To post to this group, send email to mongodb-***@googlegroups.com.
Visit this group at https://groups.google.com/group/mongodb-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/mongodb-user/1c2f7b65-afcb-4450-a487-ed8acf2cd436%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.