Discussion:
Mongo sharding deadlock
(too old to reply)
Prasanna .S
2012-05-08 06:22:11 UTC
Permalink
Hey guys,

We sharded mongo as per the docs. The issue is the balancer is not being
very clever and not throttling at all. Now we are stuck with a half-sharded
setup and we can't turn on the balancer because it'd bring the site down.
All read queries timeout when the disk write rate is 100%.

Is there a way out of this mess at all? From a high-level, mongo could have
handled this by any of the following means:

1. Replicate from the slaves. Mysql does this.
2. Throttle and replicate only as long as the disk util is < 50% or so.
3. Have the moveChunk interruptible. If reads and writes come in-between,
consider them as higher priority, so that a read/write is never waiting for
more than a second.

Have you guys heard of this problem before? Because this really make
sharding impossible. I'm not sure what others are doing in production to
work around this.

Thanks,
-Prasanna
Co-Founder & CTO, lal.com

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Nat
2012-05-08 06:24:22 UTC
Permalink
I don't think it goes into deadlock. Can you try disabling the balancer and see whether the disk write rate going down or not.
-----Original Message-----
From: "Prasanna .S" <myprasanna-***@public.gmane.org>
Sender: mongodb-user-/***@public.gmane.org
Date: Mon, 7 May 2012 23:22:11
To: <mongodb-user-/***@public.gmane.org>
Reply-To: mongodb-user-/***@public.gmane.org
Cc: Surendran Mahendran<surenspost-***@public.gmane.org>; Hrushikesh Tilak<hrushikesh.tilak-***@public.gmane.org>
Subject: [mongodb-user] Mongo sharding deadlock

Hey guys,

We sharded mongo as per the docs. The issue is the balancer is not being
very clever and not throttling at all. Now we are stuck with a half-sharded
setup and we can't turn on the balancer because it'd bring the site down.
All read queries timeout when the disk write rate is 100%.

Is there a way out of this mess at all? From a high-level, mongo could have
handled this by any of the following means:

1. Replicate from the slaves. Mysql does this.
2. Throttle and replicate only as long as the disk util is < 50% or so.
3. Have the moveChunk interruptible. If reads and writes come in-between,
consider them as higher priority, so that a read/write is never waiting for
more than a second.

Have you guys heard of this problem before? Because this really make
sharding impossible. I'm not sure what others are doing in production to
work around this.

Thanks,
-Prasanna
Co-Founder & CTO, lal.com

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.


--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna
2012-05-08 08:46:12 UTC
Permalink
And yes, disabling the balancer, brings the write rate back to zero.
Although after 4 minutes!

On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
>
> I don't think it goes into deadlock. Can you try disabling the balancer
> and see whether the disk write rate going down or not.
> ------------------------------
> *From: * "Prasanna .S" <myprasanna-***@public.gmane.org>
> *Sender: * mongodb-user-/***@public.gmane.org
> *Date: *Mon, 7 May 2012 23:22:11 -0700
> *To: *<mongodb-user-/***@public.gmane.org>
> *ReplyTo: * mongodb-user-/***@public.gmane.org
> *Cc: *Surendran Mahendran<surenspost-***@public.gmane.org>; Hrushikesh Tilak<
> hrushikesh.tilak-***@public.gmane.org>
> *Subject: *[mongodb-user] Mongo sharding deadlock
>
> Hey guys,
>
> We sharded mongo as per the docs. The issue is the balancer is not being
> very clever and not throttling at all. Now we are stuck with a half-sharded
> setup and we can't turn on the balancer because it'd bring the site down.
> All read queries timeout when the disk write rate is 100%.
>
> Is there a way out of this mess at all? From a high-level, mongo could
> have handled this by any of the following means:
>
> 1. Replicate from the slaves. Mysql does this.
> 2. Throttle and replicate only as long as the disk util is < 50% or so.
> 3. Have the moveChunk interruptible. If reads and writes come in-between,
> consider them as higher priority, so that a read/write is never waiting for
> more than a second.
>
> Have you guys heard of this problem before? Because this really make
> sharding impossible. I'm not sure what others are doing in production to
> work around this.
>
> Thanks,
> -Prasanna
> Co-Founder & CTO, lal.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To view this discussion on the web visit https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Eliot Horowitz
2012-05-08 08:49:02 UTC
Permalink
What version of mongo and what kind of ebs raid?

On Tue, May 8, 2012 at 4:46 AM, Prasanna <myprasanna-***@public.gmane.org> wrote:
> And yes, disabling the balancer, brings the write rate back to zero.
> Although after 4 minutes!
>
>
> On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
>>
>> I don't think it goes into deadlock. Can you try disabling the balancer
>> and see whether the disk write rate going down or not.
>> ________________________________
>> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>> Sender: mongodb-user-/***@public.gmane.org
>> Date: Mon, 7 May 2012 23:22:11 -0700
>> To: <mongodb-user-/***@public.gmane.org>
>> ReplyTo: mongodb-user-/***@public.gmane.org
>> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>; Hrushikesh
>> Tilak<hrushikesh.tilak-***@public.gmane.org>
>> Subject: [mongodb-user] Mongo sharding deadlock
>>
>> Hey guys,
>>
>> We sharded mongo as per the docs. The issue is the balancer is not being
>> very clever and not throttling at all. Now we are stuck with a half-sharded
>> setup and we can't turn on the balancer because it'd bring the site down.
>> All read queries timeout when the disk write rate is 100%.
>>
>> Is there a way out of this mess at all? From a high-level, mongo could
>> have handled this by any of the following means:
>>
>> 1. Replicate from the slaves. Mysql does this.
>> 2. Throttle and replicate only as long as the disk util is < 50% or so.
>> 3. Have the moveChunk interruptible. If reads and writes come in-between,
>> consider them as higher priority, so that a read/write is never waiting for
>> more than a second.
>>
>> Have you guys heard of this problem before? Because this really make
>> sharding impossible. I'm not sure what others are doing in production to
>> work around this.
>>
>> Thanks,
>> -Prasanna
>> Co-Founder & CTO, lal.com
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-08 09:03:24 UTC
Permalink
Raid 10 on 8 disks.

MongoDB shell version: 2.0.4
Tue May 8 09:02:29 [initandlisten] db version v2.0.4, pdfile version 4.5


On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:

> What version of mongo and what kind of ebs raid?
>
> On Tue, May 8, 2012 at 4:46 AM, Prasanna <myprasanna-***@public.gmane.org> wrote:
> > And yes, disabling the balancer, brings the write rate back to zero.
> > Although after 4 minutes!
> >
> >
> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
> >>
> >> I don't think it goes into deadlock. Can you try disabling the balancer
> >> and see whether the disk write rate going down or not.
> >> ________________________________
> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
> >> Sender: mongodb-user-/***@public.gmane.org
> >> Date: Mon, 7 May 2012 23:22:11 -0700
> >> To: <mongodb-user-/***@public.gmane.org>
> >> ReplyTo: mongodb-user-/***@public.gmane.org
> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>; Hrushikesh
> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
> >> Subject: [mongodb-user] Mongo sharding deadlock
> >>
> >> Hey guys,
> >>
> >> We sharded mongo as per the docs. The issue is the balancer is not being
> >> very clever and not throttling at all. Now we are stuck with a
> half-sharded
> >> setup and we can't turn on the balancer because it'd bring the site
> down.
> >> All read queries timeout when the disk write rate is 100%.
> >>
> >> Is there a way out of this mess at all? From a high-level, mongo could
> >> have handled this by any of the following means:
> >>
> >> 1. Replicate from the slaves. Mysql does this.
> >> 2. Throttle and replicate only as long as the disk util is < 50% or so.
> >> 3. Have the moveChunk interruptible. If reads and writes come
> in-between,
> >> consider them as higher priority, so that a read/write is never waiting
> for
> >> more than a second.
> >>
> >> Have you guys heard of this problem before? Because this really make
> >> sharding impossible. I'm not sure what others are doing in production to
> >> work around this.
> >>
> >> Thanks,
> >> -Prasanna
> >> Co-Founder & CTO, lal.com
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "mongodb-user" group.
> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> To unsubscribe from this group, send email to
> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> For more options, visit this group at
> >> http://groups.google.com/group/mongodb-user?hl=en.
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "mongodb-user" group.
> > To view this discussion on the web visit
> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
> >
> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > To unsubscribe from this group, send email to
> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > For more options, visit this group at
> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Eliot Horowitz
2012-05-08 11:49:33 UTC
Permalink
Are these machines in MMS?
Hard to see much in the pasted stats.



On Tue, May 8, 2012 at 5:03 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
> Raid 10 on 8 disks.
>
> MongoDB shell version: 2.0.4
> Tue May  8 09:02:29 [initandlisten] db version v2.0.4, pdfile version 4.5
>
>
> On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:
>>
>> What version of mongo and what kind of ebs raid?
>>
>> On Tue, May 8, 2012 at 4:46 AM, Prasanna <myprasanna-***@public.gmane.org> wrote:
>> > And yes, disabling the balancer, brings the write rate back to zero.
>> > Although after 4 minutes!
>> >
>> >
>> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
>> >>
>> >> I don't think it goes into deadlock. Can you try disabling the balancer
>> >> and see whether the disk write rate going down or not.
>> >> ________________________________
>> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>> >> Sender: mongodb-user-/***@public.gmane.org
>> >> Date: Mon, 7 May 2012 23:22:11 -0700
>> >> To: <mongodb-user-/***@public.gmane.org>
>> >> ReplyTo: mongodb-user-/***@public.gmane.org
>> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>; Hrushikesh
>> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>> >> Subject: [mongodb-user] Mongo sharding deadlock
>> >>
>> >> Hey guys,
>> >>
>> >> We sharded mongo as per the docs. The issue is the balancer is not
>> >> being
>> >> very clever and not throttling at all. Now we are stuck with a
>> >> half-sharded
>> >> setup and we can't turn on the balancer because it'd bring the site
>> >> down.
>> >> All read queries timeout when the disk write rate is 100%.
>> >>
>> >> Is there a way out of this mess at all? From a high-level, mongo could
>> >> have handled this by any of the following means:
>> >>
>> >> 1. Replicate from the slaves. Mysql does this.
>> >> 2. Throttle and replicate only as long as the disk util is < 50% or so.
>> >> 3. Have the moveChunk interruptible. If reads and writes come
>> >> in-between,
>> >> consider them as higher priority, so that a read/write is never waiting
>> >> for
>> >> more than a second.
>> >>
>> >> Have you guys heard of this problem before? Because this really make
>> >> sharding impossible. I'm not sure what others are doing in production
>> >> to
>> >> work around this.
>> >>
>> >> Thanks,
>> >> -Prasanna
>> >> Co-Founder & CTO, lal.com
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> To unsubscribe from this group, send email to
>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To view this discussion on the web visit
>> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >
>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > To unsubscribe from this group, send email to
>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-08 11:56:39 UTC
Permalink
Yes, indeed.

Username: arnold-***@public.gmane.org
Machine: mongo55

On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:

> Are these machines in MMS?
> Hard to see much in the pasted stats.
>
>
>
> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
> > Raid 10 on 8 disks.
> >
> > MongoDB shell version: 2.0.4
> > Tue May 8 09:02:29 [initandlisten] db version v2.0.4, pdfile version 4.5
> >
> >
> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:
> >>
> >> What version of mongo and what kind of ebs raid?
> >>
> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna <myprasanna-***@public.gmane.org> wrote:
> >> > And yes, disabling the balancer, brings the write rate back to zero.
> >> > Although after 4 minutes!
> >> >
> >> >
> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
> >> >>
> >> >> I don't think it goes into deadlock. Can you try disabling the
> balancer
> >> >> and see whether the disk write rate going down or not.
> >> >> ________________________________
> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
> >> >> Sender: mongodb-user-/***@public.gmane.org
> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
> >> >> To: <mongodb-user-/***@public.gmane.org>
> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>; Hrushikesh
> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
> >> >> Subject: [mongodb-user] Mongo sharding deadlock
> >> >>
> >> >> Hey guys,
> >> >>
> >> >> We sharded mongo as per the docs. The issue is the balancer is not
> >> >> being
> >> >> very clever and not throttling at all. Now we are stuck with a
> >> >> half-sharded
> >> >> setup and we can't turn on the balancer because it'd bring the site
> >> >> down.
> >> >> All read queries timeout when the disk write rate is 100%.
> >> >>
> >> >> Is there a way out of this mess at all? From a high-level, mongo
> could
> >> >> have handled this by any of the following means:
> >> >>
> >> >> 1. Replicate from the slaves. Mysql does this.
> >> >> 2. Throttle and replicate only as long as the disk util is < 50% or
> so.
> >> >> 3. Have the moveChunk interruptible. If reads and writes come
> >> >> in-between,
> >> >> consider them as higher priority, so that a read/write is never
> waiting
> >> >> for
> >> >> more than a second.
> >> >>
> >> >> Have you guys heard of this problem before? Because this really make
> >> >> sharding impossible. I'm not sure what others are doing in production
> >> >> to
> >> >> work around this.
> >> >>
> >> >> Thanks,
> >> >> -Prasanna
> >> >> Co-Founder & CTO, lal.com
> >> >>
> >> >> --
> >> >> You received this message because you are subscribed to the Google
> >> >> Groups
> >> >> "mongodb-user" group.
> >> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> To unsubscribe from this group, send email to
> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> For more options, visit this group at
> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups
> >> > "mongodb-user" group.
> >> > To view this discussion on the web visit
> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
> >> >
> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> > To unsubscribe from this group, send email to
> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> > For more options, visit this group at
> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "mongodb-user" group.
> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> To unsubscribe from this group, send email to
> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> For more options, visit this group at
> >> http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > To unsubscribe from this group, send email to
> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > For more options, visit this group at
> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Dan Pasette
2012-05-08 14:02:31 UTC
Permalink
Looks like the docs in that collection (lal_production.facebook_names) are
small and likely distributed throughout the disk.
Can you check your disk readahead setting?
Please run and post the results of blockev --report.

On Tue, May 8, 2012 at 7:56 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:

> Yes, indeed.
>
> Username: arnold-***@public.gmane.org
> Machine: mongo55
>
>
> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:
>
>> Are these machines in MMS?
>> Hard to see much in the pasted stats.
>>
>>
>>
>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>> > Raid 10 on 8 disks.
>> >
>> > MongoDB shell version: 2.0.4
>> > Tue May 8 09:02:29 [initandlisten] db version v2.0.4, pdfile version
>> 4.5
>> >
>> >
>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:
>> >>
>> >> What version of mongo and what kind of ebs raid?
>> >>
>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna <myprasanna-***@public.gmane.org> wrote:
>> >> > And yes, disabling the balancer, brings the write rate back to zero.
>> >> > Although after 4 minutes!
>> >> >
>> >> >
>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
>> >> >>
>> >> >> I don't think it goes into deadlock. Can you try disabling the
>> balancer
>> >> >> and see whether the disk write rate going down or not.
>> >> >> ________________________________
>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>> >> >> Sender: mongodb-user-/***@public.gmane.org
>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>> >> >> To: <mongodb-user-/***@public.gmane.org>
>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>; Hrushikesh
>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>> >> >>
>> >> >> Hey guys,
>> >> >>
>> >> >> We sharded mongo as per the docs. The issue is the balancer is not
>> >> >> being
>> >> >> very clever and not throttling at all. Now we are stuck with a
>> >> >> half-sharded
>> >> >> setup and we can't turn on the balancer because it'd bring the site
>> >> >> down.
>> >> >> All read queries timeout when the disk write rate is 100%.
>> >> >>
>> >> >> Is there a way out of this mess at all? From a high-level, mongo
>> could
>> >> >> have handled this by any of the following means:
>> >> >>
>> >> >> 1. Replicate from the slaves. Mysql does this.
>> >> >> 2. Throttle and replicate only as long as the disk util is < 50% or
>> so.
>> >> >> 3. Have the moveChunk interruptible. If reads and writes come
>> >> >> in-between,
>> >> >> consider them as higher priority, so that a read/write is never
>> waiting
>> >> >> for
>> >> >> more than a second.
>> >> >>
>> >> >> Have you guys heard of this problem before? Because this really make
>> >> >> sharding impossible. I'm not sure what others are doing in
>> production
>> >> >> to
>> >> >> work around this.
>> >> >>
>> >> >> Thanks,
>> >> >> -Prasanna
>> >> >> Co-Founder & CTO, lal.com
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "mongodb-user" group.
>> >> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> To unsubscribe from this group, send email to
>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To view this discussion on the web visit
>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >
>> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > To unsubscribe from this group, send email to
>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> To unsubscribe from this group, send email to
>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > To unsubscribe from this group, send email to
>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-08 18:49:26 UTC
Permalink
Hey Dan,

Read ahead settings are the defaults (256 MB): http://pastebin.com/MHc8s2fq
Let me know if you need something else.

@Elliot,

I spoke with some of my friends at *foursquare* and it turns out they face
the same issue as well. I've had my other friends who run companies > $100M
market cap (whom I'd rather not name in this list), tell me about their
mongo problems in the past few days. This is pretty serious stuff and will
affect if someone wants to choose mongo as their primary store or not. To
not have sharding is one thing, but to promise it and being stuck in
production is another.

I really thank you guys and appreciate the responsiveness so far. Happy to
work together and hope we can resolve this.

Thanks so much,
-Prasanna

On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <dan-***@public.gmane.org> wrote:

> Looks like the docs in that collection (lal_production.facebook_names) are
> small and likely distributed throughout the disk.
> Can you check your disk readahead setting?
> Please run and post the results of blockev --report.
>
>
> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>
>> Yes, indeed.
>>
>> Username: arnold-***@public.gmane.org
>> Machine: mongo55
>>
>>
>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:
>>
>>> Are these machines in MMS?
>>> Hard to see much in the pasted stats.
>>>
>>>
>>>
>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>>> wrote:
>>> > Raid 10 on 8 disks.
>>> >
>>> > MongoDB shell version: 2.0.4
>>> > Tue May 8 09:02:29 [initandlisten] db version v2.0.4, pdfile version
>>> 4.5
>>> >
>>> >
>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz <eliot-***@public.gmane.org>
>>> wrote:
>>> >>
>>> >> What version of mongo and what kind of ebs raid?
>>> >>
>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna <myprasanna-***@public.gmane.org>
>>> wrote:
>>> >> > And yes, disabling the balancer, brings the write rate back to zero.
>>> >> > Although after 4 minutes!
>>> >> >
>>> >> >
>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
>>> >> >>
>>> >> >> I don't think it goes into deadlock. Can you try disabling the
>>> balancer
>>> >> >> and see whether the disk write rate going down or not.
>>> >> >> ________________________________
>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>; Hrushikesh
>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>>> >> >>
>>> >> >> Hey guys,
>>> >> >>
>>> >> >> We sharded mongo as per the docs. The issue is the balancer is not
>>> >> >> being
>>> >> >> very clever and not throttling at all. Now we are stuck with a
>>> >> >> half-sharded
>>> >> >> setup and we can't turn on the balancer because it'd bring the site
>>> >> >> down.
>>> >> >> All read queries timeout when the disk write rate is 100%.
>>> >> >>
>>> >> >> Is there a way out of this mess at all? From a high-level, mongo
>>> could
>>> >> >> have handled this by any of the following means:
>>> >> >>
>>> >> >> 1. Replicate from the slaves. Mysql does this.
>>> >> >> 2. Throttle and replicate only as long as the disk util is < 50%
>>> or so.
>>> >> >> 3. Have the moveChunk interruptible. If reads and writes come
>>> >> >> in-between,
>>> >> >> consider them as higher priority, so that a read/write is never
>>> waiting
>>> >> >> for
>>> >> >> more than a second.
>>> >> >>
>>> >> >> Have you guys heard of this problem before? Because this really
>>> make
>>> >> >> sharding impossible. I'm not sure what others are doing in
>>> production
>>> >> >> to
>>> >> >> work around this.
>>> >> >>
>>> >> >> Thanks,
>>> >> >> -Prasanna
>>> >> >> Co-Founder & CTO, lal.com
>>> >> >>
>>> >> >> --
>>> >> >> You received this message because you are subscribed to the Google
>>> >> >> Groups
>>> >> >> "mongodb-user" group.
>>> >> >> To post to this group, send email to mongodb-user-/***@public.gmane.org
>>> .
>>> >> >> To unsubscribe from this group, send email to
>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >> >> For more options, visit this group at
>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>>> >> >
>>> >> > --
>>> >> > You received this message because you are subscribed to the Google
>>> >> > Groups
>>> >> > "mongodb-user" group.
>>> >> > To view this discussion on the web visit
>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>>> >> >
>>> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >> > To unsubscribe from this group, send email to
>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >> > For more options, visit this group at
>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>>> >>
>>> >> --
>>> >> You received this message because you are subscribed to the Google
>>> Groups
>>> >> "mongodb-user" group.
>>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >> To unsubscribe from this group, send email to
>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >> For more options, visit this group at
>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>>> >>
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> Groups
>>> > "mongodb-user" group.
>>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> > To unsubscribe from this group, send email to
>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> > For more options, visit this group at
>>> > http://groups.google.com/group/mongodb-user?hl=en.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "mongodb-user" group.
>>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> To unsubscribe from this group, send email to
>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> For more options, visit this group at
>>> http://groups.google.com/group/mongodb-user?hl=en.
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Scott Hernandez
2012-05-09 22:41:33 UTC
Permalink
Those values are in blocks (512 bytes), not MB.

We have an update in the upcoming 2.0.6 stable release which will help
with migrations a bit. https://jira.mongodb.org/browse/SERVER-5303 You
can click on the version like to see other important fixes.

Can you do a few migrates tonight so we can see the stats/behavior in
MMS? Please let us know when you do this so we can take a look at the
data to see if this fix will help.


On Tue, May 8, 2012 at 11:49 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
> Hey Dan,
>
> Read ahead settings are the defaults (256 MB): http://pastebin.com/MHc8s2fq
> Let me know if you need something else.
>
> @Elliot,
>
> I spoke with some of my friends at foursquare and it turns out they face the
> same issue as well. I've had my other friends who run companies > $100M
> market cap (whom I'd rather not name in this list), tell me about their
> mongo problems in the past few days. This is pretty serious stuff and will
> affect if someone wants to choose mongo as their primary store or not. To
> not have sharding is one thing, but to promise it and being stuck in
> production is another.
>
> I really thank you guys and appreciate the responsiveness so far. Happy to
> work together and hope we can resolve this.
>
> Thanks so much,
> -Prasanna
>
>
> On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <dan-***@public.gmane.org> wrote:
>>
>> Looks like the docs in that collection (lal_production.facebook_names) are
>> small and likely distributed throughout the disk.
>> Can you check your disk readahead setting?
>> Please run and post the results of blockev --report.
>>
>>
>> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>>>
>>> Yes, indeed.
>>>
>>> Username: arnold-***@public.gmane.org
>>> Machine: mongo55
>>>
>>>
>>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:
>>>>
>>>> Are these machines in MMS?
>>>> Hard to see much in the pasted stats.
>>>>
>>>>
>>>>
>>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>>>> wrote:
>>>> > Raid 10 on 8 disks.
>>>> >
>>>> > MongoDB shell version: 2.0.4
>>>> > Tue May  8 09:02:29 [initandlisten] db version v2.0.4, pdfile version
>>>> > 4.5
>>>> >
>>>> >
>>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz <eliot-***@public.gmane.org>
>>>> > wrote:
>>>> >>
>>>> >> What version of mongo and what kind of ebs raid?
>>>> >>
>>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna <myprasanna-***@public.gmane.org>
>>>> >> wrote:
>>>> >> > And yes, disabling the balancer, brings the write rate back to
>>>> >> > zero.
>>>> >> > Although after 4 minutes!
>>>> >> >
>>>> >> >
>>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
>>>> >> >>
>>>> >> >> I don't think it goes into deadlock. Can you try disabling the
>>>> >> >> balancer
>>>> >> >> and see whether the disk write rate going down or not.
>>>> >> >> ________________________________
>>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>; Hrushikesh
>>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>>>> >> >>
>>>> >> >> Hey guys,
>>>> >> >>
>>>> >> >> We sharded mongo as per the docs. The issue is the balancer is not
>>>> >> >> being
>>>> >> >> very clever and not throttling at all. Now we are stuck with a
>>>> >> >> half-sharded
>>>> >> >> setup and we can't turn on the balancer because it'd bring the
>>>> >> >> site
>>>> >> >> down.
>>>> >> >> All read queries timeout when the disk write rate is 100%.
>>>> >> >>
>>>> >> >> Is there a way out of this mess at all? From a high-level, mongo
>>>> >> >> could
>>>> >> >> have handled this by any of the following means:
>>>> >> >>
>>>> >> >> 1. Replicate from the slaves. Mysql does this.
>>>> >> >> 2. Throttle and replicate only as long as the disk util is < 50%
>>>> >> >> or so.
>>>> >> >> 3. Have the moveChunk interruptible. If reads and writes come
>>>> >> >> in-between,
>>>> >> >> consider them as higher priority, so that a read/write is never
>>>> >> >> waiting
>>>> >> >> for
>>>> >> >> more than a second.
>>>> >> >>
>>>> >> >> Have you guys heard of this problem before? Because this really
>>>> >> >> make
>>>> >> >> sharding impossible. I'm not sure what others are doing in
>>>> >> >> production
>>>> >> >> to
>>>> >> >> work around this.
>>>> >> >>
>>>> >> >> Thanks,
>>>> >> >> -Prasanna
>>>> >> >> Co-Founder & CTO, lal.com
>>>> >> >>
>>>> >> >> --
>>>> >> >> You received this message because you are subscribed to the Google
>>>> >> >> Groups
>>>> >> >> "mongodb-user" group.
>>>> >> >> To post to this group, send email to
>>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >> >> To unsubscribe from this group, send email to
>>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >> >> For more options, visit this group at
>>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>>>> >> >
>>>> >> > --
>>>> >> > You received this message because you are subscribed to the Google
>>>> >> > Groups
>>>> >> > "mongodb-user" group.
>>>> >> > To view this discussion on the web visit
>>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>>>> >> >
>>>> >> > To post to this group, send email to mongodb-user-/***@public.gmane.orgm.
>>>> >> > To unsubscribe from this group, send email to
>>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >> > For more options, visit this group at
>>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>>>> >>
>>>> >> --
>>>> >> You received this message because you are subscribed to the Google
>>>> >> Groups
>>>> >> "mongodb-user" group.
>>>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >> To unsubscribe from this group, send email to
>>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >> For more options, visit this group at
>>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>>>> >>
>>>> >
>>>> > --
>>>> > You received this message because you are subscribed to the Google
>>>> > Groups
>>>> > "mongodb-user" group.
>>>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> > To unsubscribe from this group, send email to
>>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> > For more options, visit this group at
>>>> > http://groups.google.com/group/mongodb-user?hl=en.
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "mongodb-user" group.
>>>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> To unsubscribe from this group, send email to
>>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/mongodb-user?hl=en.
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "mongodb-user" group.
>>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> To unsubscribe from this group, send email to
>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> For more options, visit this group at
>>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-09 23:30:36 UTC
Permalink
Hey Scott,

I'm going to do this now. Will start it and stop it quickly and give you
the exact times.

Thanks,
-Prasanna

On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez <scotthernandez-***@public.gmane.org>wrote:

> Those values are in blocks (512 bytes), not MB.
>
> We have an update in the upcoming 2.0.6 stable release which will help
> with migrations a bit. https://jira.mongodb.org/browse/SERVER-5303 You
> can click on the version like to see other important fixes.
>
> Can you do a few migrates tonight so we can see the stats/behavior in
> MMS? Please let us know when you do this so we can take a look at the
> data to see if this fix will help.
>
>
> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
> > Hey Dan,
> >
> > Read ahead settings are the defaults (256 MB):
> http://pastebin.com/MHc8s2fq
> > Let me know if you need something else.
> >
> > @Elliot,
> >
> > I spoke with some of my friends at foursquare and it turns out they face
> the
> > same issue as well. I've had my other friends who run companies > $100M
> > market cap (whom I'd rather not name in this list), tell me about their
> > mongo problems in the past few days. This is pretty serious stuff and
> will
> > affect if someone wants to choose mongo as their primary store or not. To
> > not have sharding is one thing, but to promise it and being stuck in
> > production is another.
> >
> > I really thank you guys and appreciate the responsiveness so far. Happy
> to
> > work together and hope we can resolve this.
> >
> > Thanks so much,
> > -Prasanna
> >
> >
> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <dan-***@public.gmane.org> wrote:
> >>
> >> Looks like the docs in that collection (lal_production.facebook_names)
> are
> >> small and likely distributed throughout the disk.
> >> Can you check your disk readahead setting?
> >> Please run and post the results of blockev --report.
> >>
> >>
> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S <myprasanna-***@public.gmane.org>
> wrote:
> >>>
> >>> Yes, indeed.
> >>>
> >>> Username: arnold-***@public.gmane.org
> >>> Machine: mongo55
> >>>
> >>>
> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz <eliot-***@public.gmane.org>
> wrote:
> >>>>
> >>>> Are these machines in MMS?
> >>>> Hard to see much in the pasted stats.
> >>>>
> >>>>
> >>>>
> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S <myprasanna-***@public.gmane.org>
> >>>> wrote:
> >>>> > Raid 10 on 8 disks.
> >>>> >
> >>>> > MongoDB shell version: 2.0.4
> >>>> > Tue May 8 09:02:29 [initandlisten] db version v2.0.4, pdfile
> version
> >>>> > 4.5
> >>>> >
> >>>> >
> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz <eliot-***@public.gmane.org>
> >>>> > wrote:
> >>>> >>
> >>>> >> What version of mongo and what kind of ebs raid?
> >>>> >>
> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna <myprasanna-***@public.gmane.org>
> >>>> >> wrote:
> >>>> >> > And yes, disabling the balancer, brings the write rate back to
> >>>> >> > zero.
> >>>> >> > Although after 4 minutes!
> >>>> >> >
> >>>> >> >
> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
> >>>> >> >>
> >>>> >> >> I don't think it goes into deadlock. Can you try disabling the
> >>>> >> >> balancer
> >>>> >> >> and see whether the disk write rate going down or not.
> >>>> >> >> ________________________________
> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>; Hrushikesh
> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
> >>>> >> >>
> >>>> >> >> Hey guys,
> >>>> >> >>
> >>>> >> >> We sharded mongo as per the docs. The issue is the balancer is
> not
> >>>> >> >> being
> >>>> >> >> very clever and not throttling at all. Now we are stuck with a
> >>>> >> >> half-sharded
> >>>> >> >> setup and we can't turn on the balancer because it'd bring the
> >>>> >> >> site
> >>>> >> >> down.
> >>>> >> >> All read queries timeout when the disk write rate is 100%.
> >>>> >> >>
> >>>> >> >> Is there a way out of this mess at all? From a high-level, mongo
> >>>> >> >> could
> >>>> >> >> have handled this by any of the following means:
> >>>> >> >>
> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
> >>>> >> >> 2. Throttle and replicate only as long as the disk util is < 50%
> >>>> >> >> or so.
> >>>> >> >> 3. Have the moveChunk interruptible. If reads and writes come
> >>>> >> >> in-between,
> >>>> >> >> consider them as higher priority, so that a read/write is never
> >>>> >> >> waiting
> >>>> >> >> for
> >>>> >> >> more than a second.
> >>>> >> >>
> >>>> >> >> Have you guys heard of this problem before? Because this really
> >>>> >> >> make
> >>>> >> >> sharding impossible. I'm not sure what others are doing in
> >>>> >> >> production
> >>>> >> >> to
> >>>> >> >> work around this.
> >>>> >> >>
> >>>> >> >> Thanks,
> >>>> >> >> -Prasanna
> >>>> >> >> Co-Founder & CTO, lal.com
> >>>> >> >>
> >>>> >> >> --
> >>>> >> >> You received this message because you are subscribed to the
> Google
> >>>> >> >> Groups
> >>>> >> >> "mongodb-user" group.
> >>>> >> >> To post to this group, send email to
> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>>> >> >> To unsubscribe from this group, send email to
> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>>> >> >> For more options, visit this group at
> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >>>> >> >
> >>>> >> > --
> >>>> >> > You received this message because you are subscribed to the
> Google
> >>>> >> > Groups
> >>>> >> > "mongodb-user" group.
> >>>> >> > To view this discussion on the web visit
> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
> >>>> >> >
> >>>> >> > To post to this group, send email to
> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>>> >> > To unsubscribe from this group, send email to
> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>>> >> > For more options, visit this group at
> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >>>> >>
> >>>> >> --
> >>>> >> You received this message because you are subscribed to the Google
> >>>> >> Groups
> >>>> >> "mongodb-user" group.
> >>>> >> To post to this group, send email to mongodb-user-/***@public.gmane.org
> .
> >>>> >> To unsubscribe from this group, send email to
> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>>> >> For more options, visit this group at
> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
> >>>> >>
> >>>> >
> >>>> > --
> >>>> > You received this message because you are subscribed to the Google
> >>>> > Groups
> >>>> > "mongodb-user" group.
> >>>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>>> > To unsubscribe from this group, send email to
> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>>> > For more options, visit this group at
> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
> >>>>
> >>>> --
> >>>> You received this message because you are subscribed to the Google
> >>>> Groups "mongodb-user" group.
> >>>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>>> To unsubscribe from this group, send email to
> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>>> For more options, visit this group at
> >>>> http://groups.google.com/group/mongodb-user?hl=en.
> >>>>
> >>>
> >>> --
> >>> You received this message because you are subscribed to the Google
> Groups
> >>> "mongodb-user" group.
> >>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> To unsubscribe from this group, send email to
> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> For more options, visit this group at
> >>> http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "mongodb-user" group.
> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> To unsubscribe from this group, send email to
> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> For more options, visit this group at
> >> http://groups.google.com/group/mongodb-user?hl=en.
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > To unsubscribe from this group, send email to
> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > For more options, visit this group at
> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-09 23:37:40 UTC
Permalink
4:37 the balancer has started and disk shot up.

On Wed, May 9, 2012 at 4:30 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:

> Hey Scott,
>
> I'm going to do this now. Will start it and stop it quickly and give you
> the exact times.
>
> Thanks,
> -Prasanna
>
>
> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez <scotthernandez-***@public.gmane.org>wrote:
>
>> Those values are in blocks (512 bytes), not MB.
>>
>> We have an update in the upcoming 2.0.6 stable release which will help
>> with migrations a bit. https://jira.mongodb.org/browse/SERVER-5303 You
>> can click on the version like to see other important fixes.
>>
>> Can you do a few migrates tonight so we can see the stats/behavior in
>> MMS? Please let us know when you do this so we can take a look at the
>> data to see if this fix will help.
>>
>>
>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>> wrote:
>> > Hey Dan,
>> >
>> > Read ahead settings are the defaults (256 MB):
>> http://pastebin.com/MHc8s2fq
>> > Let me know if you need something else.
>> >
>> > @Elliot,
>> >
>> > I spoke with some of my friends at foursquare and it turns out they
>> face the
>> > same issue as well. I've had my other friends who run companies > $100M
>> > market cap (whom I'd rather not name in this list), tell me about their
>> > mongo problems in the past few days. This is pretty serious stuff and
>> will
>> > affect if someone wants to choose mongo as their primary store or not.
>> To
>> > not have sharding is one thing, but to promise it and being stuck in
>> > production is another.
>> >
>> > I really thank you guys and appreciate the responsiveness so far. Happy
>> to
>> > work together and hope we can resolve this.
>> >
>> > Thanks so much,
>> > -Prasanna
>> >
>> >
>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <dan-***@public.gmane.org> wrote:
>> >>
>> >> Looks like the docs in that collection (lal_production.facebook_names)
>> are
>> >> small and likely distributed throughout the disk.
>> >> Can you check your disk readahead setting?
>> >> Please run and post the results of blockev --report.
>> >>
>> >>
>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>> wrote:
>> >>>
>> >>> Yes, indeed.
>> >>>
>> >>> Username: arnold-***@public.gmane.org
>> >>> Machine: mongo55
>> >>>
>> >>>
>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz <eliot-***@public.gmane.org>
>> wrote:
>> >>>>
>> >>>> Are these machines in MMS?
>> >>>> Hard to see much in the pasted stats.
>> >>>>
>> >>>>
>> >>>>
>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >>>> wrote:
>> >>>> > Raid 10 on 8 disks.
>> >>>> >
>> >>>> > MongoDB shell version: 2.0.4
>> >>>> > Tue May 8 09:02:29 [initandlisten] db version v2.0.4, pdfile
>> version
>> >>>> > 4.5
>> >>>> >
>> >>>> >
>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz <eliot-***@public.gmane.org>
>> >>>> > wrote:
>> >>>> >>
>> >>>> >> What version of mongo and what kind of ebs raid?
>> >>>> >>
>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna <myprasanna-***@public.gmane.org>
>> >>>> >> wrote:
>> >>>> >> > And yes, disabling the balancer, brings the write rate back to
>> >>>> >> > zero.
>> >>>> >> > Although after 4 minutes!
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
>> >>>> >> >>
>> >>>> >> >> I don't think it goes into deadlock. Can you try disabling the
>> >>>> >> >> balancer
>> >>>> >> >> and see whether the disk write rate going down or not.
>> >>>> >> >> ________________________________
>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>; Hrushikesh
>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>> >>>> >> >>
>> >>>> >> >> Hey guys,
>> >>>> >> >>
>> >>>> >> >> We sharded mongo as per the docs. The issue is the balancer is
>> not
>> >>>> >> >> being
>> >>>> >> >> very clever and not throttling at all. Now we are stuck with a
>> >>>> >> >> half-sharded
>> >>>> >> >> setup and we can't turn on the balancer because it'd bring the
>> >>>> >> >> site
>> >>>> >> >> down.
>> >>>> >> >> All read queries timeout when the disk write rate is 100%.
>> >>>> >> >>
>> >>>> >> >> Is there a way out of this mess at all? From a high-level,
>> mongo
>> >>>> >> >> could
>> >>>> >> >> have handled this by any of the following means:
>> >>>> >> >>
>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
>> >>>> >> >> 2. Throttle and replicate only as long as the disk util is <
>> 50%
>> >>>> >> >> or so.
>> >>>> >> >> 3. Have the moveChunk interruptible. If reads and writes come
>> >>>> >> >> in-between,
>> >>>> >> >> consider them as higher priority, so that a read/write is never
>> >>>> >> >> waiting
>> >>>> >> >> for
>> >>>> >> >> more than a second.
>> >>>> >> >>
>> >>>> >> >> Have you guys heard of this problem before? Because this really
>> >>>> >> >> make
>> >>>> >> >> sharding impossible. I'm not sure what others are doing in
>> >>>> >> >> production
>> >>>> >> >> to
>> >>>> >> >> work around this.
>> >>>> >> >>
>> >>>> >> >> Thanks,
>> >>>> >> >> -Prasanna
>> >>>> >> >> Co-Founder & CTO, lal.com
>> >>>> >> >>
>> >>>> >> >> --
>> >>>> >> >> You received this message because you are subscribed to the
>> Google
>> >>>> >> >> Groups
>> >>>> >> >> "mongodb-user" group.
>> >>>> >> >> To post to this group, send email to
>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>>> >> >> To unsubscribe from this group, send email to
>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>>> >> >> For more options, visit this group at
>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>>> >> >
>> >>>> >> > --
>> >>>> >> > You received this message because you are subscribed to the
>> Google
>> >>>> >> > Groups
>> >>>> >> > "mongodb-user" group.
>> >>>> >> > To view this discussion on the web visit
>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >>>> >> >
>> >>>> >> > To post to this group, send email to
>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>>> >> > To unsubscribe from this group, send email to
>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>>> >> > For more options, visit this group at
>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>>> >>
>> >>>> >> --
>> >>>> >> You received this message because you are subscribed to the Google
>> >>>> >> Groups
>> >>>> >> "mongodb-user" group.
>> >>>> >> To post to this group, send email to
>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>>> >> To unsubscribe from this group, send email to
>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>>> >> For more options, visit this group at
>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>>> >>
>> >>>> >
>> >>>> > --
>> >>>> > You received this message because you are subscribed to the Google
>> >>>> > Groups
>> >>>> > "mongodb-user" group.
>> >>>> > To post to this group, send email to mongodb-user-/***@public.gmane.org
>> .
>> >>>> > To unsubscribe from this group, send email to
>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>>> > For more options, visit this group at
>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>>>
>> >>>> --
>> >>>> You received this message because you are subscribed to the Google
>> >>>> Groups "mongodb-user" group.
>> >>>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>>> To unsubscribe from this group, send email to
>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>>> For more options, visit this group at
>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >>>>
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to the Google
>> Groups
>> >>> "mongodb-user" group.
>> >>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> To unsubscribe from this group, send email to
>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> For more options, visit this group at
>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> To unsubscribe from this group, send email to
>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > To unsubscribe from this group, send email to
>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-09 23:46:55 UTC
Permalink
Balancer stopped at 4:45

On Wed, May 9, 2012 at 4:37 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:

> 4:37 the balancer has started and disk shot up.
>
>
> On Wed, May 9, 2012 at 4:30 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>
>> Hey Scott,
>>
>> I'm going to do this now. Will start it and stop it quickly and give you
>> the exact times.
>>
>> Thanks,
>> -Prasanna
>>
>>
>> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez <scotthernandez-***@public.gmane.org
>> > wrote:
>>
>>> Those values are in blocks (512 bytes), not MB.
>>>
>>> We have an update in the upcoming 2.0.6 stable release which will help
>>> with migrations a bit. https://jira.mongodb.org/browse/SERVER-5303 You
>>> can click on the version like to see other important fixes.
>>>
>>> Can you do a few migrates tonight so we can see the stats/behavior in
>>> MMS? Please let us know when you do this so we can take a look at the
>>> data to see if this fix will help.
>>>
>>>
>>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>>> wrote:
>>> > Hey Dan,
>>> >
>>> > Read ahead settings are the defaults (256 MB):
>>> http://pastebin.com/MHc8s2fq
>>> > Let me know if you need something else.
>>> >
>>> > @Elliot,
>>> >
>>> > I spoke with some of my friends at foursquare and it turns out they
>>> face the
>>> > same issue as well. I've had my other friends who run companies > $100M
>>> > market cap (whom I'd rather not name in this list), tell me about their
>>> > mongo problems in the past few days. This is pretty serious stuff and
>>> will
>>> > affect if someone wants to choose mongo as their primary store or not.
>>> To
>>> > not have sharding is one thing, but to promise it and being stuck in
>>> > production is another.
>>> >
>>> > I really thank you guys and appreciate the responsiveness so far.
>>> Happy to
>>> > work together and hope we can resolve this.
>>> >
>>> > Thanks so much,
>>> > -Prasanna
>>> >
>>> >
>>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <dan-***@public.gmane.org> wrote:
>>> >>
>>> >> Looks like the docs in that collection
>>> (lal_production.facebook_names) are
>>> >> small and likely distributed throughout the disk.
>>> >> Can you check your disk readahead setting?
>>> >> Please run and post the results of blockev --report.
>>> >>
>>> >>
>>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>>> wrote:
>>> >>>
>>> >>> Yes, indeed.
>>> >>>
>>> >>> Username: arnold-***@public.gmane.org
>>> >>> Machine: mongo55
>>> >>>
>>> >>>
>>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz <eliot-***@public.gmane.org>
>>> wrote:
>>> >>>>
>>> >>>> Are these machines in MMS?
>>> >>>> Hard to see much in the pasted stats.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>>> >>>> wrote:
>>> >>>> > Raid 10 on 8 disks.
>>> >>>> >
>>> >>>> > MongoDB shell version: 2.0.4
>>> >>>> > Tue May 8 09:02:29 [initandlisten] db version v2.0.4, pdfile
>>> version
>>> >>>> > 4.5
>>> >>>> >
>>> >>>> >
>>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz <eliot-***@public.gmane.org>
>>> >>>> > wrote:
>>> >>>> >>
>>> >>>> >> What version of mongo and what kind of ebs raid?
>>> >>>> >>
>>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna <myprasanna-***@public.gmane.org>
>>> >>>> >> wrote:
>>> >>>> >> > And yes, disabling the balancer, brings the write rate back to
>>> >>>> >> > zero.
>>> >>>> >> > Although after 4 minutes!
>>> >>>> >> >
>>> >>>> >> >
>>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
>>> >>>> >> >>
>>> >>>> >> >> I don't think it goes into deadlock. Can you try disabling the
>>> >>>> >> >> balancer
>>> >>>> >> >> and see whether the disk write rate going down or not.
>>> >>>> >> >> ________________________________
>>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>; Hrushikesh
>>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>>> >>>> >> >>
>>> >>>> >> >> Hey guys,
>>> >>>> >> >>
>>> >>>> >> >> We sharded mongo as per the docs. The issue is the balancer
>>> is not
>>> >>>> >> >> being
>>> >>>> >> >> very clever and not throttling at all. Now we are stuck with a
>>> >>>> >> >> half-sharded
>>> >>>> >> >> setup and we can't turn on the balancer because it'd bring the
>>> >>>> >> >> site
>>> >>>> >> >> down.
>>> >>>> >> >> All read queries timeout when the disk write rate is 100%.
>>> >>>> >> >>
>>> >>>> >> >> Is there a way out of this mess at all? From a high-level,
>>> mongo
>>> >>>> >> >> could
>>> >>>> >> >> have handled this by any of the following means:
>>> >>>> >> >>
>>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
>>> >>>> >> >> 2. Throttle and replicate only as long as the disk util is <
>>> 50%
>>> >>>> >> >> or so.
>>> >>>> >> >> 3. Have the moveChunk interruptible. If reads and writes come
>>> >>>> >> >> in-between,
>>> >>>> >> >> consider them as higher priority, so that a read/write is
>>> never
>>> >>>> >> >> waiting
>>> >>>> >> >> for
>>> >>>> >> >> more than a second.
>>> >>>> >> >>
>>> >>>> >> >> Have you guys heard of this problem before? Because this
>>> really
>>> >>>> >> >> make
>>> >>>> >> >> sharding impossible. I'm not sure what others are doing in
>>> >>>> >> >> production
>>> >>>> >> >> to
>>> >>>> >> >> work around this.
>>> >>>> >> >>
>>> >>>> >> >> Thanks,
>>> >>>> >> >> -Prasanna
>>> >>>> >> >> Co-Founder & CTO, lal.com
>>> >>>> >> >>
>>> >>>> >> >> --
>>> >>>> >> >> You received this message because you are subscribed to the
>>> Google
>>> >>>> >> >> Groups
>>> >>>> >> >> "mongodb-user" group.
>>> >>>> >> >> To post to this group, send email to
>>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> >> >> To unsubscribe from this group, send email to
>>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> >> >> For more options, visit this group at
>>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>>> >>>> >> >
>>> >>>> >> > --
>>> >>>> >> > You received this message because you are subscribed to the
>>> Google
>>> >>>> >> > Groups
>>> >>>> >> > "mongodb-user" group.
>>> >>>> >> > To view this discussion on the web visit
>>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>>> >>>> >> >
>>> >>>> >> > To post to this group, send email to
>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> >> > To unsubscribe from this group, send email to
>>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> >> > For more options, visit this group at
>>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>>> >>>> >>
>>> >>>> >> --
>>> >>>> >> You received this message because you are subscribed to the
>>> Google
>>> >>>> >> Groups
>>> >>>> >> "mongodb-user" group.
>>> >>>> >> To post to this group, send email to
>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> >> To unsubscribe from this group, send email to
>>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> >> For more options, visit this group at
>>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>>> >>>> >>
>>> >>>> >
>>> >>>> > --
>>> >>>> > You received this message because you are subscribed to the Google
>>> >>>> > Groups
>>> >>>> > "mongodb-user" group.
>>> >>>> > To post to this group, send email to
>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> > To unsubscribe from this group, send email to
>>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> > For more options, visit this group at
>>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>>> >>>>
>>> >>>> --
>>> >>>> You received this message because you are subscribed to the Google
>>> >>>> Groups "mongodb-user" group.
>>> >>>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> To unsubscribe from this group, send email to
>>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> For more options, visit this group at
>>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>>> >>>>
>>> >>>
>>> >>> --
>>> >>> You received this message because you are subscribed to the Google
>>> Groups
>>> >>> "mongodb-user" group.
>>> >>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>> To unsubscribe from this group, send email to
>>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>> For more options, visit this group at
>>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>>> >>
>>> >>
>>> >> --
>>> >> You received this message because you are subscribed to the Google
>>> Groups
>>> >> "mongodb-user" group.
>>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >> To unsubscribe from this group, send email to
>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >> For more options, visit this group at
>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>>> >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> Groups
>>> > "mongodb-user" group.
>>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> > To unsubscribe from this group, send email to
>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> > For more options, visit this group at
>>> > http://groups.google.com/group/mongodb-user?hl=en.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "mongodb-user" group.
>>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> To unsubscribe from this group, send email to
>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> For more options, visit this group at
>>> http://groups.google.com/group/mongodb-user?hl=en.
>>>
>>>
>>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-09 23:47:53 UTC
Permalink
Let me know if you need anything else..

On Wed, May 9, 2012 at 4:46 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:

> Balancer stopped at 4:45
>
>
> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>
>> 4:37 the balancer has started and disk shot up.
>>
>>
>> On Wed, May 9, 2012 at 4:30 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>>
>>> Hey Scott,
>>>
>>> I'm going to do this now. Will start it and stop it quickly and give you
>>> the exact times.
>>>
>>> Thanks,
>>> -Prasanna
>>>
>>>
>>> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez <
>>> scotthernandez-***@public.gmane.org> wrote:
>>>
>>>> Those values are in blocks (512 bytes), not MB.
>>>>
>>>> We have an update in the upcoming 2.0.6 stable release which will help
>>>> with migrations a bit. https://jira.mongodb.org/browse/SERVER-5303 You
>>>> can click on the version like to see other important fixes.
>>>>
>>>> Can you do a few migrates tonight so we can see the stats/behavior in
>>>> MMS? Please let us know when you do this so we can take a look at the
>>>> data to see if this fix will help.
>>>>
>>>>
>>>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>>>> wrote:
>>>> > Hey Dan,
>>>> >
>>>> > Read ahead settings are the defaults (256 MB):
>>>> http://pastebin.com/MHc8s2fq
>>>> > Let me know if you need something else.
>>>> >
>>>> > @Elliot,
>>>> >
>>>> > I spoke with some of my friends at foursquare and it turns out they
>>>> face the
>>>> > same issue as well. I've had my other friends who run companies >
>>>> $100M
>>>> > market cap (whom I'd rather not name in this list), tell me about
>>>> their
>>>> > mongo problems in the past few days. This is pretty serious stuff and
>>>> will
>>>> > affect if someone wants to choose mongo as their primary store or
>>>> not. To
>>>> > not have sharding is one thing, but to promise it and being stuck in
>>>> > production is another.
>>>> >
>>>> > I really thank you guys and appreciate the responsiveness so far.
>>>> Happy to
>>>> > work together and hope we can resolve this.
>>>> >
>>>> > Thanks so much,
>>>> > -Prasanna
>>>> >
>>>> >
>>>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <dan-***@public.gmane.org> wrote:
>>>> >>
>>>> >> Looks like the docs in that collection
>>>> (lal_production.facebook_names) are
>>>> >> small and likely distributed throughout the disk.
>>>> >> Can you check your disk readahead setting?
>>>> >> Please run and post the results of blockev --report.
>>>> >>
>>>> >>
>>>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>>>> wrote:
>>>> >>>
>>>> >>> Yes, indeed.
>>>> >>>
>>>> >>> Username: arnold-***@public.gmane.org
>>>> >>> Machine: mongo55
>>>> >>>
>>>> >>>
>>>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz <eliot-***@public.gmane.org>
>>>> wrote:
>>>> >>>>
>>>> >>>> Are these machines in MMS?
>>>> >>>> Hard to see much in the pasted stats.
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>>>> >>>> wrote:
>>>> >>>> > Raid 10 on 8 disks.
>>>> >>>> >
>>>> >>>> > MongoDB shell version: 2.0.4
>>>> >>>> > Tue May 8 09:02:29 [initandlisten] db version v2.0.4, pdfile
>>>> version
>>>> >>>> > 4.5
>>>> >>>> >
>>>> >>>> >
>>>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz <eliot-***@public.gmane.org>
>>>> >>>> > wrote:
>>>> >>>> >>
>>>> >>>> >> What version of mongo and what kind of ebs raid?
>>>> >>>> >>
>>>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna <myprasanna-***@public.gmane.org>
>>>> >>>> >> wrote:
>>>> >>>> >> > And yes, disabling the balancer, brings the write rate back to
>>>> >>>> >> > zero.
>>>> >>>> >> > Although after 4 minutes!
>>>> >>>> >> >
>>>> >>>> >> >
>>>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
>>>> >>>> >> >>
>>>> >>>> >> >> I don't think it goes into deadlock. Can you try disabling
>>>> the
>>>> >>>> >> >> balancer
>>>> >>>> >> >> and see whether the disk write rate going down or not.
>>>> >>>> >> >> ________________________________
>>>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>>>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>>>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>>>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>>>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>>>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>; Hrushikesh
>>>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>>>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>>>> >>>> >> >>
>>>> >>>> >> >> Hey guys,
>>>> >>>> >> >>
>>>> >>>> >> >> We sharded mongo as per the docs. The issue is the balancer
>>>> is not
>>>> >>>> >> >> being
>>>> >>>> >> >> very clever and not throttling at all. Now we are stuck with
>>>> a
>>>> >>>> >> >> half-sharded
>>>> >>>> >> >> setup and we can't turn on the balancer because it'd bring
>>>> the
>>>> >>>> >> >> site
>>>> >>>> >> >> down.
>>>> >>>> >> >> All read queries timeout when the disk write rate is 100%.
>>>> >>>> >> >>
>>>> >>>> >> >> Is there a way out of this mess at all? From a high-level,
>>>> mongo
>>>> >>>> >> >> could
>>>> >>>> >> >> have handled this by any of the following means:
>>>> >>>> >> >>
>>>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
>>>> >>>> >> >> 2. Throttle and replicate only as long as the disk util is <
>>>> 50%
>>>> >>>> >> >> or so.
>>>> >>>> >> >> 3. Have the moveChunk interruptible. If reads and writes come
>>>> >>>> >> >> in-between,
>>>> >>>> >> >> consider them as higher priority, so that a read/write is
>>>> never
>>>> >>>> >> >> waiting
>>>> >>>> >> >> for
>>>> >>>> >> >> more than a second.
>>>> >>>> >> >>
>>>> >>>> >> >> Have you guys heard of this problem before? Because this
>>>> really
>>>> >>>> >> >> make
>>>> >>>> >> >> sharding impossible. I'm not sure what others are doing in
>>>> >>>> >> >> production
>>>> >>>> >> >> to
>>>> >>>> >> >> work around this.
>>>> >>>> >> >>
>>>> >>>> >> >> Thanks,
>>>> >>>> >> >> -Prasanna
>>>> >>>> >> >> Co-Founder & CTO, lal.com
>>>> >>>> >> >>
>>>> >>>> >> >> --
>>>> >>>> >> >> You received this message because you are subscribed to the
>>>> Google
>>>> >>>> >> >> Groups
>>>> >>>> >> >> "mongodb-user" group.
>>>> >>>> >> >> To post to this group, send email to
>>>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >>>> >> >> To unsubscribe from this group, send email to
>>>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >>>> >> >> For more options, visit this group at
>>>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>>>> >>>> >> >
>>>> >>>> >> > --
>>>> >>>> >> > You received this message because you are subscribed to the
>>>> Google
>>>> >>>> >> > Groups
>>>> >>>> >> > "mongodb-user" group.
>>>> >>>> >> > To view this discussion on the web visit
>>>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>>>> >>>> >> >
>>>> >>>> >> > To post to this group, send email to
>>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >>>> >> > To unsubscribe from this group, send email to
>>>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >>>> >> > For more options, visit this group at
>>>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>>>> >>>> >>
>>>> >>>> >> --
>>>> >>>> >> You received this message because you are subscribed to the
>>>> Google
>>>> >>>> >> Groups
>>>> >>>> >> "mongodb-user" group.
>>>> >>>> >> To post to this group, send email to
>>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >>>> >> To unsubscribe from this group, send email to
>>>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >>>> >> For more options, visit this group at
>>>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>>>> >>>> >>
>>>> >>>> >
>>>> >>>> > --
>>>> >>>> > You received this message because you are subscribed to the
>>>> Google
>>>> >>>> > Groups
>>>> >>>> > "mongodb-user" group.
>>>> >>>> > To post to this group, send email to
>>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >>>> > To unsubscribe from this group, send email to
>>>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >>>> > For more options, visit this group at
>>>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>>>> >>>>
>>>> >>>> --
>>>> >>>> You received this message because you are subscribed to the Google
>>>> >>>> Groups "mongodb-user" group.
>>>> >>>> To post to this group, send email to mongodb-user-/***@public.gmane.org
>>>> .
>>>> >>>> To unsubscribe from this group, send email to
>>>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >>>> For more options, visit this group at
>>>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>>>> >>>>
>>>> >>>
>>>> >>> --
>>>> >>> You received this message because you are subscribed to the Google
>>>> Groups
>>>> >>> "mongodb-user" group.
>>>> >>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >>> To unsubscribe from this group, send email to
>>>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >>> For more options, visit this group at
>>>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>>>> >>
>>>> >>
>>>> >> --
>>>> >> You received this message because you are subscribed to the Google
>>>> Groups
>>>> >> "mongodb-user" group.
>>>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >> To unsubscribe from this group, send email to
>>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> >> For more options, visit this group at
>>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>>>> >
>>>> >
>>>> > --
>>>> > You received this message because you are subscribed to the Google
>>>> Groups
>>>> > "mongodb-user" group.
>>>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> > To unsubscribe from this group, send email to
>>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> > For more options, visit this group at
>>>> > http://groups.google.com/group/mongodb-user?hl=en.
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "mongodb-user" group.
>>>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> To unsubscribe from this group, send email to
>>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/mongodb-user?hl=en.
>>>>
>>>>
>>>
>>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Scott Hernandez
2012-05-10 00:17:39 UTC
Permalink
Okay, looking at the receiving shard mongo55, and donor shard mongo43,
it look like the fix in 2.0.6 (due out in the next two weeks of so)
will help you out a lot.

On Wed, May 9, 2012 at 4:37 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
> 4:37 the balancer has started and disk shot up.
>
>
> On Wed, May 9, 2012 at 4:30 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>>
>> Hey Scott,
>>
>> I'm going to do this now. Will start it and stop it quickly and give you
>> the exact times.
>>
>> Thanks,
>> -Prasanna
>>
>>
>> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez <scotthernandez-***@public.gmane.orgm>
>> wrote:
>>>
>>> Those values are in blocks (512 bytes), not MB.
>>>
>>> We have an update in the upcoming 2.0.6 stable release which will help
>>> with migrations a bit. https://jira.mongodb.org/browse/SERVER-5303 You
>>> can click on the version like to see other important fixes.
>>>
>>> Can you do a few migrates tonight so we can see the stats/behavior in
>>> MMS? Please let  us know when you do this so we can take a look at the
>>> data to see if this fix will help.
>>>
>>>
>>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>>> wrote:
>>> > Hey Dan,
>>> >
>>> > Read ahead settings are the defaults (256
>>> > MB): http://pastebin.com/MHc8s2fq
>>> > Let me know if you need something else.
>>> >
>>> > @Elliot,
>>> >
>>> > I spoke with some of my friends at foursquare and it turns out they
>>> > face the
>>> > same issue as well. I've had my other friends who run companies > $100M
>>> > market cap (whom I'd rather not name in this list), tell me about their
>>> > mongo problems in the past few days. This is pretty serious stuff and
>>> > will
>>> > affect if someone wants to choose mongo as their primary store or not.
>>> > To
>>> > not have sharding is one thing, but to promise it and being stuck in
>>> > production is another.
>>> >
>>> > I really thank you guys and appreciate the responsiveness so far. Happy
>>> > to
>>> > work together and hope we can resolve this.
>>> >
>>> > Thanks so much,
>>> > -Prasanna
>>> >
>>> >
>>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <dan-***@public.gmane.org> wrote:
>>> >>
>>> >> Looks like the docs in that collection (lal_production.facebook_names)
>>> >> are
>>> >> small and likely distributed throughout the disk.
>>> >> Can you check your disk readahead setting?
>>> >> Please run and post the results of blockev --report.
>>> >>
>>> >>
>>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>>> >> wrote:
>>> >>>
>>> >>> Yes, indeed.
>>> >>>
>>> >>> Username: arnold-***@public.gmane.org
>>> >>> Machine: mongo55
>>> >>>
>>> >>>
>>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz <eliot-***@public.gmane.org>
>>> >>> wrote:
>>> >>>>
>>> >>>> Are these machines in MMS?
>>> >>>> Hard to see much in the pasted stats.
>>> >>>>
>>> >>>>
>>> >>>>
>>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>>> >>>> wrote:
>>> >>>> > Raid 10 on 8 disks.
>>> >>>> >
>>> >>>> > MongoDB shell version: 2.0.4
>>> >>>> > Tue May  8 09:02:29 [initandlisten] db version v2.0.4, pdfile
>>> >>>> > version
>>> >>>> > 4.5
>>> >>>> >
>>> >>>> >
>>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz <eliot-***@public.gmane.org>
>>> >>>> > wrote:
>>> >>>> >>
>>> >>>> >> What version of mongo and what kind of ebs raid?
>>> >>>> >>
>>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna <myprasanna-***@public.gmane.org>
>>> >>>> >> wrote:
>>> >>>> >> > And yes, disabling the balancer, brings the write rate back to
>>> >>>> >> > zero.
>>> >>>> >> > Although after 4 minutes!
>>> >>>> >> >
>>> >>>> >> >
>>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
>>> >>>> >> >>
>>> >>>> >> >> I don't think it goes into deadlock. Can you try disabling the
>>> >>>> >> >> balancer
>>> >>>> >> >> and see whether the disk write rate going down or not.
>>> >>>> >> >> ________________________________
>>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>; Hrushikesh
>>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>>> >>>> >> >>
>>> >>>> >> >> Hey guys,
>>> >>>> >> >>
>>> >>>> >> >> We sharded mongo as per the docs. The issue is the balancer is
>>> >>>> >> >> not
>>> >>>> >> >> being
>>> >>>> >> >> very clever and not throttling at all. Now we are stuck with a
>>> >>>> >> >> half-sharded
>>> >>>> >> >> setup and we can't turn on the balancer because it'd bring the
>>> >>>> >> >> site
>>> >>>> >> >> down.
>>> >>>> >> >> All read queries timeout when the disk write rate is 100%.
>>> >>>> >> >>
>>> >>>> >> >> Is there a way out of this mess at all? From a high-level,
>>> >>>> >> >> mongo
>>> >>>> >> >> could
>>> >>>> >> >> have handled this by any of the following means:
>>> >>>> >> >>
>>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
>>> >>>> >> >> 2. Throttle and replicate only as long as the disk util is <
>>> >>>> >> >> 50%
>>> >>>> >> >> or so.
>>> >>>> >> >> 3. Have the moveChunk interruptible. If reads and writes come
>>> >>>> >> >> in-between,
>>> >>>> >> >> consider them as higher priority, so that a read/write is
>>> >>>> >> >> never
>>> >>>> >> >> waiting
>>> >>>> >> >> for
>>> >>>> >> >> more than a second.
>>> >>>> >> >>
>>> >>>> >> >> Have you guys heard of this problem before? Because this
>>> >>>> >> >> really
>>> >>>> >> >> make
>>> >>>> >> >> sharding impossible. I'm not sure what others are doing in
>>> >>>> >> >> production
>>> >>>> >> >> to
>>> >>>> >> >> work around this.
>>> >>>> >> >>
>>> >>>> >> >> Thanks,
>>> >>>> >> >> -Prasanna
>>> >>>> >> >> Co-Founder & CTO, lal.com
>>> >>>> >> >>
>>> >>>> >> >> --
>>> >>>> >> >> You received this message because you are subscribed to the
>>> >>>> >> >> Google
>>> >>>> >> >> Groups
>>> >>>> >> >> "mongodb-user" group.
>>> >>>> >> >> To post to this group, send email to
>>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> >> >> To unsubscribe from this group, send email to
>>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> >> >> For more options, visit this group at
>>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>>> >>>> >> >
>>> >>>> >> > --
>>> >>>> >> > You received this message because you are subscribed to the
>>> >>>> >> > Google
>>> >>>> >> > Groups
>>> >>>> >> > "mongodb-user" group.
>>> >>>> >> > To view this discussion on the web visit
>>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>>> >>>> >> >
>>> >>>> >> > To post to this group, send email to
>>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> >> > To unsubscribe from this group, send email to
>>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> >> > For more options, visit this group at
>>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>>> >>>> >>
>>> >>>> >> --
>>> >>>> >> You received this message because you are subscribed to the
>>> >>>> >> Google
>>> >>>> >> Groups
>>> >>>> >> "mongodb-user" group.
>>> >>>> >> To post to this group, send email to
>>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> >> To unsubscribe from this group, send email to
>>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> >> For more options, visit this group at
>>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>>> >>>> >>
>>> >>>> >
>>> >>>> > --
>>> >>>> > You received this message because you are subscribed to the Google
>>> >>>> > Groups
>>> >>>> > "mongodb-user" group.
>>> >>>> > To post to this group, send email to
>>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> > To unsubscribe from this group, send email to
>>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> > For more options, visit this group at
>>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>>> >>>>
>>> >>>> --
>>> >>>> You received this message because you are subscribed to the Google
>>> >>>> Groups "mongodb-user" group.
>>> >>>> To post to this group, send email to mongodb-user-/***@public.gmane.org.
>>> >>>> To unsubscribe from this group, send email to
>>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>>> For more options, visit this group at
>>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>>> >>>>
>>> >>>
>>> >>> --
>>> >>> You received this message because you are subscribed to the Google
>>> >>> Groups
>>> >>> "mongodb-user" group.
>>> >>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>> To unsubscribe from this group, send email to
>>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >>> For more options, visit this group at
>>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>>> >>
>>> >>
>>> >> --
>>> >> You received this message because you are subscribed to the Google
>>> >> Groups
>>> >> "mongodb-user" group.
>>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >> To unsubscribe from this group, send email to
>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> >> For more options, visit this group at
>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>>> >
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> > Groups
>>> > "mongodb-user" group.
>>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> > To unsubscribe from this group, send email to
>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> > For more options, visit this group at
>>> > http://groups.google.com/group/mongodb-user?hl=en.
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups
>>> "mongodb-user" group.
>>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> To unsubscribe from this group, send email to
>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>>> For more options, visit this group at
>>> http://groups.google.com/group/mongodb-user?hl=en.
>>>
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-10 00:37:31 UTC
Permalink
Scott,

I tried to understand what's going on there. But with a quick glance, I
assume, a lock was being held longer than necessary and was fixed. It's
unclear how this can help me. The only way to reliably make sure the site
keeps running while a moveChunk is executing, is if, you yield every second
within moveChunk and make sure all pending reads complete, before the next
moveChunk period starts.

Are you guys doing it, or planning to do it? Is there any other high-level
way to guarantee the reads can be answered within a couple of seconds?

Thanks

On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez <scotthernandez-***@public.gmane.org>wrote:

> Okay, looking at the receiving shard mongo55, and donor shard mongo43,
> it look like the fix in 2.0.6 (due out in the next two weeks of so)
> will help you out a lot.
>
> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
> > 4:37 the balancer has started and disk shot up.
> >
> >
> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S <myprasanna-***@public.gmane.org>
> wrote:
> >>
> >> Hey Scott,
> >>
> >> I'm going to do this now. Will start it and stop it quickly and give you
> >> the exact times.
> >>
> >> Thanks,
> >> -Prasanna
> >>
> >>
> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez <
> scotthernandez-***@public.gmane.org>
> >> wrote:
> >>>
> >>> Those values are in blocks (512 bytes), not MB.
> >>>
> >>> We have an update in the upcoming 2.0.6 stable release which will help
> >>> with migrations a bit. https://jira.mongodb.org/browse/SERVER-5303 You
> >>> can click on the version like to see other important fixes.
> >>>
> >>> Can you do a few migrates tonight so we can see the stats/behavior in
> >>> MMS? Please let us know when you do this so we can take a look at the
> >>> data to see if this fix will help.
> >>>
> >>>
> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S <myprasanna-***@public.gmane.org>
> >>> wrote:
> >>> > Hey Dan,
> >>> >
> >>> > Read ahead settings are the defaults (256
> >>> > MB): http://pastebin.com/MHc8s2fq
> >>> > Let me know if you need something else.
> >>> >
> >>> > @Elliot,
> >>> >
> >>> > I spoke with some of my friends at foursquare and it turns out they
> >>> > face the
> >>> > same issue as well. I've had my other friends who run companies >
> $100M
> >>> > market cap (whom I'd rather not name in this list), tell me about
> their
> >>> > mongo problems in the past few days. This is pretty serious stuff and
> >>> > will
> >>> > affect if someone wants to choose mongo as their primary store or
> not.
> >>> > To
> >>> > not have sharding is one thing, but to promise it and being stuck in
> >>> > production is another.
> >>> >
> >>> > I really thank you guys and appreciate the responsiveness so far.
> Happy
> >>> > to
> >>> > work together and hope we can resolve this.
> >>> >
> >>> > Thanks so much,
> >>> > -Prasanna
> >>> >
> >>> >
> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <dan-***@public.gmane.org> wrote:
> >>> >>
> >>> >> Looks like the docs in that collection
> (lal_production.facebook_names)
> >>> >> are
> >>> >> small and likely distributed throughout the disk.
> >>> >> Can you check your disk readahead setting?
> >>> >> Please run and post the results of blockev --report.
> >>> >>
> >>> >>
> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S <myprasanna-***@public.gmane.org>
> >>> >> wrote:
> >>> >>>
> >>> >>> Yes, indeed.
> >>> >>>
> >>> >>> Username: arnold-***@public.gmane.org
> >>> >>> Machine: mongo55
> >>> >>>
> >>> >>>
> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz <eliot-***@public.gmane.org>
> >>> >>> wrote:
> >>> >>>>
> >>> >>>> Are these machines in MMS?
> >>> >>>> Hard to see much in the pasted stats.
> >>> >>>>
> >>> >>>>
> >>> >>>>
> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S <myprasanna-***@public.gmane.org
> >
> >>> >>>> wrote:
> >>> >>>> > Raid 10 on 8 disks.
> >>> >>>> >
> >>> >>>> > MongoDB shell version: 2.0.4
> >>> >>>> > Tue May 8 09:02:29 [initandlisten] db version v2.0.4, pdfile
> >>> >>>> > version
> >>> >>>> > 4.5
> >>> >>>> >
> >>> >>>> >
> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz <eliot-***@public.gmane.org
> >
> >>> >>>> > wrote:
> >>> >>>> >>
> >>> >>>> >> What version of mongo and what kind of ebs raid?
> >>> >>>> >>
> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna <myprasanna-***@public.gmane.org
> >
> >>> >>>> >> wrote:
> >>> >>>> >> > And yes, disabling the balancer, brings the write rate back
> to
> >>> >>>> >> > zero.
> >>> >>>> >> > Although after 4 minutes!
> >>> >>>> >> >
> >>> >>>> >> >
> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
> >>> >>>> >> >>
> >>> >>>> >> >> I don't think it goes into deadlock. Can you try disabling
> the
> >>> >>>> >> >> balancer
> >>> >>>> >> >> and see whether the disk write rate going down or not.
> >>> >>>> >> >> ________________________________
> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>; Hrushikesh
> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
> >>> >>>> >> >>
> >>> >>>> >> >> Hey guys,
> >>> >>>> >> >>
> >>> >>>> >> >> We sharded mongo as per the docs. The issue is the balancer
> is
> >>> >>>> >> >> not
> >>> >>>> >> >> being
> >>> >>>> >> >> very clever and not throttling at all. Now we are stuck
> with a
> >>> >>>> >> >> half-sharded
> >>> >>>> >> >> setup and we can't turn on the balancer because it'd bring
> the
> >>> >>>> >> >> site
> >>> >>>> >> >> down.
> >>> >>>> >> >> All read queries timeout when the disk write rate is 100%.
> >>> >>>> >> >>
> >>> >>>> >> >> Is there a way out of this mess at all? From a high-level,
> >>> >>>> >> >> mongo
> >>> >>>> >> >> could
> >>> >>>> >> >> have handled this by any of the following means:
> >>> >>>> >> >>
> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
> >>> >>>> >> >> 2. Throttle and replicate only as long as the disk util is <
> >>> >>>> >> >> 50%
> >>> >>>> >> >> or so.
> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads and writes
> come
> >>> >>>> >> >> in-between,
> >>> >>>> >> >> consider them as higher priority, so that a read/write is
> >>> >>>> >> >> never
> >>> >>>> >> >> waiting
> >>> >>>> >> >> for
> >>> >>>> >> >> more than a second.
> >>> >>>> >> >>
> >>> >>>> >> >> Have you guys heard of this problem before? Because this
> >>> >>>> >> >> really
> >>> >>>> >> >> make
> >>> >>>> >> >> sharding impossible. I'm not sure what others are doing in
> >>> >>>> >> >> production
> >>> >>>> >> >> to
> >>> >>>> >> >> work around this.
> >>> >>>> >> >>
> >>> >>>> >> >> Thanks,
> >>> >>>> >> >> -Prasanna
> >>> >>>> >> >> Co-Founder & CTO, lal.com
> >>> >>>> >> >>
> >>> >>>> >> >> --
> >>> >>>> >> >> You received this message because you are subscribed to the
> >>> >>>> >> >> Google
> >>> >>>> >> >> Groups
> >>> >>>> >> >> "mongodb-user" group.
> >>> >>>> >> >> To post to this group, send email to
> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> >>>> >> >> To unsubscribe from this group, send email to
> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> >>>> >> >> For more options, visit this group at
> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >>> >>>> >> >
> >>> >>>> >> > --
> >>> >>>> >> > You received this message because you are subscribed to the
> >>> >>>> >> > Google
> >>> >>>> >> > Groups
> >>> >>>> >> > "mongodb-user" group.
> >>> >>>> >> > To view this discussion on the web visit
> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
> >>> >>>> >> >
> >>> >>>> >> > To post to this group, send email to
> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> >>>> >> > To unsubscribe from this group, send email to
> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> >>>> >> > For more options, visit this group at
> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >>> >>>> >>
> >>> >>>> >> --
> >>> >>>> >> You received this message because you are subscribed to the
> >>> >>>> >> Google
> >>> >>>> >> Groups
> >>> >>>> >> "mongodb-user" group.
> >>> >>>> >> To post to this group, send email to
> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> >>>> >> To unsubscribe from this group, send email to
> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> >>>> >> For more options, visit this group at
> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
> >>> >>>> >>
> >>> >>>> >
> >>> >>>> > --
> >>> >>>> > You received this message because you are subscribed to the
> Google
> >>> >>>> > Groups
> >>> >>>> > "mongodb-user" group.
> >>> >>>> > To post to this group, send email to
> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> >>>> > To unsubscribe from this group, send email to
> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> >>>> > For more options, visit this group at
> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
> >>> >>>>
> >>> >>>> --
> >>> >>>> You received this message because you are subscribed to the Google
> >>> >>>> Groups "mongodb-user" group.
> >>> >>>> To post to this group, send email to
> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> >>>> To unsubscribe from this group, send email to
> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> >>>> For more options, visit this group at
> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
> >>> >>>>
> >>> >>>
> >>> >>> --
> >>> >>> You received this message because you are subscribed to the Google
> >>> >>> Groups
> >>> >>> "mongodb-user" group.
> >>> >>> To post to this group, send email to mongodb-user-/***@public.gmane.org
> .
> >>> >>> To unsubscribe from this group, send email to
> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> >>> For more options, visit this group at
> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
> >>> >>
> >>> >>
> >>> >> --
> >>> >> You received this message because you are subscribed to the Google
> >>> >> Groups
> >>> >> "mongodb-user" group.
> >>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> >> To unsubscribe from this group, send email to
> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> >> For more options, visit this group at
> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
> >>> >
> >>> >
> >>> > --
> >>> > You received this message because you are subscribed to the Google
> >>> > Groups
> >>> > "mongodb-user" group.
> >>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> > To unsubscribe from this group, send email to
> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> > For more options, visit this group at
> >>> > http://groups.google.com/group/mongodb-user?hl=en.
> >>>
> >>> --
> >>> You received this message because you are subscribed to the Google
> Groups
> >>> "mongodb-user" group.
> >>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> To unsubscribe from this group, send email to
> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >>> For more options, visit this group at
> >>> http://groups.google.com/group/mongodb-user?hl=en.
> >>>
> >>
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > To unsubscribe from this group, send email to
> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > For more options, visit this group at
> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Scott Hernandez
2012-05-10 01:18:47 UTC
Permalink
The logic there is to do each document write, during the migrate, in a
separate lock so other operations can be interleaved. We have seen
this change dramatically reduce the constant lock percentage and allow
reads, and other writes through during a initial move.

There is already a delay between moves which doesn't seem to be your problem.

On Wed, May 9, 2012 at 5:37 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
> Scott,
>
> I tried to understand what's going on there. But with a quick glance, I
> assume, a lock was being held longer than necessary and was fixed. It's
> unclear how this can help me. The only way to reliably make sure the site
> keeps running while a moveChunk is executing, is if, you yield every second
> within moveChunk and make sure all pending reads complete, before the next
> moveChunk period starts.
>
> Are you guys doing it, or planning to do it? Is there any other high-level
> way to guarantee the reads can be answered within a couple of seconds?
>
> Thanks
>
>
> On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez <scotthernandez-***@public.gmane.org>
> wrote:
>>
>> Okay, looking at the receiving shard mongo55, and donor shard mongo43,
>> it look like the fix in 2.0.6 (due out in the next two weeks of so)
>> will help you out a lot.
>>
>> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>> > 4:37 the balancer has started and disk shot up.
>> >
>> >
>> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>> > wrote:
>> >>
>> >> Hey Scott,
>> >>
>> >> I'm going to do this now. Will start it and stop it quickly and give
>> >> you
>> >> the exact times.
>> >>
>> >> Thanks,
>> >> -Prasanna
>> >>
>> >>
>> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
>> >> <scotthernandez-***@public.gmane.org>
>> >> wrote:
>> >>>
>> >>> Those values are in blocks (512 bytes), not MB.
>> >>>
>> >>> We have an update in the upcoming 2.0.6 stable release which will help
>> >>> with migrations a bit. https://jira.mongodb.org/browse/SERVER-5303 You
>> >>> can click on the version like to see other important fixes.
>> >>>
>> >>> Can you do a few migrates tonight so we can see the stats/behavior in
>> >>> MMS? Please let  us know when you do this so we can take a look at the
>> >>> data to see if this fix will help.
>> >>>
>> >>>
>> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >>> wrote:
>> >>> > Hey Dan,
>> >>> >
>> >>> > Read ahead settings are the defaults (256
>> >>> > MB): http://pastebin.com/MHc8s2fq
>> >>> > Let me know if you need something else.
>> >>> >
>> >>> > @Elliot,
>> >>> >
>> >>> > I spoke with some of my friends at foursquare and it turns out they
>> >>> > face the
>> >>> > same issue as well. I've had my other friends who run companies >
>> >>> > $100M
>> >>> > market cap (whom I'd rather not name in this list), tell me about
>> >>> > their
>> >>> > mongo problems in the past few days. This is pretty serious stuff
>> >>> > and
>> >>> > will
>> >>> > affect if someone wants to choose mongo as their primary store or
>> >>> > not.
>> >>> > To
>> >>> > not have sharding is one thing, but to promise it and being stuck in
>> >>> > production is another.
>> >>> >
>> >>> > I really thank you guys and appreciate the responsiveness so far.
>> >>> > Happy
>> >>> > to
>> >>> > work together and hope we can resolve this.
>> >>> >
>> >>> > Thanks so much,
>> >>> > -Prasanna
>> >>> >
>> >>> >
>> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <dan-***@public.gmane.org> wrote:
>> >>> >>
>> >>> >> Looks like the docs in that collection
>> >>> >> (lal_production.facebook_names)
>> >>> >> are
>> >>> >> small and likely distributed throughout the disk.
>> >>> >> Can you check your disk readahead setting?
>> >>> >> Please run and post the results of blockev --report.
>> >>> >>
>> >>> >>
>> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >>> >> wrote:
>> >>> >>>
>> >>> >>> Yes, indeed.
>> >>> >>>
>> >>> >>> Username: arnold-***@public.gmane.org
>> >>> >>> Machine: mongo55
>> >>> >>>
>> >>> >>>
>> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz <eliot-***@public.gmane.org>
>> >>> >>> wrote:
>> >>> >>>>
>> >>> >>>> Are these machines in MMS?
>> >>> >>>> Hard to see much in the pasted stats.
>> >>> >>>>
>> >>> >>>>
>> >>> >>>>
>> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
>> >>> >>>> <myprasanna-***@public.gmane.org>
>> >>> >>>> wrote:
>> >>> >>>> > Raid 10 on 8 disks.
>> >>> >>>> >
>> >>> >>>> > MongoDB shell version: 2.0.4
>> >>> >>>> > Tue May  8 09:02:29 [initandlisten] db version v2.0.4, pdfile
>> >>> >>>> > version
>> >>> >>>> > 4.5
>> >>> >>>> >
>> >>> >>>> >
>> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
>> >>> >>>> > <eliot-***@public.gmane.org>
>> >>> >>>> > wrote:
>> >>> >>>> >>
>> >>> >>>> >> What version of mongo and what kind of ebs raid?
>> >>> >>>> >>
>> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
>> >>> >>>> >> <myprasanna-***@public.gmane.org>
>> >>> >>>> >> wrote:
>> >>> >>>> >> > And yes, disabling the balancer, brings the write rate back
>> >>> >>>> >> > to
>> >>> >>>> >> > zero.
>> >>> >>>> >> > Although after 4 minutes!
>> >>> >>>> >> >
>> >>> >>>> >> >
>> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
>> >>> >>>> >> >>
>> >>> >>>> >> >> I don't think it goes into deadlock. Can you try disabling
>> >>> >>>> >> >> the
>> >>> >>>> >> >> balancer
>> >>> >>>> >> >> and see whether the disk write rate going down or not.
>> >>> >>>> >> >> ________________________________
>> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>; Hrushikesh
>> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>> >>> >>>> >> >>
>> >>> >>>> >> >> Hey guys,
>> >>> >>>> >> >>
>> >>> >>>> >> >> We sharded mongo as per the docs. The issue is the balancer
>> >>> >>>> >> >> is
>> >>> >>>> >> >> not
>> >>> >>>> >> >> being
>> >>> >>>> >> >> very clever and not throttling at all. Now we are stuck
>> >>> >>>> >> >> with a
>> >>> >>>> >> >> half-sharded
>> >>> >>>> >> >> setup and we can't turn on the balancer because it'd bring
>> >>> >>>> >> >> the
>> >>> >>>> >> >> site
>> >>> >>>> >> >> down.
>> >>> >>>> >> >> All read queries timeout when the disk write rate is 100%.
>> >>> >>>> >> >>
>> >>> >>>> >> >> Is there a way out of this mess at all? From a high-level,
>> >>> >>>> >> >> mongo
>> >>> >>>> >> >> could
>> >>> >>>> >> >> have handled this by any of the following means:
>> >>> >>>> >> >>
>> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
>> >>> >>>> >> >> 2. Throttle and replicate only as long as the disk util is
>> >>> >>>> >> >> <
>> >>> >>>> >> >> 50%
>> >>> >>>> >> >> or so.
>> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads and writes
>> >>> >>>> >> >> come
>> >>> >>>> >> >> in-between,
>> >>> >>>> >> >> consider them as higher priority, so that a read/write is
>> >>> >>>> >> >> never
>> >>> >>>> >> >> waiting
>> >>> >>>> >> >> for
>> >>> >>>> >> >> more than a second.
>> >>> >>>> >> >>
>> >>> >>>> >> >> Have you guys heard of this problem before? Because this
>> >>> >>>> >> >> really
>> >>> >>>> >> >> make
>> >>> >>>> >> >> sharding impossible. I'm not sure what others are doing in
>> >>> >>>> >> >> production
>> >>> >>>> >> >> to
>> >>> >>>> >> >> work around this.
>> >>> >>>> >> >>
>> >>> >>>> >> >> Thanks,
>> >>> >>>> >> >> -Prasanna
>> >>> >>>> >> >> Co-Founder & CTO, lal.com
>> >>> >>>> >> >>
>> >>> >>>> >> >> --
>> >>> >>>> >> >> You received this message because you are subscribed to the
>> >>> >>>> >> >> Google
>> >>> >>>> >> >> Groups
>> >>> >>>> >> >> "mongodb-user" group.
>> >>> >>>> >> >> To post to this group, send email to
>> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> >>>> >> >> To unsubscribe from this group, send email to
>> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> >>>> >> >> For more options, visit this group at
>> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>> >>>> >> >
>> >>> >>>> >> > --
>> >>> >>>> >> > You received this message because you are subscribed to the
>> >>> >>>> >> > Google
>> >>> >>>> >> > Groups
>> >>> >>>> >> > "mongodb-user" group.
>> >>> >>>> >> > To view this discussion on the web visit
>> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >>> >>>> >> >
>> >>> >>>> >> > To post to this group, send email to
>> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> >>>> >> > To unsubscribe from this group, send email to
>> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> >>>> >> > For more options, visit this group at
>> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>> >>>> >>
>> >>> >>>> >> --
>> >>> >>>> >> You received this message because you are subscribed to the
>> >>> >>>> >> Google
>> >>> >>>> >> Groups
>> >>> >>>> >> "mongodb-user" group.
>> >>> >>>> >> To post to this group, send email to
>> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> >>>> >> To unsubscribe from this group, send email to
>> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> >>>> >> For more options, visit this group at
>> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>> >>>> >>
>> >>> >>>> >
>> >>> >>>> > --
>> >>> >>>> > You received this message because you are subscribed to the
>> >>> >>>> > Google
>> >>> >>>> > Groups
>> >>> >>>> > "mongodb-user" group.
>> >>> >>>> > To post to this group, send email to
>> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> >>>> > To unsubscribe from this group, send email to
>> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> >>>> > For more options, visit this group at
>> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>> >>>>
>> >>> >>>> --
>> >>> >>>> You received this message because you are subscribed to the
>> >>> >>>> Google
>> >>> >>>> Groups "mongodb-user" group.
>> >>> >>>> To post to this group, send email to
>> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> >>>> To unsubscribe from this group, send email to
>> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> >>>> For more options, visit this group at
>> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >>> >>>>
>> >>> >>>
>> >>> >>> --
>> >>> >>> You received this message because you are subscribed to the Google
>> >>> >>> Groups
>> >>> >>> "mongodb-user" group.
>> >>> >>> To post to this group, send email to
>> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> >>> To unsubscribe from this group, send email to
>> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> >>> For more options, visit this group at
>> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >>> >>
>> >>> >>
>> >>> >> --
>> >>> >> You received this message because you are subscribed to the Google
>> >>> >> Groups
>> >>> >> "mongodb-user" group.
>> >>> >> To post to this group, send email to mongodb-user-/***@public.gmane.orgm.
>> >>> >> To unsubscribe from this group, send email to
>> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> >> For more options, visit this group at
>> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>> >
>> >>> >
>> >>> > --
>> >>> > You received this message because you are subscribed to the Google
>> >>> > Groups
>> >>> > "mongodb-user" group.
>> >>> > To post to this group, send email to mongodb-user-/***@public.gmane.org.
>> >>> > To unsubscribe from this group, send email to
>> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> > For more options, visit this group at
>> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>>
>> >>> --
>> >>> You received this message because you are subscribed to the Google
>> >>> Groups
>> >>> "mongodb-user" group.
>> >>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> To unsubscribe from this group, send email to
>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >>> For more options, visit this group at
>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >>>
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > To unsubscribe from this group, send email to
>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-10 02:46:23 UTC
Permalink
Scott,

The below are heuristics, and not guarantees. What you want from a DB
stand-point is the latter.

The delay between moves, won't provide a guarantee, unless you make sure
that
(1) the next iteration of moveChunk, does not start unless all pending
reads complete.
(2) Each iteration takes less than 1second.

Thoughts?

On Wed, May 9, 2012 at 6:18 PM, Scott Hernandez <scotthernandez-***@public.gmane.org>wrote:

> The logic there is to do each document write, during the migrate, in a
> separate lock so other operations can be interleaved. We have seen
> this change dramatically reduce the constant lock percentage and allow
> reads, and other writes through during a initial move.
>
> There is already a delay between moves which doesn't seem to be your
> problem.
>
> On Wed, May 9, 2012 at 5:37 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
> > Scott,
> >
> > I tried to understand what's going on there. But with a quick glance, I
> > assume, a lock was being held longer than necessary and was fixed. It's
> > unclear how this can help me. The only way to reliably make sure the site
> > keeps running while a moveChunk is executing, is if, you yield every
> second
> > within moveChunk and make sure all pending reads complete, before the
> next
> > moveChunk period starts.
> >
> > Are you guys doing it, or planning to do it? Is there any other
> high-level
> > way to guarantee the reads can be answered within a couple of seconds?
> >
> > Thanks
> >
> >
> > On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez <
> scotthernandez-***@public.gmane.org>
> > wrote:
> >>
> >> Okay, looking at the receiving shard mongo55, and donor shard mongo43,
> >> it look like the fix in 2.0.6 (due out in the next two weeks of so)
> >> will help you out a lot.
> >>
> >> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S <myprasanna-***@public.gmane.org>
> wrote:
> >> > 4:37 the balancer has started and disk shot up.
> >> >
> >> >
> >> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S <myprasanna-***@public.gmane.org>
> >> > wrote:
> >> >>
> >> >> Hey Scott,
> >> >>
> >> >> I'm going to do this now. Will start it and stop it quickly and give
> >> >> you
> >> >> the exact times.
> >> >>
> >> >> Thanks,
> >> >> -Prasanna
> >> >>
> >> >>
> >> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
> >> >> <scotthernandez-***@public.gmane.org>
> >> >> wrote:
> >> >>>
> >> >>> Those values are in blocks (512 bytes), not MB.
> >> >>>
> >> >>> We have an update in the upcoming 2.0.6 stable release which will
> help
> >> >>> with migrations a bit. https://jira.mongodb.org/browse/SERVER-5303You
> >> >>> can click on the version like to see other important fixes.
> >> >>>
> >> >>> Can you do a few migrates tonight so we can see the stats/behavior
> in
> >> >>> MMS? Please let us know when you do this so we can take a look at
> the
> >> >>> data to see if this fix will help.
> >> >>>
> >> >>>
> >> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S <myprasanna-***@public.gmane.org>
> >> >>> wrote:
> >> >>> > Hey Dan,
> >> >>> >
> >> >>> > Read ahead settings are the defaults (256
> >> >>> > MB): http://pastebin.com/MHc8s2fq
> >> >>> > Let me know if you need something else.
> >> >>> >
> >> >>> > @Elliot,
> >> >>> >
> >> >>> > I spoke with some of my friends at foursquare and it turns out
> they
> >> >>> > face the
> >> >>> > same issue as well. I've had my other friends who run companies >
> >> >>> > $100M
> >> >>> > market cap (whom I'd rather not name in this list), tell me about
> >> >>> > their
> >> >>> > mongo problems in the past few days. This is pretty serious stuff
> >> >>> > and
> >> >>> > will
> >> >>> > affect if someone wants to choose mongo as their primary store or
> >> >>> > not.
> >> >>> > To
> >> >>> > not have sharding is one thing, but to promise it and being stuck
> in
> >> >>> > production is another.
> >> >>> >
> >> >>> > I really thank you guys and appreciate the responsiveness so far.
> >> >>> > Happy
> >> >>> > to
> >> >>> > work together and hope we can resolve this.
> >> >>> >
> >> >>> > Thanks so much,
> >> >>> > -Prasanna
> >> >>> >
> >> >>> >
> >> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <dan-***@public.gmane.org>
> wrote:
> >> >>> >>
> >> >>> >> Looks like the docs in that collection
> >> >>> >> (lal_production.facebook_names)
> >> >>> >> are
> >> >>> >> small and likely distributed throughout the disk.
> >> >>> >> Can you check your disk readahead setting?
> >> >>> >> Please run and post the results of blockev --report.
> >> >>> >>
> >> >>> >>
> >> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S <
> myprasanna-***@public.gmane.org>
> >> >>> >> wrote:
> >> >>> >>>
> >> >>> >>> Yes, indeed.
> >> >>> >>>
> >> >>> >>> Username: arnold-***@public.gmane.org
> >> >>> >>> Machine: mongo55
> >> >>> >>>
> >> >>> >>>
> >> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz <eliot-***@public.gmane.org
> >
> >> >>> >>> wrote:
> >> >>> >>>>
> >> >>> >>>> Are these machines in MMS?
> >> >>> >>>> Hard to see much in the pasted stats.
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>>
> >> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
> >> >>> >>>> <myprasanna-***@public.gmane.org>
> >> >>> >>>> wrote:
> >> >>> >>>> > Raid 10 on 8 disks.
> >> >>> >>>> >
> >> >>> >>>> > MongoDB shell version: 2.0.4
> >> >>> >>>> > Tue May 8 09:02:29 [initandlisten] db version v2.0.4, pdfile
> >> >>> >>>> > version
> >> >>> >>>> > 4.5
> >> >>> >>>> >
> >> >>> >>>> >
> >> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
> >> >>> >>>> > <eliot-***@public.gmane.org>
> >> >>> >>>> > wrote:
> >> >>> >>>> >>
> >> >>> >>>> >> What version of mongo and what kind of ebs raid?
> >> >>> >>>> >>
> >> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
> >> >>> >>>> >> <myprasanna-***@public.gmane.org>
> >> >>> >>>> >> wrote:
> >> >>> >>>> >> > And yes, disabling the balancer, brings the write rate
> back
> >> >>> >>>> >> > to
> >> >>> >>>> >> > zero.
> >> >>> >>>> >> > Although after 4 minutes!
> >> >>> >>>> >> >
> >> >>> >>>> >> >
> >> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
> >> >>> >>>> >> >>
> >> >>> >>>> >> >> I don't think it goes into deadlock. Can you try
> disabling
> >> >>> >>>> >> >> the
> >> >>> >>>> >> >> balancer
> >> >>> >>>> >> >> and see whether the disk write rate going down or not.
> >> >>> >>>> >> >> ________________________________
> >> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
> >> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
> >> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
> >> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
> >> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
> >> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>;
> Hrushikesh
> >> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
> >> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
> >> >>> >>>> >> >>
> >> >>> >>>> >> >> Hey guys,
> >> >>> >>>> >> >>
> >> >>> >>>> >> >> We sharded mongo as per the docs. The issue is the
> balancer
> >> >>> >>>> >> >> is
> >> >>> >>>> >> >> not
> >> >>> >>>> >> >> being
> >> >>> >>>> >> >> very clever and not throttling at all. Now we are stuck
> >> >>> >>>> >> >> with a
> >> >>> >>>> >> >> half-sharded
> >> >>> >>>> >> >> setup and we can't turn on the balancer because it'd
> bring
> >> >>> >>>> >> >> the
> >> >>> >>>> >> >> site
> >> >>> >>>> >> >> down.
> >> >>> >>>> >> >> All read queries timeout when the disk write rate is
> 100%.
> >> >>> >>>> >> >>
> >> >>> >>>> >> >> Is there a way out of this mess at all? From a
> high-level,
> >> >>> >>>> >> >> mongo
> >> >>> >>>> >> >> could
> >> >>> >>>> >> >> have handled this by any of the following means:
> >> >>> >>>> >> >>
> >> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
> >> >>> >>>> >> >> 2. Throttle and replicate only as long as the disk util
> is
> >> >>> >>>> >> >> <
> >> >>> >>>> >> >> 50%
> >> >>> >>>> >> >> or so.
> >> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads and writes
> >> >>> >>>> >> >> come
> >> >>> >>>> >> >> in-between,
> >> >>> >>>> >> >> consider them as higher priority, so that a read/write is
> >> >>> >>>> >> >> never
> >> >>> >>>> >> >> waiting
> >> >>> >>>> >> >> for
> >> >>> >>>> >> >> more than a second.
> >> >>> >>>> >> >>
> >> >>> >>>> >> >> Have you guys heard of this problem before? Because this
> >> >>> >>>> >> >> really
> >> >>> >>>> >> >> make
> >> >>> >>>> >> >> sharding impossible. I'm not sure what others are doing
> in
> >> >>> >>>> >> >> production
> >> >>> >>>> >> >> to
> >> >>> >>>> >> >> work around this.
> >> >>> >>>> >> >>
> >> >>> >>>> >> >> Thanks,
> >> >>> >>>> >> >> -Prasanna
> >> >>> >>>> >> >> Co-Founder & CTO, lal.com
> >> >>> >>>> >> >>
> >> >>> >>>> >> >> --
> >> >>> >>>> >> >> You received this message because you are subscribed to
> the
> >> >>> >>>> >> >> Google
> >> >>> >>>> >> >> Groups
> >> >>> >>>> >> >> "mongodb-user" group.
> >> >>> >>>> >> >> To post to this group, send email to
> >> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> >>>> >> >> To unsubscribe from this group, send email to
> >> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> >>>> >> >> For more options, visit this group at
> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >>> >>>> >> >
> >> >>> >>>> >> > --
> >> >>> >>>> >> > You received this message because you are subscribed to
> the
> >> >>> >>>> >> > Google
> >> >>> >>>> >> > Groups
> >> >>> >>>> >> > "mongodb-user" group.
> >> >>> >>>> >> > To view this discussion on the web visit
> >> >>> >>>> >> >
> https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
> >> >>> >>>> >> >
> >> >>> >>>> >> > To post to this group, send email to
> >> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> >>>> >> > To unsubscribe from this group, send email to
> >> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> >>>> >> > For more options, visit this group at
> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >>> >>>> >>
> >> >>> >>>> >> --
> >> >>> >>>> >> You received this message because you are subscribed to the
> >> >>> >>>> >> Google
> >> >>> >>>> >> Groups
> >> >>> >>>> >> "mongodb-user" group.
> >> >>> >>>> >> To post to this group, send email to
> >> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> >>>> >> To unsubscribe from this group, send email to
> >> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> >>>> >> For more options, visit this group at
> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >>> >>>> >>
> >> >>> >>>> >
> >> >>> >>>> > --
> >> >>> >>>> > You received this message because you are subscribed to the
> >> >>> >>>> > Google
> >> >>> >>>> > Groups
> >> >>> >>>> > "mongodb-user" group.
> >> >>> >>>> > To post to this group, send email to
> >> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> >>>> > To unsubscribe from this group, send email to
> >> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> >>>> > For more options, visit this group at
> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >>> >>>>
> >> >>> >>>> --
> >> >>> >>>> You received this message because you are subscribed to the
> >> >>> >>>> Google
> >> >>> >>>> Groups "mongodb-user" group.
> >> >>> >>>> To post to this group, send email to
> >> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> >>>> To unsubscribe from this group, send email to
> >> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> >>>> For more options, visit this group at
> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >>> >>>>
> >> >>> >>>
> >> >>> >>> --
> >> >>> >>> You received this message because you are subscribed to the
> Google
> >> >>> >>> Groups
> >> >>> >>> "mongodb-user" group.
> >> >>> >>> To post to this group, send email to
> >> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> >>> To unsubscribe from this group, send email to
> >> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> >>> For more options, visit this group at
> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >>> >>
> >> >>> >>
> >> >>> >> --
> >> >>> >> You received this message because you are subscribed to the
> Google
> >> >>> >> Groups
> >> >>> >> "mongodb-user" group.
> >> >>> >> To post to this group, send email to
> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> >> To unsubscribe from this group, send email to
> >> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> >> For more options, visit this group at
> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >>> >
> >> >>> >
> >> >>> > --
> >> >>> > You received this message because you are subscribed to the Google
> >> >>> > Groups
> >> >>> > "mongodb-user" group.
> >> >>> > To post to this group, send email to
> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> > To unsubscribe from this group, send email to
> >> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> > For more options, visit this group at
> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >>>
> >> >>> --
> >> >>> You received this message because you are subscribed to the Google
> >> >>> Groups
> >> >>> "mongodb-user" group.
> >> >>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> To unsubscribe from this group, send email to
> >> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >>> For more options, visit this group at
> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >>>
> >> >>
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups
> >> > "mongodb-user" group.
> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> > To unsubscribe from this group, send email to
> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> > For more options, visit this group at
> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "mongodb-user" group.
> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> To unsubscribe from this group, send email to
> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> For more options, visit this group at
> >> http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > To unsubscribe from this group, send email to
> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > For more options, visit this group at
> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Scott Hernandez
2012-05-10 14:19:32 UTC
Permalink
There are lots of ways to look at this with some competing goals.
While it might be paramount for some situations/use-cases to complete
all reads before a move starts, it might not be for others (which I
have seen). This fix will allow reads (and other writes) to get
processes while chunks moves happen, period. The issue of locking, and
contentions on those locks, is very important and there are lots of
improvements being made which help across the boards.

Feel free to add comments or create new issues with your ideas of how
your system would like things to best behave:
http://jira.mongodb.org/browse/server

On Wed, May 9, 2012 at 7:46 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
> Scott,
>
> The below are heuristics, and not guarantees. What you want from a DB
> stand-point is the latter.
>
> The delay between moves, won't provide a guarantee, unless you make sure
> that
> (1) the next iteration of moveChunk, does not start unless all pending reads
> complete.
> (2) Each iteration takes less than 1second.
>
> Thoughts?
>
>
> On Wed, May 9, 2012 at 6:18 PM, Scott Hernandez <scotthernandez-***@public.gmane.org>
> wrote:
>>
>> The logic there is to do each document write, during the migrate, in a
>> separate lock so other operations can be interleaved. We have seen
>> this change dramatically reduce the constant lock percentage and allow
>> reads, and other writes through during a initial move.
>>
>> There is already a delay between moves which doesn't seem to be your
>> problem.
>>
>> On Wed, May 9, 2012 at 5:37 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>> > Scott,
>> >
>> > I tried to understand what's going on there. But with a quick glance, I
>> > assume, a lock was being held longer than necessary and was fixed. It's
>> > unclear how this can help me. The only way to reliably make sure the
>> > site
>> > keeps running while a moveChunk is executing, is if, you yield every
>> > second
>> > within moveChunk and make sure all pending reads complete, before the
>> > next
>> > moveChunk period starts.
>> >
>> > Are you guys doing it, or planning to do it? Is there any other
>> > high-level
>> > way to guarantee the reads can be answered within a couple of seconds?
>> >
>> > Thanks
>> >
>> >
>> > On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez
>> > <scotthernandez-***@public.gmane.org>
>> > wrote:
>> >>
>> >> Okay, looking at the receiving shard mongo55, and donor shard mongo43,
>> >> it look like the fix in 2.0.6 (due out in the next two weeks of so)
>> >> will help you out a lot.
>> >>
>> >> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> wrote:
>> >> > 4:37 the balancer has started and disk shot up.
>> >> >
>> >> >
>> >> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> > wrote:
>> >> >>
>> >> >> Hey Scott,
>> >> >>
>> >> >> I'm going to do this now. Will start it and stop it quickly and give
>> >> >> you
>> >> >> the exact times.
>> >> >>
>> >> >> Thanks,
>> >> >> -Prasanna
>> >> >>
>> >> >>
>> >> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
>> >> >> <scotthernandez-***@public.gmane.org>
>> >> >> wrote:
>> >> >>>
>> >> >>> Those values are in blocks (512 bytes), not MB.
>> >> >>>
>> >> >>> We have an update in the upcoming 2.0.6 stable release which will
>> >> >>> help
>> >> >>> with migrations a bit. https://jira.mongodb.org/browse/SERVER-5303
>> >> >>> You
>> >> >>> can click on the version like to see other important fixes.
>> >> >>>
>> >> >>> Can you do a few migrates tonight so we can see the stats/behavior
>> >> >>> in
>> >> >>> MMS? Please let  us know when you do this so we can take a look at
>> >> >>> the
>> >> >>> data to see if this fix will help.
>> >> >>>
>> >> >>>
>> >> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S <myprasanna-***@public.gmane.orgm>
>> >> >>> wrote:
>> >> >>> > Hey Dan,
>> >> >>> >
>> >> >>> > Read ahead settings are the defaults (256
>> >> >>> > MB): http://pastebin.com/MHc8s2fq
>> >> >>> > Let me know if you need something else.
>> >> >>> >
>> >> >>> > @Elliot,
>> >> >>> >
>> >> >>> > I spoke with some of my friends at foursquare and it turns out
>> >> >>> > they
>> >> >>> > face the
>> >> >>> > same issue as well. I've had my other friends who run companies >
>> >> >>> > $100M
>> >> >>> > market cap (whom I'd rather not name in this list), tell me about
>> >> >>> > their
>> >> >>> > mongo problems in the past few days. This is pretty serious stuff
>> >> >>> > and
>> >> >>> > will
>> >> >>> > affect if someone wants to choose mongo as their primary store or
>> >> >>> > not.
>> >> >>> > To
>> >> >>> > not have sharding is one thing, but to promise it and being stuck
>> >> >>> > in
>> >> >>> > production is another.
>> >> >>> >
>> >> >>> > I really thank you guys and appreciate the responsiveness so far.
>> >> >>> > Happy
>> >> >>> > to
>> >> >>> > work together and hope we can resolve this.
>> >> >>> >
>> >> >>> > Thanks so much,
>> >> >>> > -Prasanna
>> >> >>> >
>> >> >>> >
>> >> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <dan-***@public.gmane.org>
>> >> >>> > wrote:
>> >> >>> >>
>> >> >>> >> Looks like the docs in that collection
>> >> >>> >> (lal_production.facebook_names)
>> >> >>> >> are
>> >> >>> >> small and likely distributed throughout the disk.
>> >> >>> >> Can you check your disk readahead setting?
>> >> >>> >> Please run and post the results of blockev --report.
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S
>> >> >>> >> <myprasanna-***@public.gmane.org>
>> >> >>> >> wrote:
>> >> >>> >>>
>> >> >>> >>> Yes, indeed.
>> >> >>> >>>
>> >> >>> >>> Username: arnold-***@public.gmane.org
>> >> >>> >>> Machine: mongo55
>> >> >>> >>>
>> >> >>> >>>
>> >> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz
>> >> >>> >>> <eliot-***@public.gmane.org>
>> >> >>> >>> wrote:
>> >> >>> >>>>
>> >> >>> >>>> Are these machines in MMS?
>> >> >>> >>>> Hard to see much in the pasted stats.
>> >> >>> >>>>
>> >> >>> >>>>
>> >> >>> >>>>
>> >> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
>> >> >>> >>>> <myprasanna-***@public.gmane.org>
>> >> >>> >>>> wrote:
>> >> >>> >>>> > Raid 10 on 8 disks.
>> >> >>> >>>> >
>> >> >>> >>>> > MongoDB shell version: 2.0.4
>> >> >>> >>>> > Tue May  8 09:02:29 [initandlisten] db version v2.0.4,
>> >> >>> >>>> > pdfile
>> >> >>> >>>> > version
>> >> >>> >>>> > 4.5
>> >> >>> >>>> >
>> >> >>> >>>> >
>> >> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
>> >> >>> >>>> > <eliot-***@public.gmane.org>
>> >> >>> >>>> > wrote:
>> >> >>> >>>> >>
>> >> >>> >>>> >> What version of mongo and what kind of ebs raid?
>> >> >>> >>>> >>
>> >> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
>> >> >>> >>>> >> <myprasanna-***@public.gmane.org>
>> >> >>> >>>> >> wrote:
>> >> >>> >>>> >> > And yes, disabling the balancer, brings the write rate
>> >> >>> >>>> >> > back
>> >> >>> >>>> >> > to
>> >> >>> >>>> >> > zero.
>> >> >>> >>>> >> > Although after 4 minutes!
>> >> >>> >>>> >> >
>> >> >>> >>>> >> >
>> >> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
>> >> >>> >>>> >> >>
>> >> >>> >>>> >> >> I don't think it goes into deadlock. Can you try
>> >> >>> >>>> >> >> disabling
>> >> >>> >>>> >> >> the
>> >> >>> >>>> >> >> balancer
>> >> >>> >>>> >> >> and see whether the disk write rate going down or not.
>> >> >>> >>>> >> >> ________________________________
>> >> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>> >> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>> >> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>> >> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>> >> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>> >> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>;
>> >> >>> >>>> >> >> Hrushikesh
>> >> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>> >> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>> >> >>> >>>> >> >>
>> >> >>> >>>> >> >> Hey guys,
>> >> >>> >>>> >> >>
>> >> >>> >>>> >> >> We sharded mongo as per the docs. The issue is the
>> >> >>> >>>> >> >> balancer
>> >> >>> >>>> >> >> is
>> >> >>> >>>> >> >> not
>> >> >>> >>>> >> >> being
>> >> >>> >>>> >> >> very clever and not throttling at all. Now we are stuck
>> >> >>> >>>> >> >> with a
>> >> >>> >>>> >> >> half-sharded
>> >> >>> >>>> >> >> setup and we can't turn on the balancer because it'd
>> >> >>> >>>> >> >> bring
>> >> >>> >>>> >> >> the
>> >> >>> >>>> >> >> site
>> >> >>> >>>> >> >> down.
>> >> >>> >>>> >> >> All read queries timeout when the disk write rate is
>> >> >>> >>>> >> >> 100%.
>> >> >>> >>>> >> >>
>> >> >>> >>>> >> >> Is there a way out of this mess at all? From a
>> >> >>> >>>> >> >> high-level,
>> >> >>> >>>> >> >> mongo
>> >> >>> >>>> >> >> could
>> >> >>> >>>> >> >> have handled this by any of the following means:
>> >> >>> >>>> >> >>
>> >> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
>> >> >>> >>>> >> >> 2. Throttle and replicate only as long as the disk util
>> >> >>> >>>> >> >> is
>> >> >>> >>>> >> >> <
>> >> >>> >>>> >> >> 50%
>> >> >>> >>>> >> >> or so.
>> >> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads and writes
>> >> >>> >>>> >> >> come
>> >> >>> >>>> >> >> in-between,
>> >> >>> >>>> >> >> consider them as higher priority, so that a read/write
>> >> >>> >>>> >> >> is
>> >> >>> >>>> >> >> never
>> >> >>> >>>> >> >> waiting
>> >> >>> >>>> >> >> for
>> >> >>> >>>> >> >> more than a second.
>> >> >>> >>>> >> >>
>> >> >>> >>>> >> >> Have you guys heard of this problem before? Because this
>> >> >>> >>>> >> >> really
>> >> >>> >>>> >> >> make
>> >> >>> >>>> >> >> sharding impossible. I'm not sure what others are doing
>> >> >>> >>>> >> >> in
>> >> >>> >>>> >> >> production
>> >> >>> >>>> >> >> to
>> >> >>> >>>> >> >> work around this.
>> >> >>> >>>> >> >>
>> >> >>> >>>> >> >> Thanks,
>> >> >>> >>>> >> >> -Prasanna
>> >> >>> >>>> >> >> Co-Founder & CTO, lal.com
>> >> >>> >>>> >> >>
>> >> >>> >>>> >> >> --
>> >> >>> >>>> >> >> You received this message because you are subscribed to
>> >> >>> >>>> >> >> the
>> >> >>> >>>> >> >> Google
>> >> >>> >>>> >> >> Groups
>> >> >>> >>>> >> >> "mongodb-user" group.
>> >> >>> >>>> >> >> To post to this group, send email to
>> >> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> >>>> >> >> To unsubscribe from this group, send email to
>> >> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> >>>> >> >> For more options, visit this group at
>> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>> >>>> >> >
>> >> >>> >>>> >> > --
>> >> >>> >>>> >> > You received this message because you are subscribed to
>> >> >>> >>>> >> > the
>> >> >>> >>>> >> > Google
>> >> >>> >>>> >> > Groups
>> >> >>> >>>> >> > "mongodb-user" group.
>> >> >>> >>>> >> > To view this discussion on the web visit
>> >> >>> >>>> >> >
>> >> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >>> >>>> >> >
>> >> >>> >>>> >> > To post to this group, send email to
>> >> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> >>>> >> > To unsubscribe from this group, send email to
>> >> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> >>>> >> > For more options, visit this group at
>> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>> >>>> >>
>> >> >>> >>>> >> --
>> >> >>> >>>> >> You received this message because you are subscribed to the
>> >> >>> >>>> >> Google
>> >> >>> >>>> >> Groups
>> >> >>> >>>> >> "mongodb-user" group.
>> >> >>> >>>> >> To post to this group, send email to
>> >> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> >>>> >> To unsubscribe from this group, send email to
>> >> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> >>>> >> For more options, visit this group at
>> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>> >>>> >>
>> >> >>> >>>> >
>> >> >>> >>>> > --
>> >> >>> >>>> > You received this message because you are subscribed to the
>> >> >>> >>>> > Google
>> >> >>> >>>> > Groups
>> >> >>> >>>> > "mongodb-user" group.
>> >> >>> >>>> > To post to this group, send email to
>> >> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> >>>> > To unsubscribe from this group, send email to
>> >> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> >>>> > For more options, visit this group at
>> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>> >>>>
>> >> >>> >>>> --
>> >> >>> >>>> You received this message because you are subscribed to the
>> >> >>> >>>> Google
>> >> >>> >>>> Groups "mongodb-user" group.
>> >> >>> >>>> To post to this group, send email to
>> >> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> >>>> To unsubscribe from this group, send email to
>> >> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> >>>> For more options, visit this group at
>> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>> >>>>
>> >> >>> >>>
>> >> >>> >>> --
>> >> >>> >>> You received this message because you are subscribed to the
>> >> >>> >>> Google
>> >> >>> >>> Groups
>> >> >>> >>> "mongodb-user" group.
>> >> >>> >>> To post to this group, send email to
>> >> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> >>> To unsubscribe from this group, send email to
>> >> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> >>> For more options, visit this group at
>> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>> >>
>> >> >>> >>
>> >> >>> >> --
>> >> >>> >> You received this message because you are subscribed to the
>> >> >>> >> Google
>> >> >>> >> Groups
>> >> >>> >> "mongodb-user" group.
>> >> >>> >> To post to this group, send email to
>> >> >>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> >> To unsubscribe from this group, send email to
>> >> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> >> For more options, visit this group at
>> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>> >
>> >> >>> >
>> >> >>> > --
>> >> >>> > You received this message because you are subscribed to the
>> >> >>> > Google
>> >> >>> > Groups
>> >> >>> > "mongodb-user" group.
>> >> >>> > To post to this group, send email to
>> >> >>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> > To unsubscribe from this group, send email to
>> >> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> > For more options, visit this group at
>> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>>
>> >> >>> --
>> >> >>> You received this message because you are subscribed to the Google
>> >> >>> Groups
>> >> >>> "mongodb-user" group.
>> >> >>> To post to this group, send email to mongodb-user-/***@public.gmane.orgm.
>> >> >>> To unsubscribe from this group, send email to
>> >> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >>> For more options, visit this group at
>> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>>
>> >> >>
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > To unsubscribe from this group, send email to
>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> To unsubscribe from this group, send email to
>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > To unsubscribe from this group, send email to
>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-10 14:33:41 UTC
Permalink
Scott,

> While it might be paramount for some situations/use-cases to complete all
reads before a move starts, it might not be for others (which I
have seen).

Can you please explain the "might not be for others" part? Can you give me
an example?


On Thu, May 10, 2012 at 7:19 AM, Scott Hernandez
<scotthernandez-***@public.gmane.org>wrote:

> There are lots of ways to look at this with some competing goals.
> While it might be paramount for some situations/use-cases to complete
> all reads before a move starts, it might not be for others (which I
> have seen). This fix will allow reads (and other writes) to get
> processes while chunks moves happen, period. The issue of locking, and
> contentions on those locks, is very important and there are lots of
> improvements being made which help across the boards.
>
> Feel free to add comments or create new issues with your ideas of how
> your system would like things to best behave:
> http://jira.mongodb.org/browse/server
>
> On Wed, May 9, 2012 at 7:46 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
> > Scott,
> >
> > The below are heuristics, and not guarantees. What you want from a DB
> > stand-point is the latter.
> >
> > The delay between moves, won't provide a guarantee, unless you make sure
> > that
> > (1) the next iteration of moveChunk, does not start unless all pending
> reads
> > complete.
> > (2) Each iteration takes less than 1second.
> >
> > Thoughts?
> >
> >
> > On Wed, May 9, 2012 at 6:18 PM, Scott Hernandez <
> scotthernandez-***@public.gmane.org>
> > wrote:
> >>
> >> The logic there is to do each document write, during the migrate, in a
> >> separate lock so other operations can be interleaved. We have seen
> >> this change dramatically reduce the constant lock percentage and allow
> >> reads, and other writes through during a initial move.
> >>
> >> There is already a delay between moves which doesn't seem to be your
> >> problem.
> >>
> >> On Wed, May 9, 2012 at 5:37 PM, Prasanna .S <myprasanna-***@public.gmane.org>
> wrote:
> >> > Scott,
> >> >
> >> > I tried to understand what's going on there. But with a quick glance,
> I
> >> > assume, a lock was being held longer than necessary and was fixed.
> It's
> >> > unclear how this can help me. The only way to reliably make sure the
> >> > site
> >> > keeps running while a moveChunk is executing, is if, you yield every
> >> > second
> >> > within moveChunk and make sure all pending reads complete, before the
> >> > next
> >> > moveChunk period starts.
> >> >
> >> > Are you guys doing it, or planning to do it? Is there any other
> >> > high-level
> >> > way to guarantee the reads can be answered within a couple of seconds?
> >> >
> >> > Thanks
> >> >
> >> >
> >> > On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez
> >> > <scotthernandez-***@public.gmane.org>
> >> > wrote:
> >> >>
> >> >> Okay, looking at the receiving shard mongo55, and donor shard
> mongo43,
> >> >> it look like the fix in 2.0.6 (due out in the next two weeks of so)
> >> >> will help you out a lot.
> >> >>
> >> >> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S <myprasanna-***@public.gmane.org>
> >> >> wrote:
> >> >> > 4:37 the balancer has started and disk shot up.
> >> >> >
> >> >> >
> >> >> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S <myprasanna-***@public.gmane.org>
> >> >> > wrote:
> >> >> >>
> >> >> >> Hey Scott,
> >> >> >>
> >> >> >> I'm going to do this now. Will start it and stop it quickly and
> give
> >> >> >> you
> >> >> >> the exact times.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> -Prasanna
> >> >> >>
> >> >> >>
> >> >> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
> >> >> >> <scotthernandez-***@public.gmane.org>
> >> >> >> wrote:
> >> >> >>>
> >> >> >>> Those values are in blocks (512 bytes), not MB.
> >> >> >>>
> >> >> >>> We have an update in the upcoming 2.0.6 stable release which will
> >> >> >>> help
> >> >> >>> with migrations a bit.
> https://jira.mongodb.org/browse/SERVER-5303
> >> >> >>> You
> >> >> >>> can click on the version like to see other important fixes.
> >> >> >>>
> >> >> >>> Can you do a few migrates tonight so we can see the
> stats/behavior
> >> >> >>> in
> >> >> >>> MMS? Please let us know when you do this so we can take a look
> at
> >> >> >>> the
> >> >> >>> data to see if this fix will help.
> >> >> >>>
> >> >> >>>
> >> >> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S <
> myprasanna-***@public.gmane.org>
> >> >> >>> wrote:
> >> >> >>> > Hey Dan,
> >> >> >>> >
> >> >> >>> > Read ahead settings are the defaults (256
> >> >> >>> > MB): http://pastebin.com/MHc8s2fq
> >> >> >>> > Let me know if you need something else.
> >> >> >>> >
> >> >> >>> > @Elliot,
> >> >> >>> >
> >> >> >>> > I spoke with some of my friends at foursquare and it turns out
> >> >> >>> > they
> >> >> >>> > face the
> >> >> >>> > same issue as well. I've had my other friends who run
> companies >
> >> >> >>> > $100M
> >> >> >>> > market cap (whom I'd rather not name in this list), tell me
> about
> >> >> >>> > their
> >> >> >>> > mongo problems in the past few days. This is pretty serious
> stuff
> >> >> >>> > and
> >> >> >>> > will
> >> >> >>> > affect if someone wants to choose mongo as their primary store
> or
> >> >> >>> > not.
> >> >> >>> > To
> >> >> >>> > not have sharding is one thing, but to promise it and being
> stuck
> >> >> >>> > in
> >> >> >>> > production is another.
> >> >> >>> >
> >> >> >>> > I really thank you guys and appreciate the responsiveness so
> far.
> >> >> >>> > Happy
> >> >> >>> > to
> >> >> >>> > work together and hope we can resolve this.
> >> >> >>> >
> >> >> >>> > Thanks so much,
> >> >> >>> > -Prasanna
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <dan-***@public.gmane.org>
> >> >> >>> > wrote:
> >> >> >>> >>
> >> >> >>> >> Looks like the docs in that collection
> >> >> >>> >> (lal_production.facebook_names)
> >> >> >>> >> are
> >> >> >>> >> small and likely distributed throughout the disk.
> >> >> >>> >> Can you check your disk readahead setting?
> >> >> >>> >> Please run and post the results of blockev --report.
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S
> >> >> >>> >> <myprasanna-***@public.gmane.org>
> >> >> >>> >> wrote:
> >> >> >>> >>>
> >> >> >>> >>> Yes, indeed.
> >> >> >>> >>>
> >> >> >>> >>> Username: arnold-***@public.gmane.org
> >> >> >>> >>> Machine: mongo55
> >> >> >>> >>>
> >> >> >>> >>>
> >> >> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz
> >> >> >>> >>> <eliot-***@public.gmane.org>
> >> >> >>> >>> wrote:
> >> >> >>> >>>>
> >> >> >>> >>>> Are these machines in MMS?
> >> >> >>> >>>> Hard to see much in the pasted stats.
> >> >> >>> >>>>
> >> >> >>> >>>>
> >> >> >>> >>>>
> >> >> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
> >> >> >>> >>>> <myprasanna-***@public.gmane.org>
> >> >> >>> >>>> wrote:
> >> >> >>> >>>> > Raid 10 on 8 disks.
> >> >> >>> >>>> >
> >> >> >>> >>>> > MongoDB shell version: 2.0.4
> >> >> >>> >>>> > Tue May 8 09:02:29 [initandlisten] db version v2.0.4,
> >> >> >>> >>>> > pdfile
> >> >> >>> >>>> > version
> >> >> >>> >>>> > 4.5
> >> >> >>> >>>> >
> >> >> >>> >>>> >
> >> >> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
> >> >> >>> >>>> > <eliot-***@public.gmane.org>
> >> >> >>> >>>> > wrote:
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> What version of mongo and what kind of ebs raid?
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
> >> >> >>> >>>> >> <myprasanna-***@public.gmane.org>
> >> >> >>> >>>> >> wrote:
> >> >> >>> >>>> >> > And yes, disabling the balancer, brings the write rate
> >> >> >>> >>>> >> > back
> >> >> >>> >>>> >> > to
> >> >> >>> >>>> >> > zero.
> >> >> >>> >>>> >> > Although after 4 minutes!
> >> >> >>> >>>> >> >
> >> >> >>> >>>> >> >
> >> >> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> I don't think it goes into deadlock. Can you try
> >> >> >>> >>>> >> >> disabling
> >> >> >>> >>>> >> >> the
> >> >> >>> >>>> >> >> balancer
> >> >> >>> >>>> >> >> and see whether the disk write rate going down or not.
> >> >> >>> >>>> >> >> ________________________________
> >> >> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
> >> >> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
> >> >> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
> >> >> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
> >> >> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
> >> >> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>;
> >> >> >>> >>>> >> >> Hrushikesh
> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
> >> >> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> Hey guys,
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> We sharded mongo as per the docs. The issue is the
> >> >> >>> >>>> >> >> balancer
> >> >> >>> >>>> >> >> is
> >> >> >>> >>>> >> >> not
> >> >> >>> >>>> >> >> being
> >> >> >>> >>>> >> >> very clever and not throttling at all. Now we are
> stuck
> >> >> >>> >>>> >> >> with a
> >> >> >>> >>>> >> >> half-sharded
> >> >> >>> >>>> >> >> setup and we can't turn on the balancer because it'd
> >> >> >>> >>>> >> >> bring
> >> >> >>> >>>> >> >> the
> >> >> >>> >>>> >> >> site
> >> >> >>> >>>> >> >> down.
> >> >> >>> >>>> >> >> All read queries timeout when the disk write rate is
> >> >> >>> >>>> >> >> 100%.
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> Is there a way out of this mess at all? From a
> >> >> >>> >>>> >> >> high-level,
> >> >> >>> >>>> >> >> mongo
> >> >> >>> >>>> >> >> could
> >> >> >>> >>>> >> >> have handled this by any of the following means:
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
> >> >> >>> >>>> >> >> 2. Throttle and replicate only as long as the disk
> util
> >> >> >>> >>>> >> >> is
> >> >> >>> >>>> >> >> <
> >> >> >>> >>>> >> >> 50%
> >> >> >>> >>>> >> >> or so.
> >> >> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads and
> writes
> >> >> >>> >>>> >> >> come
> >> >> >>> >>>> >> >> in-between,
> >> >> >>> >>>> >> >> consider them as higher priority, so that a read/write
> >> >> >>> >>>> >> >> is
> >> >> >>> >>>> >> >> never
> >> >> >>> >>>> >> >> waiting
> >> >> >>> >>>> >> >> for
> >> >> >>> >>>> >> >> more than a second.
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> Have you guys heard of this problem before? Because
> this
> >> >> >>> >>>> >> >> really
> >> >> >>> >>>> >> >> make
> >> >> >>> >>>> >> >> sharding impossible. I'm not sure what others are
> doing
> >> >> >>> >>>> >> >> in
> >> >> >>> >>>> >> >> production
> >> >> >>> >>>> >> >> to
> >> >> >>> >>>> >> >> work around this.
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> Thanks,
> >> >> >>> >>>> >> >> -Prasanna
> >> >> >>> >>>> >> >> Co-Founder & CTO, lal.com
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> --
> >> >> >>> >>>> >> >> You received this message because you are subscribed
> to
> >> >> >>> >>>> >> >> the
> >> >> >>> >>>> >> >> Google
> >> >> >>> >>>> >> >> Groups
> >> >> >>> >>>> >> >> "mongodb-user" group.
> >> >> >>> >>>> >> >> To post to this group, send email to
> >> >> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> >>>> >> >> To unsubscribe from this group, send email to
> >> >> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> >>>> >> >> For more options, visit this group at
> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>> >>>> >> >
> >> >> >>> >>>> >> > --
> >> >> >>> >>>> >> > You received this message because you are subscribed to
> >> >> >>> >>>> >> > the
> >> >> >>> >>>> >> > Google
> >> >> >>> >>>> >> > Groups
> >> >> >>> >>>> >> > "mongodb-user" group.
> >> >> >>> >>>> >> > To view this discussion on the web visit
> >> >> >>> >>>> >> >
> >> >> >>> >>>> >> >
> https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
> >> >> >>> >>>> >> >
> >> >> >>> >>>> >> > To post to this group, send email to
> >> >> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> >>>> >> > To unsubscribe from this group, send email to
> >> >> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> >>>> >> > For more options, visit this group at
> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> --
> >> >> >>> >>>> >> You received this message because you are subscribed to
> the
> >> >> >>> >>>> >> Google
> >> >> >>> >>>> >> Groups
> >> >> >>> >>>> >> "mongodb-user" group.
> >> >> >>> >>>> >> To post to this group, send email to
> >> >> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> >>>> >> To unsubscribe from this group, send email to
> >> >> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> >>>> >> For more options, visit this group at
> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>> >>>> >>
> >> >> >>> >>>> >
> >> >> >>> >>>> > --
> >> >> >>> >>>> > You received this message because you are subscribed to
> the
> >> >> >>> >>>> > Google
> >> >> >>> >>>> > Groups
> >> >> >>> >>>> > "mongodb-user" group.
> >> >> >>> >>>> > To post to this group, send email to
> >> >> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> >>>> > To unsubscribe from this group, send email to
> >> >> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> >>>> > For more options, visit this group at
> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>> >>>>
> >> >> >>> >>>> --
> >> >> >>> >>>> You received this message because you are subscribed to the
> >> >> >>> >>>> Google
> >> >> >>> >>>> Groups "mongodb-user" group.
> >> >> >>> >>>> To post to this group, send email to
> >> >> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> >>>> To unsubscribe from this group, send email to
> >> >> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> >>>> For more options, visit this group at
> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>> >>>>
> >> >> >>> >>>
> >> >> >>> >>> --
> >> >> >>> >>> You received this message because you are subscribed to the
> >> >> >>> >>> Google
> >> >> >>> >>> Groups
> >> >> >>> >>> "mongodb-user" group.
> >> >> >>> >>> To post to this group, send email to
> >> >> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> >>> To unsubscribe from this group, send email to
> >> >> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> >>> For more options, visit this group at
> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>> >>
> >> >> >>> >>
> >> >> >>> >> --
> >> >> >>> >> You received this message because you are subscribed to the
> >> >> >>> >> Google
> >> >> >>> >> Groups
> >> >> >>> >> "mongodb-user" group.
> >> >> >>> >> To post to this group, send email to
> >> >> >>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> >> To unsubscribe from this group, send email to
> >> >> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> >> For more options, visit this group at
> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > --
> >> >> >>> > You received this message because you are subscribed to the
> >> >> >>> > Google
> >> >> >>> > Groups
> >> >> >>> > "mongodb-user" group.
> >> >> >>> > To post to this group, send email to
> >> >> >>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> > To unsubscribe from this group, send email to
> >> >> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> > For more options, visit this group at
> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>>
> >> >> >>> --
> >> >> >>> You received this message because you are subscribed to the
> Google
> >> >> >>> Groups
> >> >> >>> "mongodb-user" group.
> >> >> >>> To post to this group, send email to
> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> To unsubscribe from this group, send email to
> >> >> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >>> For more options, visit this group at
> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>>
> >> >> >>
> >> >> >
> >> >> > --
> >> >> > You received this message because you are subscribed to the Google
> >> >> > Groups
> >> >> > "mongodb-user" group.
> >> >> > To post to this group, send email to mongodb-user-/***@public.gmane.org
> .
> >> >> > To unsubscribe from this group, send email to
> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> > For more options, visit this group at
> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >>
> >> >> --
> >> >> You received this message because you are subscribed to the Google
> >> >> Groups
> >> >> "mongodb-user" group.
> >> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> To unsubscribe from this group, send email to
> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> For more options, visit this group at
> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >>
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups
> >> > "mongodb-user" group.
> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> > To unsubscribe from this group, send email to
> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> > For more options, visit this group at
> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "mongodb-user" group.
> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> To unsubscribe from this group, send email to
> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> For more options, visit this group at
> >> http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > To unsubscribe from this group, send email to
> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > For more options, visit this group at
> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-10 14:35:53 UTC
Permalink
Let me re-phrase that: When would someone ever rather have the site to
timeout to make progress on replication?

On Thu, May 10, 2012 at 7:33 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:

> Scott,
>
> > While it might be paramount for some situations/use-cases to complete all
> reads before a move starts, it might not be for others (which I
> have seen).
>
> Can you please explain the "might not be for others" part? Can you give me
> an example?
>
>
> On Thu, May 10, 2012 at 7:19 AM, Scott Hernandez <scotthernandez-***@public.gmane.org
> > wrote:
>
>> There are lots of ways to look at this with some competing goals.
>> While it might be paramount for some situations/use-cases to complete
>> all reads before a move starts, it might not be for others (which I
>> have seen). This fix will allow reads (and other writes) to get
>> processes while chunks moves happen, period. The issue of locking, and
>> contentions on those locks, is very important and there are lots of
>> improvements being made which help across the boards.
>>
>> Feel free to add comments or create new issues with your ideas of how
>> your system would like things to best behave:
>> http://jira.mongodb.org/browse/server
>>
>> On Wed, May 9, 2012 at 7:46 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>> > Scott,
>> >
>> > The below are heuristics, and not guarantees. What you want from a DB
>> > stand-point is the latter.
>> >
>> > The delay between moves, won't provide a guarantee, unless you make sure
>> > that
>> > (1) the next iteration of moveChunk, does not start unless all pending
>> reads
>> > complete.
>> > (2) Each iteration takes less than 1second.
>> >
>> > Thoughts?
>> >
>> >
>> > On Wed, May 9, 2012 at 6:18 PM, Scott Hernandez <
>> scotthernandez-***@public.gmane.org>
>> > wrote:
>> >>
>> >> The logic there is to do each document write, during the migrate, in a
>> >> separate lock so other operations can be interleaved. We have seen
>> >> this change dramatically reduce the constant lock percentage and allow
>> >> reads, and other writes through during a initial move.
>> >>
>> >> There is already a delay between moves which doesn't seem to be your
>> >> problem.
>> >>
>> >> On Wed, May 9, 2012 at 5:37 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>> wrote:
>> >> > Scott,
>> >> >
>> >> > I tried to understand what's going on there. But with a quick
>> glance, I
>> >> > assume, a lock was being held longer than necessary and was fixed.
>> It's
>> >> > unclear how this can help me. The only way to reliably make sure the
>> >> > site
>> >> > keeps running while a moveChunk is executing, is if, you yield every
>> >> > second
>> >> > within moveChunk and make sure all pending reads complete, before the
>> >> > next
>> >> > moveChunk period starts.
>> >> >
>> >> > Are you guys doing it, or planning to do it? Is there any other
>> >> > high-level
>> >> > way to guarantee the reads can be answered within a couple of
>> seconds?
>> >> >
>> >> > Thanks
>> >> >
>> >> >
>> >> > On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez
>> >> > <scotthernandez-***@public.gmane.org>
>> >> > wrote:
>> >> >>
>> >> >> Okay, looking at the receiving shard mongo55, and donor shard
>> mongo43,
>> >> >> it look like the fix in 2.0.6 (due out in the next two weeks of so)
>> >> >> will help you out a lot.
>> >> >>
>> >> >> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> >> wrote:
>> >> >> > 4:37 the balancer has started and disk shot up.
>> >> >> >
>> >> >> >
>> >> >> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S <myprasanna-***@public.gmane.org
>> >
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Hey Scott,
>> >> >> >>
>> >> >> >> I'm going to do this now. Will start it and stop it quickly and
>> give
>> >> >> >> you
>> >> >> >> the exact times.
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >> -Prasanna
>> >> >> >>
>> >> >> >>
>> >> >> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
>> >> >> >> <scotthernandez-***@public.gmane.org>
>> >> >> >> wrote:
>> >> >> >>>
>> >> >> >>> Those values are in blocks (512 bytes), not MB.
>> >> >> >>>
>> >> >> >>> We have an update in the upcoming 2.0.6 stable release which
>> will
>> >> >> >>> help
>> >> >> >>> with migrations a bit.
>> https://jira.mongodb.org/browse/SERVER-5303
>> >> >> >>> You
>> >> >> >>> can click on the version like to see other important fixes.
>> >> >> >>>
>> >> >> >>> Can you do a few migrates tonight so we can see the
>> stats/behavior
>> >> >> >>> in
>> >> >> >>> MMS? Please let us know when you do this so we can take a look
>> at
>> >> >> >>> the
>> >> >> >>> data to see if this fix will help.
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S <
>> myprasanna-***@public.gmane.org>
>> >> >> >>> wrote:
>> >> >> >>> > Hey Dan,
>> >> >> >>> >
>> >> >> >>> > Read ahead settings are the defaults (256
>> >> >> >>> > MB): http://pastebin.com/MHc8s2fq
>> >> >> >>> > Let me know if you need something else.
>> >> >> >>> >
>> >> >> >>> > @Elliot,
>> >> >> >>> >
>> >> >> >>> > I spoke with some of my friends at foursquare and it turns out
>> >> >> >>> > they
>> >> >> >>> > face the
>> >> >> >>> > same issue as well. I've had my other friends who run
>> companies >
>> >> >> >>> > $100M
>> >> >> >>> > market cap (whom I'd rather not name in this list), tell me
>> about
>> >> >> >>> > their
>> >> >> >>> > mongo problems in the past few days. This is pretty serious
>> stuff
>> >> >> >>> > and
>> >> >> >>> > will
>> >> >> >>> > affect if someone wants to choose mongo as their primary
>> store or
>> >> >> >>> > not.
>> >> >> >>> > To
>> >> >> >>> > not have sharding is one thing, but to promise it and being
>> stuck
>> >> >> >>> > in
>> >> >> >>> > production is another.
>> >> >> >>> >
>> >> >> >>> > I really thank you guys and appreciate the responsiveness so
>> far.
>> >> >> >>> > Happy
>> >> >> >>> > to
>> >> >> >>> > work together and hope we can resolve this.
>> >> >> >>> >
>> >> >> >>> > Thanks so much,
>> >> >> >>> > -Prasanna
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <dan-***@public.gmane.org>
>> >> >> >>> > wrote:
>> >> >> >>> >>
>> >> >> >>> >> Looks like the docs in that collection
>> >> >> >>> >> (lal_production.facebook_names)
>> >> >> >>> >> are
>> >> >> >>> >> small and likely distributed throughout the disk.
>> >> >> >>> >> Can you check your disk readahead setting?
>> >> >> >>> >> Please run and post the results of blockev --report.
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S
>> >> >> >>> >> <myprasanna-***@public.gmane.org>
>> >> >> >>> >> wrote:
>> >> >> >>> >>>
>> >> >> >>> >>> Yes, indeed.
>> >> >> >>> >>>
>> >> >> >>> >>> Username: arnold-***@public.gmane.org
>> >> >> >>> >>> Machine: mongo55
>> >> >> >>> >>>
>> >> >> >>> >>>
>> >> >> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz
>> >> >> >>> >>> <eliot-***@public.gmane.org>
>> >> >> >>> >>> wrote:
>> >> >> >>> >>>>
>> >> >> >>> >>>> Are these machines in MMS?
>> >> >> >>> >>>> Hard to see much in the pasted stats.
>> >> >> >>> >>>>
>> >> >> >>> >>>>
>> >> >> >>> >>>>
>> >> >> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
>> >> >> >>> >>>> <myprasanna-***@public.gmane.org>
>> >> >> >>> >>>> wrote:
>> >> >> >>> >>>> > Raid 10 on 8 disks.
>> >> >> >>> >>>> >
>> >> >> >>> >>>> > MongoDB shell version: 2.0.4
>> >> >> >>> >>>> > Tue May 8 09:02:29 [initandlisten] db version v2.0.4,
>> >> >> >>> >>>> > pdfile
>> >> >> >>> >>>> > version
>> >> >> >>> >>>> > 4.5
>> >> >> >>> >>>> >
>> >> >> >>> >>>> >
>> >> >> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
>> >> >> >>> >>>> > <eliot-***@public.gmane.org>
>> >> >> >>> >>>> > wrote:
>> >> >> >>> >>>> >>
>> >> >> >>> >>>> >> What version of mongo and what kind of ebs raid?
>> >> >> >>> >>>> >>
>> >> >> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
>> >> >> >>> >>>> >> <myprasanna-***@public.gmane.org>
>> >> >> >>> >>>> >> wrote:
>> >> >> >>> >>>> >> > And yes, disabling the balancer, brings the write rate
>> >> >> >>> >>>> >> > back
>> >> >> >>> >>>> >> > to
>> >> >> >>> >>>> >> > zero.
>> >> >> >>> >>>> >> > Although after 4 minutes!
>> >> >> >>> >>>> >> >
>> >> >> >>> >>>> >> >
>> >> >> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
>> >> >> >>> >>>> >> >>
>> >> >> >>> >>>> >> >> I don't think it goes into deadlock. Can you try
>> >> >> >>> >>>> >> >> disabling
>> >> >> >>> >>>> >> >> the
>> >> >> >>> >>>> >> >> balancer
>> >> >> >>> >>>> >> >> and see whether the disk write rate going down or
>> not.
>> >> >> >>> >>>> >> >> ________________________________
>> >> >> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>> >> >> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>> >> >> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>> >> >> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>> >> >> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>> >> >> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>;
>> >> >> >>> >>>> >> >> Hrushikesh
>> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>> >> >> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>> >> >> >>> >>>> >> >>
>> >> >> >>> >>>> >> >> Hey guys,
>> >> >> >>> >>>> >> >>
>> >> >> >>> >>>> >> >> We sharded mongo as per the docs. The issue is the
>> >> >> >>> >>>> >> >> balancer
>> >> >> >>> >>>> >> >> is
>> >> >> >>> >>>> >> >> not
>> >> >> >>> >>>> >> >> being
>> >> >> >>> >>>> >> >> very clever and not throttling at all. Now we are
>> stuck
>> >> >> >>> >>>> >> >> with a
>> >> >> >>> >>>> >> >> half-sharded
>> >> >> >>> >>>> >> >> setup and we can't turn on the balancer because it'd
>> >> >> >>> >>>> >> >> bring
>> >> >> >>> >>>> >> >> the
>> >> >> >>> >>>> >> >> site
>> >> >> >>> >>>> >> >> down.
>> >> >> >>> >>>> >> >> All read queries timeout when the disk write rate is
>> >> >> >>> >>>> >> >> 100%.
>> >> >> >>> >>>> >> >>
>> >> >> >>> >>>> >> >> Is there a way out of this mess at all? From a
>> >> >> >>> >>>> >> >> high-level,
>> >> >> >>> >>>> >> >> mongo
>> >> >> >>> >>>> >> >> could
>> >> >> >>> >>>> >> >> have handled this by any of the following means:
>> >> >> >>> >>>> >> >>
>> >> >> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
>> >> >> >>> >>>> >> >> 2. Throttle and replicate only as long as the disk
>> util
>> >> >> >>> >>>> >> >> is
>> >> >> >>> >>>> >> >> <
>> >> >> >>> >>>> >> >> 50%
>> >> >> >>> >>>> >> >> or so.
>> >> >> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads and
>> writes
>> >> >> >>> >>>> >> >> come
>> >> >> >>> >>>> >> >> in-between,
>> >> >> >>> >>>> >> >> consider them as higher priority, so that a
>> read/write
>> >> >> >>> >>>> >> >> is
>> >> >> >>> >>>> >> >> never
>> >> >> >>> >>>> >> >> waiting
>> >> >> >>> >>>> >> >> for
>> >> >> >>> >>>> >> >> more than a second.
>> >> >> >>> >>>> >> >>
>> >> >> >>> >>>> >> >> Have you guys heard of this problem before? Because
>> this
>> >> >> >>> >>>> >> >> really
>> >> >> >>> >>>> >> >> make
>> >> >> >>> >>>> >> >> sharding impossible. I'm not sure what others are
>> doing
>> >> >> >>> >>>> >> >> in
>> >> >> >>> >>>> >> >> production
>> >> >> >>> >>>> >> >> to
>> >> >> >>> >>>> >> >> work around this.
>> >> >> >>> >>>> >> >>
>> >> >> >>> >>>> >> >> Thanks,
>> >> >> >>> >>>> >> >> -Prasanna
>> >> >> >>> >>>> >> >> Co-Founder & CTO, lal.com
>> >> >> >>> >>>> >> >>
>> >> >> >>> >>>> >> >> --
>> >> >> >>> >>>> >> >> You received this message because you are subscribed
>> to
>> >> >> >>> >>>> >> >> the
>> >> >> >>> >>>> >> >> Google
>> >> >> >>> >>>> >> >> Groups
>> >> >> >>> >>>> >> >> "mongodb-user" group.
>> >> >> >>> >>>> >> >> To post to this group, send email to
>> >> >> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> >> >> To unsubscribe from this group, send email to
>> >> >> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> >> >> For more options, visit this group at
>> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>> >>>> >> >
>> >> >> >>> >>>> >> > --
>> >> >> >>> >>>> >> > You received this message because you are subscribed
>> to
>> >> >> >>> >>>> >> > the
>> >> >> >>> >>>> >> > Google
>> >> >> >>> >>>> >> > Groups
>> >> >> >>> >>>> >> > "mongodb-user" group.
>> >> >> >>> >>>> >> > To view this discussion on the web visit
>> >> >> >>> >>>> >> >
>> >> >> >>> >>>> >> >
>> https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >> >>> >>>> >> >
>> >> >> >>> >>>> >> > To post to this group, send email to
>> >> >> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> >> > To unsubscribe from this group, send email to
>> >> >> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> >> > For more options, visit this group at
>> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>> >>>> >>
>> >> >> >>> >>>> >> --
>> >> >> >>> >>>> >> You received this message because you are subscribed to
>> the
>> >> >> >>> >>>> >> Google
>> >> >> >>> >>>> >> Groups
>> >> >> >>> >>>> >> "mongodb-user" group.
>> >> >> >>> >>>> >> To post to this group, send email to
>> >> >> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> >> To unsubscribe from this group, send email to
>> >> >> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> >> For more options, visit this group at
>> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>> >>>> >>
>> >> >> >>> >>>> >
>> >> >> >>> >>>> > --
>> >> >> >>> >>>> > You received this message because you are subscribed to
>> the
>> >> >> >>> >>>> > Google
>> >> >> >>> >>>> > Groups
>> >> >> >>> >>>> > "mongodb-user" group.
>> >> >> >>> >>>> > To post to this group, send email to
>> >> >> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> > To unsubscribe from this group, send email to
>> >> >> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> > For more options, visit this group at
>> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>> >>>>
>> >> >> >>> >>>> --
>> >> >> >>> >>>> You received this message because you are subscribed to the
>> >> >> >>> >>>> Google
>> >> >> >>> >>>> Groups "mongodb-user" group.
>> >> >> >>> >>>> To post to this group, send email to
>> >> >> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> To unsubscribe from this group, send email to
>> >> >> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> For more options, visit this group at
>> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>> >>>>
>> >> >> >>> >>>
>> >> >> >>> >>> --
>> >> >> >>> >>> You received this message because you are subscribed to the
>> >> >> >>> >>> Google
>> >> >> >>> >>> Groups
>> >> >> >>> >>> "mongodb-user" group.
>> >> >> >>> >>> To post to this group, send email to
>> >> >> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>> To unsubscribe from this group, send email to
>> >> >> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>> For more options, visit this group at
>> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >> --
>> >> >> >>> >> You received this message because you are subscribed to the
>> >> >> >>> >> Google
>> >> >> >>> >> Groups
>> >> >> >>> >> "mongodb-user" group.
>> >> >> >>> >> To post to this group, send email to
>> >> >> >>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >> To unsubscribe from this group, send email to
>> >> >> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >> For more options, visit this group at
>> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>> > --
>> >> >> >>> > You received this message because you are subscribed to the
>> >> >> >>> > Google
>> >> >> >>> > Groups
>> >> >> >>> > "mongodb-user" group.
>> >> >> >>> > To post to this group, send email to
>> >> >> >>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> > To unsubscribe from this group, send email to
>> >> >> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> > For more options, visit this group at
>> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>>
>> >> >> >>> --
>> >> >> >>> You received this message because you are subscribed to the
>> Google
>> >> >> >>> Groups
>> >> >> >>> "mongodb-user" group.
>> >> >> >>> To post to this group, send email to
>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> To unsubscribe from this group, send email to
>> >> >> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> For more options, visit this group at
>> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>>
>> >> >> >>
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "mongodb-user" group.
>> >> >> > To post to this group, send email to
>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > To unsubscribe from this group, send email to
>> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "mongodb-user" group.
>> >> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> To unsubscribe from this group, send email to
>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > To unsubscribe from this group, send email to
>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> To unsubscribe from this group, send email to
>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > To unsubscribe from this group, send email to
>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Scott Hernandez
2012-05-10 14:37:50 UTC
Permalink
A simple example is a system where they want the balancer to run as
quickly as possible at low traffic times to balancer data and don't
care about the impact of (user) reads. This can be a big deal when you
are already under-provisioned and need to add a new shard to handle
users more effectively -- it has come up a few times.

On Thu, May 10, 2012 at 7:33 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
> Scott,
>
>> While it might be paramount for some situations/use-cases to complete all
>> reads before a move starts, it might not be for others (which I
> have seen).
>
> Can you please explain the "might not be for others" part? Can you give me
> an example?
>
>
> On Thu, May 10, 2012 at 7:19 AM, Scott Hernandez <scotthernandez-***@public.gmane.orgm>
> wrote:
>>
>> There are lots of ways to look at this with some competing goals.
>> While it might be paramount for some situations/use-cases to complete
>> all reads before a move starts, it might not be for others (which I
>> have seen). This fix will allow reads (and other writes) to get
>> processes while chunks moves happen, period. The issue of locking, and
>> contentions on those locks, is very important and there are lots of
>> improvements being made which help across the boards.
>>
>> Feel free to add comments or create new issues with your ideas of how
>> your system would like things to best behave:
>> http://jira.mongodb.org/browse/server
>>
>> On Wed, May 9, 2012 at 7:46 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>> > Scott,
>> >
>> > The below are heuristics, and not guarantees. What you want from a DB
>> > stand-point is the latter.
>> >
>> > The delay between moves, won't provide a guarantee, unless you make sure
>> > that
>> > (1) the next iteration of moveChunk, does not start unless all pending
>> > reads
>> > complete.
>> > (2) Each iteration takes less than 1second.
>> >
>> > Thoughts?
>> >
>> >
>> > On Wed, May 9, 2012 at 6:18 PM, Scott Hernandez
>> > <scotthernandez-***@public.gmane.org>
>> > wrote:
>> >>
>> >> The logic there is to do each document write, during the migrate, in a
>> >> separate lock so other operations can be interleaved. We have seen
>> >> this change dramatically reduce the constant lock percentage and allow
>> >> reads, and other writes through during a initial move.
>> >>
>> >> There is already a delay between moves which doesn't seem to be your
>> >> problem.
>> >>
>> >> On Wed, May 9, 2012 at 5:37 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> wrote:
>> >> > Scott,
>> >> >
>> >> > I tried to understand what's going on there. But with a quick glance,
>> >> > I
>> >> > assume, a lock was being held longer than necessary and was fixed.
>> >> > It's
>> >> > unclear how this can help me. The only way to reliably make sure the
>> >> > site
>> >> > keeps running while a moveChunk is executing, is if, you yield every
>> >> > second
>> >> > within moveChunk and make sure all pending reads complete, before the
>> >> > next
>> >> > moveChunk period starts.
>> >> >
>> >> > Are you guys doing it, or planning to do it? Is there any other
>> >> > high-level
>> >> > way to guarantee the reads can be answered within a couple of
>> >> > seconds?
>> >> >
>> >> > Thanks
>> >> >
>> >> >
>> >> > On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez
>> >> > <scotthernandez-***@public.gmane.org>
>> >> > wrote:
>> >> >>
>> >> >> Okay, looking at the receiving shard mongo55, and donor shard
>> >> >> mongo43,
>> >> >> it look like the fix in 2.0.6 (due out in the next two weeks of so)
>> >> >> will help you out a lot.
>> >> >>
>> >> >> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> >> wrote:
>> >> >> > 4:37 the balancer has started and disk shot up.
>> >> >> >
>> >> >> >
>> >> >> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S <myprasanna-***@public.gmane.orgm>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Hey Scott,
>> >> >> >>
>> >> >> >> I'm going to do this now. Will start it and stop it quickly and
>> >> >> >> give
>> >> >> >> you
>> >> >> >> the exact times.
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >> -Prasanna
>> >> >> >>
>> >> >> >>
>> >> >> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
>> >> >> >> <scotthernandez-***@public.gmane.org>
>> >> >> >> wrote:
>> >> >> >>>
>> >> >> >>> Those values are in blocks (512 bytes), not MB.
>> >> >> >>>
>> >> >> >>> We have an update in the upcoming 2.0.6 stable release which
>> >> >> >>> will
>> >> >> >>> help
>> >> >> >>> with migrations a bit.
>> >> >> >>> https://jira.mongodb.org/browse/SERVER-5303
>> >> >> >>> You
>> >> >> >>> can click on the version like to see other important fixes.
>> >> >> >>>
>> >> >> >>> Can you do a few migrates tonight so we can see the
>> >> >> >>> stats/behavior
>> >> >> >>> in
>> >> >> >>> MMS? Please let  us know when you do this so we can take a look
>> >> >> >>> at
>> >> >> >>> the
>> >> >> >>> data to see if this fix will help.
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S
>> >> >> >>> <myprasanna-***@public.gmane.org>
>> >> >> >>> wrote:
>> >> >> >>> > Hey Dan,
>> >> >> >>> >
>> >> >> >>> > Read ahead settings are the defaults (256
>> >> >> >>> > MB): http://pastebin.com/MHc8s2fq
>> >> >> >>> > Let me know if you need something else.
>> >> >> >>> >
>> >> >> >>> > @Elliot,
>> >> >> >>> >
>> >> >> >>> > I spoke with some of my friends at foursquare and it turns out
>> >> >> >>> > they
>> >> >> >>> > face the
>> >> >> >>> > same issue as well. I've had my other friends who run
>> >> >> >>> > companies >
>> >> >> >>> > $100M
>> >> >> >>> > market cap (whom I'd rather not name in this list), tell me
>> >> >> >>> > about
>> >> >> >>> > their
>> >> >> >>> > mongo problems in the past few days. This is pretty serious
>> >> >> >>> > stuff
>> >> >> >>> > and
>> >> >> >>> > will
>> >> >> >>> > affect if someone wants to choose mongo as their primary store
>> >> >> >>> > or
>> >> >> >>> > not.
>> >> >> >>> > To
>> >> >> >>> > not have sharding is one thing, but to promise it and being
>> >> >> >>> > stuck
>> >> >> >>> > in
>> >> >> >>> > production is another.
>> >> >> >>> >
>> >> >> >>> > I really thank you guys and appreciate the responsiveness so
>> >> >> >>> > far.
>> >> >> >>> > Happy
>> >> >> >>> > to
>> >> >> >>> > work together and hope we can resolve this.
>> >> >> >>> >
>> >> >> >>> > Thanks so much,
>> >> >> >>> > -Prasanna
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <dan-***@public.gmane.org>
>> >> >> >>> > wrote:
>> >> >> >>> >>
>> >> >> >>> >> Looks like the docs in that collection
>> >> >> >>> >> (lal_production.facebook_names)
>> >> >> >>> >> are
>> >> >> >>> >> small and likely distributed throughout the disk.
>> >> >> >>> >> Can you check your disk readahead setting?
>> >> >> >>> >> Please run and post the results of blockev --report.
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S
>> >> >> >>> >> <myprasanna-***@public.gmane.org>
>> >> >> >>> >> wrote:
>> >> >> >>> >>>
>> >> >> >>> >>> Yes, indeed.
>> >> >> >>> >>>
>> >> >> >>> >>> Username: arnold-***@public.gmane.org
>> >> >> >>> >>> Machine: mongo55
>> >> >> >>> >>>
>> >> >> >>> >>>
>> >> >> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz
>> >> >> >>> >>> <eliot-***@public.gmane.org>
>> >> >> >>> >>> wrote:
>> >> >> >>> >>>>
>> >> >> >>> >>>> Are these machines in MMS?
>> >> >> >>> >>>> Hard to see much in the pasted stats.
>> >> >> >>> >>>>
>> >> >> >>> >>>>
>> >> >> >>> >>>>
>> >> >> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
>> >> >> >>> >>>> <myprasanna-***@public.gmane.org>
>> >> >> >>> >>>> wrote:
>> >> >> >>> >>>> > Raid 10 on 8 disks.
>> >> >> >>> >>>> >
>> >> >> >>> >>>> > MongoDB shell version: 2.0.4
>> >> >> >>> >>>> > Tue May  8 09:02:29 [initandlisten] db version v2.0.4,
>> >> >> >>> >>>> > pdfile
>> >> >> >>> >>>> > version
>> >> >> >>> >>>> > 4.5
>> >> >> >>> >>>> >
>> >> >> >>> >>>> >
>> >> >> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
>> >> >> >>> >>>> > <eliot-***@public.gmane.org>
>> >> >> >>> >>>> > wrote:
>> >> >> >>> >>>> >>
>> >> >> >>> >>>> >> What version of mongo and what kind of ebs raid?
>> >> >> >>> >>>> >>
>> >> >> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
>> >> >> >>> >>>> >> <myprasanna-***@public.gmane.org>
>> >> >> >>> >>>> >> wrote:
>> >> >> >>> >>>> >> > And yes, disabling the balancer, brings the write rate
>> >> >> >>> >>>> >> > back
>> >> >> >>> >>>> >> > to
>> >> >> >>> >>>> >> > zero.
>> >> >> >>> >>>> >> > Although after 4 minutes!
>> >> >> >>> >>>> >> >
>> >> >> >>> >>>> >> >
>> >> >> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
>> >> >> >>> >>>> >> >>
>> >> >> >>> >>>> >> >> I don't think it goes into deadlock. Can you try
>> >> >> >>> >>>> >> >> disabling
>> >> >> >>> >>>> >> >> the
>> >> >> >>> >>>> >> >> balancer
>> >> >> >>> >>>> >> >> and see whether the disk write rate going down or
>> >> >> >>> >>>> >> >> not.
>> >> >> >>> >>>> >> >> ________________________________
>> >> >> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>> >> >> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>> >> >> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>> >> >> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>> >> >> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>> >> >> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>;
>> >> >> >>> >>>> >> >> Hrushikesh
>> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>> >> >> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>> >> >> >>> >>>> >> >>
>> >> >> >>> >>>> >> >> Hey guys,
>> >> >> >>> >>>> >> >>
>> >> >> >>> >>>> >> >> We sharded mongo as per the docs. The issue is the
>> >> >> >>> >>>> >> >> balancer
>> >> >> >>> >>>> >> >> is
>> >> >> >>> >>>> >> >> not
>> >> >> >>> >>>> >> >> being
>> >> >> >>> >>>> >> >> very clever and not throttling at all. Now we are
>> >> >> >>> >>>> >> >> stuck
>> >> >> >>> >>>> >> >> with a
>> >> >> >>> >>>> >> >> half-sharded
>> >> >> >>> >>>> >> >> setup and we can't turn on the balancer because it'd
>> >> >> >>> >>>> >> >> bring
>> >> >> >>> >>>> >> >> the
>> >> >> >>> >>>> >> >> site
>> >> >> >>> >>>> >> >> down.
>> >> >> >>> >>>> >> >> All read queries timeout when the disk write rate is
>> >> >> >>> >>>> >> >> 100%.
>> >> >> >>> >>>> >> >>
>> >> >> >>> >>>> >> >> Is there a way out of this mess at all? From a
>> >> >> >>> >>>> >> >> high-level,
>> >> >> >>> >>>> >> >> mongo
>> >> >> >>> >>>> >> >> could
>> >> >> >>> >>>> >> >> have handled this by any of the following means:
>> >> >> >>> >>>> >> >>
>> >> >> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
>> >> >> >>> >>>> >> >> 2. Throttle and replicate only as long as the disk
>> >> >> >>> >>>> >> >> util
>> >> >> >>> >>>> >> >> is
>> >> >> >>> >>>> >> >> <
>> >> >> >>> >>>> >> >> 50%
>> >> >> >>> >>>> >> >> or so.
>> >> >> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads and
>> >> >> >>> >>>> >> >> writes
>> >> >> >>> >>>> >> >> come
>> >> >> >>> >>>> >> >> in-between,
>> >> >> >>> >>>> >> >> consider them as higher priority, so that a
>> >> >> >>> >>>> >> >> read/write
>> >> >> >>> >>>> >> >> is
>> >> >> >>> >>>> >> >> never
>> >> >> >>> >>>> >> >> waiting
>> >> >> >>> >>>> >> >> for
>> >> >> >>> >>>> >> >> more than a second.
>> >> >> >>> >>>> >> >>
>> >> >> >>> >>>> >> >> Have you guys heard of this problem before? Because
>> >> >> >>> >>>> >> >> this
>> >> >> >>> >>>> >> >> really
>> >> >> >>> >>>> >> >> make
>> >> >> >>> >>>> >> >> sharding impossible. I'm not sure what others are
>> >> >> >>> >>>> >> >> doing
>> >> >> >>> >>>> >> >> in
>> >> >> >>> >>>> >> >> production
>> >> >> >>> >>>> >> >> to
>> >> >> >>> >>>> >> >> work around this.
>> >> >> >>> >>>> >> >>
>> >> >> >>> >>>> >> >> Thanks,
>> >> >> >>> >>>> >> >> -Prasanna
>> >> >> >>> >>>> >> >> Co-Founder & CTO, lal.com
>> >> >> >>> >>>> >> >>
>> >> >> >>> >>>> >> >> --
>> >> >> >>> >>>> >> >> You received this message because you are subscribed
>> >> >> >>> >>>> >> >> to
>> >> >> >>> >>>> >> >> the
>> >> >> >>> >>>> >> >> Google
>> >> >> >>> >>>> >> >> Groups
>> >> >> >>> >>>> >> >> "mongodb-user" group.
>> >> >> >>> >>>> >> >> To post to this group, send email to
>> >> >> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> >> >> To unsubscribe from this group, send email to
>> >> >> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> >> >> For more options, visit this group at
>> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>> >>>> >> >
>> >> >> >>> >>>> >> > --
>> >> >> >>> >>>> >> > You received this message because you are subscribed
>> >> >> >>> >>>> >> > to
>> >> >> >>> >>>> >> > the
>> >> >> >>> >>>> >> > Google
>> >> >> >>> >>>> >> > Groups
>> >> >> >>> >>>> >> > "mongodb-user" group.
>> >> >> >>> >>>> >> > To view this discussion on the web visit
>> >> >> >>> >>>> >> >
>> >> >> >>> >>>> >> >
>> >> >> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >> >>> >>>> >> >
>> >> >> >>> >>>> >> > To post to this group, send email to
>> >> >> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> >> > To unsubscribe from this group, send email to
>> >> >> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> >> > For more options, visit this group at
>> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>> >>>> >>
>> >> >> >>> >>>> >> --
>> >> >> >>> >>>> >> You received this message because you are subscribed to
>> >> >> >>> >>>> >> the
>> >> >> >>> >>>> >> Google
>> >> >> >>> >>>> >> Groups
>> >> >> >>> >>>> >> "mongodb-user" group.
>> >> >> >>> >>>> >> To post to this group, send email to
>> >> >> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> >> To unsubscribe from this group, send email to
>> >> >> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> >> For more options, visit this group at
>> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>> >>>> >>
>> >> >> >>> >>>> >
>> >> >> >>> >>>> > --
>> >> >> >>> >>>> > You received this message because you are subscribed to
>> >> >> >>> >>>> > the
>> >> >> >>> >>>> > Google
>> >> >> >>> >>>> > Groups
>> >> >> >>> >>>> > "mongodb-user" group.
>> >> >> >>> >>>> > To post to this group, send email to
>> >> >> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> > To unsubscribe from this group, send email to
>> >> >> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> > For more options, visit this group at
>> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>> >>>>
>> >> >> >>> >>>> --
>> >> >> >>> >>>> You received this message because you are subscribed to the
>> >> >> >>> >>>> Google
>> >> >> >>> >>>> Groups "mongodb-user" group.
>> >> >> >>> >>>> To post to this group, send email to
>> >> >> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> To unsubscribe from this group, send email to
>> >> >> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>>> For more options, visit this group at
>> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>> >>>>
>> >> >> >>> >>>
>> >> >> >>> >>> --
>> >> >> >>> >>> You received this message because you are subscribed to the
>> >> >> >>> >>> Google
>> >> >> >>> >>> Groups
>> >> >> >>> >>> "mongodb-user" group.
>> >> >> >>> >>> To post to this group, send email to
>> >> >> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>> To unsubscribe from this group, send email to
>> >> >> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >>> For more options, visit this group at
>> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>> >>
>> >> >> >>> >>
>> >> >> >>> >> --
>> >> >> >>> >> You received this message because you are subscribed to the
>> >> >> >>> >> Google
>> >> >> >>> >> Groups
>> >> >> >>> >> "mongodb-user" group.
>> >> >> >>> >> To post to this group, send email to
>> >> >> >>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >> To unsubscribe from this group, send email to
>> >> >> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> >> For more options, visit this group at
>> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>> > --
>> >> >> >>> > You received this message because you are subscribed to the
>> >> >> >>> > Google
>> >> >> >>> > Groups
>> >> >> >>> > "mongodb-user" group.
>> >> >> >>> > To post to this group, send email to
>> >> >> >>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> > To unsubscribe from this group, send email to
>> >> >> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> > For more options, visit this group at
>> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>>
>> >> >> >>> --
>> >> >> >>> You received this message because you are subscribed to the
>> >> >> >>> Google
>> >> >> >>> Groups
>> >> >> >>> "mongodb-user" group.
>> >> >> >>> To post to this group, send email to
>> >> >> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> To unsubscribe from this group, send email to
>> >> >> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >>> For more options, visit this group at
>> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>>
>> >> >> >>
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "mongodb-user" group.
>> >> >> > To post to this group, send email to
>> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > To unsubscribe from this group, send email to
>> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "mongodb-user" group.
>> >> >> To post to this group, send email to mongodb-user-/***@public.gmane.org.
>> >> >> To unsubscribe from this group, send email to
>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > To unsubscribe from this group, send email to
>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> To unsubscribe from this group, send email to
>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > To unsubscribe from this group, send email to
>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-10 14:44:04 UTC
Permalink
That use case can easily be supported by just shutting off the reads from
the application. Basically they are anyways taking downtime -- that doesn't
take special support, does it?

In the normal case, don't you think the reads are the highest priority?
Come on, this is production.. And I'm telling you, your top clients have
this problem. Whats worse for a DB than downtime?

On Thu, May 10, 2012 at 7:37 AM, Scott Hernandez
<scotthernandez-***@public.gmane.org>wrote:

> A simple example is a system where they want the balancer to run as
> quickly as possible at low traffic times to balancer data and don't
> care about the impact of (user) reads. This can be a big deal when you
> are already under-provisioned and need to add a new shard to handle
> users more effectively -- it has come up a few times.
>
> On Thu, May 10, 2012 at 7:33 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
> > Scott,
> >
> >> While it might be paramount for some situations/use-cases to
> complete all
> >> reads before a move starts, it might not be for others (which I
> > have seen).
> >
> > Can you please explain the "might not be for others" part? Can you give
> me
> > an example?
> >
> >
> > On Thu, May 10, 2012 at 7:19 AM, Scott Hernandez <
> scotthernandez-***@public.gmane.org>
> > wrote:
> >>
> >> There are lots of ways to look at this with some competing goals.
> >> While it might be paramount for some situations/use-cases to complete
> >> all reads before a move starts, it might not be for others (which I
> >> have seen). This fix will allow reads (and other writes) to get
> >> processes while chunks moves happen, period. The issue of locking, and
> >> contentions on those locks, is very important and there are lots of
> >> improvements being made which help across the boards.
> >>
> >> Feel free to add comments or create new issues with your ideas of how
> >> your system would like things to best behave:
> >> http://jira.mongodb.org/browse/server
> >>
> >> On Wed, May 9, 2012 at 7:46 PM, Prasanna .S <myprasanna-***@public.gmane.org>
> wrote:
> >> > Scott,
> >> >
> >> > The below are heuristics, and not guarantees. What you want from a DB
> >> > stand-point is the latter.
> >> >
> >> > The delay between moves, won't provide a guarantee, unless you make
> sure
> >> > that
> >> > (1) the next iteration of moveChunk, does not start unless all pending
> >> > reads
> >> > complete.
> >> > (2) Each iteration takes less than 1second.
> >> >
> >> > Thoughts?
> >> >
> >> >
> >> > On Wed, May 9, 2012 at 6:18 PM, Scott Hernandez
> >> > <scotthernandez-***@public.gmane.org>
> >> > wrote:
> >> >>
> >> >> The logic there is to do each document write, during the migrate, in
> a
> >> >> separate lock so other operations can be interleaved. We have seen
> >> >> this change dramatically reduce the constant lock percentage and
> allow
> >> >> reads, and other writes through during a initial move.
> >> >>
> >> >> There is already a delay between moves which doesn't seem to be your
> >> >> problem.
> >> >>
> >> >> On Wed, May 9, 2012 at 5:37 PM, Prasanna .S <myprasanna-***@public.gmane.org>
> >> >> wrote:
> >> >> > Scott,
> >> >> >
> >> >> > I tried to understand what's going on there. But with a quick
> glance,
> >> >> > I
> >> >> > assume, a lock was being held longer than necessary and was fixed.
> >> >> > It's
> >> >> > unclear how this can help me. The only way to reliably make sure
> the
> >> >> > site
> >> >> > keeps running while a moveChunk is executing, is if, you yield
> every
> >> >> > second
> >> >> > within moveChunk and make sure all pending reads complete, before
> the
> >> >> > next
> >> >> > moveChunk period starts.
> >> >> >
> >> >> > Are you guys doing it, or planning to do it? Is there any other
> >> >> > high-level
> >> >> > way to guarantee the reads can be answered within a couple of
> >> >> > seconds?
> >> >> >
> >> >> > Thanks
> >> >> >
> >> >> >
> >> >> > On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez
> >> >> > <scotthernandez-***@public.gmane.org>
> >> >> > wrote:
> >> >> >>
> >> >> >> Okay, looking at the receiving shard mongo55, and donor shard
> >> >> >> mongo43,
> >> >> >> it look like the fix in 2.0.6 (due out in the next two weeks of
> so)
> >> >> >> will help you out a lot.
> >> >> >>
> >> >> >> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S <myprasanna-***@public.gmane.org
> >
> >> >> >> wrote:
> >> >> >> > 4:37 the balancer has started and disk shot up.
> >> >> >> >
> >> >> >> >
> >> >> >> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S <
> myprasanna-***@public.gmane.org>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> Hey Scott,
> >> >> >> >>
> >> >> >> >> I'm going to do this now. Will start it and stop it quickly and
> >> >> >> >> give
> >> >> >> >> you
> >> >> >> >> the exact times.
> >> >> >> >>
> >> >> >> >> Thanks,
> >> >> >> >> -Prasanna
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
> >> >> >> >> <scotthernandez-***@public.gmane.org>
> >> >> >> >> wrote:
> >> >> >> >>>
> >> >> >> >>> Those values are in blocks (512 bytes), not MB.
> >> >> >> >>>
> >> >> >> >>> We have an update in the upcoming 2.0.6 stable release which
> >> >> >> >>> will
> >> >> >> >>> help
> >> >> >> >>> with migrations a bit.
> >> >> >> >>> https://jira.mongodb.org/browse/SERVER-5303
> >> >> >> >>> You
> >> >> >> >>> can click on the version like to see other important fixes.
> >> >> >> >>>
> >> >> >> >>> Can you do a few migrates tonight so we can see the
> >> >> >> >>> stats/behavior
> >> >> >> >>> in
> >> >> >> >>> MMS? Please let us know when you do this so we can take a
> look
> >> >> >> >>> at
> >> >> >> >>> the
> >> >> >> >>> data to see if this fix will help.
> >> >> >> >>>
> >> >> >> >>>
> >> >> >> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S
> >> >> >> >>> <myprasanna-***@public.gmane.org>
> >> >> >> >>> wrote:
> >> >> >> >>> > Hey Dan,
> >> >> >> >>> >
> >> >> >> >>> > Read ahead settings are the defaults (256
> >> >> >> >>> > MB): http://pastebin.com/MHc8s2fq
> >> >> >> >>> > Let me know if you need something else.
> >> >> >> >>> >
> >> >> >> >>> > @Elliot,
> >> >> >> >>> >
> >> >> >> >>> > I spoke with some of my friends at foursquare and it turns
> out
> >> >> >> >>> > they
> >> >> >> >>> > face the
> >> >> >> >>> > same issue as well. I've had my other friends who run
> >> >> >> >>> > companies >
> >> >> >> >>> > $100M
> >> >> >> >>> > market cap (whom I'd rather not name in this list), tell me
> >> >> >> >>> > about
> >> >> >> >>> > their
> >> >> >> >>> > mongo problems in the past few days. This is pretty serious
> >> >> >> >>> > stuff
> >> >> >> >>> > and
> >> >> >> >>> > will
> >> >> >> >>> > affect if someone wants to choose mongo as their primary
> store
> >> >> >> >>> > or
> >> >> >> >>> > not.
> >> >> >> >>> > To
> >> >> >> >>> > not have sharding is one thing, but to promise it and being
> >> >> >> >>> > stuck
> >> >> >> >>> > in
> >> >> >> >>> > production is another.
> >> >> >> >>> >
> >> >> >> >>> > I really thank you guys and appreciate the responsiveness so
> >> >> >> >>> > far.
> >> >> >> >>> > Happy
> >> >> >> >>> > to
> >> >> >> >>> > work together and hope we can resolve this.
> >> >> >> >>> >
> >> >> >> >>> > Thanks so much,
> >> >> >> >>> > -Prasanna
> >> >> >> >>> >
> >> >> >> >>> >
> >> >> >> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <dan-***@public.gmane.org>
> >> >> >> >>> > wrote:
> >> >> >> >>> >>
> >> >> >> >>> >> Looks like the docs in that collection
> >> >> >> >>> >> (lal_production.facebook_names)
> >> >> >> >>> >> are
> >> >> >> >>> >> small and likely distributed throughout the disk.
> >> >> >> >>> >> Can you check your disk readahead setting?
> >> >> >> >>> >> Please run and post the results of blockev --report.
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S
> >> >> >> >>> >> <myprasanna-***@public.gmane.org>
> >> >> >> >>> >> wrote:
> >> >> >> >>> >>>
> >> >> >> >>> >>> Yes, indeed.
> >> >> >> >>> >>>
> >> >> >> >>> >>> Username: arnold-***@public.gmane.org
> >> >> >> >>> >>> Machine: mongo55
> >> >> >> >>> >>>
> >> >> >> >>> >>>
> >> >> >> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz
> >> >> >> >>> >>> <eliot-***@public.gmane.org>
> >> >> >> >>> >>> wrote:
> >> >> >> >>> >>>>
> >> >> >> >>> >>>> Are these machines in MMS?
> >> >> >> >>> >>>> Hard to see much in the pasted stats.
> >> >> >> >>> >>>>
> >> >> >> >>> >>>>
> >> >> >> >>> >>>>
> >> >> >> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
> >> >> >> >>> >>>> <myprasanna-***@public.gmane.org>
> >> >> >> >>> >>>> wrote:
> >> >> >> >>> >>>> > Raid 10 on 8 disks.
> >> >> >> >>> >>>> >
> >> >> >> >>> >>>> > MongoDB shell version: 2.0.4
> >> >> >> >>> >>>> > Tue May 8 09:02:29 [initandlisten] db version v2.0.4,
> >> >> >> >>> >>>> > pdfile
> >> >> >> >>> >>>> > version
> >> >> >> >>> >>>> > 4.5
> >> >> >> >>> >>>> >
> >> >> >> >>> >>>> >
> >> >> >> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
> >> >> >> >>> >>>> > <eliot-***@public.gmane.org>
> >> >> >> >>> >>>> > wrote:
> >> >> >> >>> >>>> >>
> >> >> >> >>> >>>> >> What version of mongo and what kind of ebs raid?
> >> >> >> >>> >>>> >>
> >> >> >> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
> >> >> >> >>> >>>> >> <myprasanna-***@public.gmane.org>
> >> >> >> >>> >>>> >> wrote:
> >> >> >> >>> >>>> >> > And yes, disabling the balancer, brings the write
> rate
> >> >> >> >>> >>>> >> > back
> >> >> >> >>> >>>> >> > to
> >> >> >> >>> >>>> >> > zero.
> >> >> >> >>> >>>> >> > Although after 4 minutes!
> >> >> >> >>> >>>> >> >
> >> >> >> >>> >>>> >> >
> >> >> >> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat wrote:
> >> >> >> >>> >>>> >> >>
> >> >> >> >>> >>>> >> >> I don't think it goes into deadlock. Can you try
> >> >> >> >>> >>>> >> >> disabling
> >> >> >> >>> >>>> >> >> the
> >> >> >> >>> >>>> >> >> balancer
> >> >> >> >>> >>>> >> >> and see whether the disk write rate going down or
> >> >> >> >>> >>>> >> >> not.
> >> >> >> >>> >>>> >> >> ________________________________
> >> >> >> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
> >> >> >> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
> >> >> >> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
> >> >> >> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
> >> >> >> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
> >> >> >> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>;
> >> >> >> >>> >>>> >> >> Hrushikesh
> >> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
> >> >> >> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
> >> >> >> >>> >>>> >> >>
> >> >> >> >>> >>>> >> >> Hey guys,
> >> >> >> >>> >>>> >> >>
> >> >> >> >>> >>>> >> >> We sharded mongo as per the docs. The issue is the
> >> >> >> >>> >>>> >> >> balancer
> >> >> >> >>> >>>> >> >> is
> >> >> >> >>> >>>> >> >> not
> >> >> >> >>> >>>> >> >> being
> >> >> >> >>> >>>> >> >> very clever and not throttling at all. Now we are
> >> >> >> >>> >>>> >> >> stuck
> >> >> >> >>> >>>> >> >> with a
> >> >> >> >>> >>>> >> >> half-sharded
> >> >> >> >>> >>>> >> >> setup and we can't turn on the balancer because
> it'd
> >> >> >> >>> >>>> >> >> bring
> >> >> >> >>> >>>> >> >> the
> >> >> >> >>> >>>> >> >> site
> >> >> >> >>> >>>> >> >> down.
> >> >> >> >>> >>>> >> >> All read queries timeout when the disk write rate
> is
> >> >> >> >>> >>>> >> >> 100%.
> >> >> >> >>> >>>> >> >>
> >> >> >> >>> >>>> >> >> Is there a way out of this mess at all? From a
> >> >> >> >>> >>>> >> >> high-level,
> >> >> >> >>> >>>> >> >> mongo
> >> >> >> >>> >>>> >> >> could
> >> >> >> >>> >>>> >> >> have handled this by any of the following means:
> >> >> >> >>> >>>> >> >>
> >> >> >> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
> >> >> >> >>> >>>> >> >> 2. Throttle and replicate only as long as the disk
> >> >> >> >>> >>>> >> >> util
> >> >> >> >>> >>>> >> >> is
> >> >> >> >>> >>>> >> >> <
> >> >> >> >>> >>>> >> >> 50%
> >> >> >> >>> >>>> >> >> or so.
> >> >> >> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads and
> >> >> >> >>> >>>> >> >> writes
> >> >> >> >>> >>>> >> >> come
> >> >> >> >>> >>>> >> >> in-between,
> >> >> >> >>> >>>> >> >> consider them as higher priority, so that a
> >> >> >> >>> >>>> >> >> read/write
> >> >> >> >>> >>>> >> >> is
> >> >> >> >>> >>>> >> >> never
> >> >> >> >>> >>>> >> >> waiting
> >> >> >> >>> >>>> >> >> for
> >> >> >> >>> >>>> >> >> more than a second.
> >> >> >> >>> >>>> >> >>
> >> >> >> >>> >>>> >> >> Have you guys heard of this problem before? Because
> >> >> >> >>> >>>> >> >> this
> >> >> >> >>> >>>> >> >> really
> >> >> >> >>> >>>> >> >> make
> >> >> >> >>> >>>> >> >> sharding impossible. I'm not sure what others are
> >> >> >> >>> >>>> >> >> doing
> >> >> >> >>> >>>> >> >> in
> >> >> >> >>> >>>> >> >> production
> >> >> >> >>> >>>> >> >> to
> >> >> >> >>> >>>> >> >> work around this.
> >> >> >> >>> >>>> >> >>
> >> >> >> >>> >>>> >> >> Thanks,
> >> >> >> >>> >>>> >> >> -Prasanna
> >> >> >> >>> >>>> >> >> Co-Founder & CTO, lal.com
> >> >> >> >>> >>>> >> >>
> >> >> >> >>> >>>> >> >> --
> >> >> >> >>> >>>> >> >> You received this message because you are
> subscribed
> >> >> >> >>> >>>> >> >> to
> >> >> >> >>> >>>> >> >> the
> >> >> >> >>> >>>> >> >> Google
> >> >> >> >>> >>>> >> >> Groups
> >> >> >> >>> >>>> >> >> "mongodb-user" group.
> >> >> >> >>> >>>> >> >> To post to this group, send email to
> >> >> >> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> >>>> >> >> To unsubscribe from this group, send email to
> >> >> >> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> >>>> >> >> For more options, visit this group at
> >> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >>> >>>> >> >
> >> >> >> >>> >>>> >> > --
> >> >> >> >>> >>>> >> > You received this message because you are subscribed
> >> >> >> >>> >>>> >> > to
> >> >> >> >>> >>>> >> > the
> >> >> >> >>> >>>> >> > Google
> >> >> >> >>> >>>> >> > Groups
> >> >> >> >>> >>>> >> > "mongodb-user" group.
> >> >> >> >>> >>>> >> > To view this discussion on the web visit
> >> >> >> >>> >>>> >> >
> >> >> >> >>> >>>> >> >
> >> >> >> >>> >>>> >> >
> https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
> >> >> >> >>> >>>> >> >
> >> >> >> >>> >>>> >> > To post to this group, send email to
> >> >> >> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to
> >> >> >> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> >>>> >> > For more options, visit this group at
> >> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >>> >>>> >>
> >> >> >> >>> >>>> >> --
> >> >> >> >>> >>>> >> You received this message because you are subscribed
> to
> >> >> >> >>> >>>> >> the
> >> >> >> >>> >>>> >> Google
> >> >> >> >>> >>>> >> Groups
> >> >> >> >>> >>>> >> "mongodb-user" group.
> >> >> >> >>> >>>> >> To post to this group, send email to
> >> >> >> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to
> >> >> >> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> >>>> >> For more options, visit this group at
> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >>> >>>> >>
> >> >> >> >>> >>>> >
> >> >> >> >>> >>>> > --
> >> >> >> >>> >>>> > You received this message because you are subscribed to
> >> >> >> >>> >>>> > the
> >> >> >> >>> >>>> > Google
> >> >> >> >>> >>>> > Groups
> >> >> >> >>> >>>> > "mongodb-user" group.
> >> >> >> >>> >>>> > To post to this group, send email to
> >> >> >> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> >>>> > To unsubscribe from this group, send email to
> >> >> >> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> >>>> > For more options, visit this group at
> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >>> >>>>
> >> >> >> >>> >>>> --
> >> >> >> >>> >>>> You received this message because you are subscribed to
> the
> >> >> >> >>> >>>> Google
> >> >> >> >>> >>>> Groups "mongodb-user" group.
> >> >> >> >>> >>>> To post to this group, send email to
> >> >> >> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> >>>> To unsubscribe from this group, send email to
> >> >> >> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> >>>> For more options, visit this group at
> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >>> >>>>
> >> >> >> >>> >>>
> >> >> >> >>> >>> --
> >> >> >> >>> >>> You received this message because you are subscribed to
> the
> >> >> >> >>> >>> Google
> >> >> >> >>> >>> Groups
> >> >> >> >>> >>> "mongodb-user" group.
> >> >> >> >>> >>> To post to this group, send email to
> >> >> >> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> >>> To unsubscribe from this group, send email to
> >> >> >> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> >>> For more options, visit this group at
> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >>> >>
> >> >> >> >>> >>
> >> >> >> >>> >> --
> >> >> >> >>> >> You received this message because you are subscribed to the
> >> >> >> >>> >> Google
> >> >> >> >>> >> Groups
> >> >> >> >>> >> "mongodb-user" group.
> >> >> >> >>> >> To post to this group, send email to
> >> >> >> >>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> >> To unsubscribe from this group, send email to
> >> >> >> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> >> For more options, visit this group at
> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >>> >
> >> >> >> >>> >
> >> >> >> >>> > --
> >> >> >> >>> > You received this message because you are subscribed to the
> >> >> >> >>> > Google
> >> >> >> >>> > Groups
> >> >> >> >>> > "mongodb-user" group.
> >> >> >> >>> > To post to this group, send email to
> >> >> >> >>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> > To unsubscribe from this group, send email to
> >> >> >> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> > For more options, visit this group at
> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >>>
> >> >> >> >>> --
> >> >> >> >>> You received this message because you are subscribed to the
> >> >> >> >>> Google
> >> >> >> >>> Groups
> >> >> >> >>> "mongodb-user" group.
> >> >> >> >>> To post to this group, send email to
> >> >> >> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> To unsubscribe from this group, send email to
> >> >> >> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >>> For more options, visit this group at
> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >>>
> >> >> >> >>
> >> >> >> >
> >> >> >> > --
> >> >> >> > You received this message because you are subscribed to the
> Google
> >> >> >> > Groups
> >> >> >> > "mongodb-user" group.
> >> >> >> > To post to this group, send email to
> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> > To unsubscribe from this group, send email to
> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> > For more options, visit this group at
> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>
> >> >> >> --
> >> >> >> You received this message because you are subscribed to the Google
> >> >> >> Groups
> >> >> >> "mongodb-user" group.
> >> >> >> To post to this group, send email to
> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> To unsubscribe from this group, send email to
> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> For more options, visit this group at
> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>
> >> >> >
> >> >> > --
> >> >> > You received this message because you are subscribed to the Google
> >> >> > Groups
> >> >> > "mongodb-user" group.
> >> >> > To post to this group, send email to mongodb-user-/***@public.gmane.org
> .
> >> >> > To unsubscribe from this group, send email to
> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> > For more options, visit this group at
> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >>
> >> >> --
> >> >> You received this message because you are subscribed to the Google
> >> >> Groups
> >> >> "mongodb-user" group.
> >> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> To unsubscribe from this group, send email to
> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> For more options, visit this group at
> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >>
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups
> >> > "mongodb-user" group.
> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> > To unsubscribe from this group, send email to
> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> > For more options, visit this group at
> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "mongodb-user" group.
> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> To unsubscribe from this group, send email to
> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> For more options, visit this group at
> >> http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > To unsubscribe from this group, send email to
> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > For more options, visit this group at
> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Scott Hernandez
2012-05-10 15:04:33 UTC
Permalink
You seem to looking at this as a binary result. Slow reads do not mean
downtime. There are many use-cases where many times performance
degradation during maintenance windows is acceptable but downtime
(stopping reads) is not.

We are working with everyone to make sure they can get the best
performance and uptime for their systems. If you have more specific
areas where you are seeing issue, please create a new thread/post or
jira issue with the problem (or vote/watch for existing ones).

Nobody thinks the system is perfect as is (how could it be when there
are so many different use cases), and there are lots of improvements,
fixes and changes going in to address these areas for improvement.


On Thu, May 10, 2012 at 7:44 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
> That use case can easily be supported by just shutting off the reads from
> the application. Basically they are anyways taking downtime -- that doesn't
> take special support, does it?
>
> In the normal case, don't you think the reads are the highest priority? Come
> on, this is production.. And I'm telling you, your top clients have this
> problem. Whats worse for a DB than downtime?
>
>
> On Thu, May 10, 2012 at 7:37 AM, Scott Hernandez <scotthernandez-***@public.gmane.orgm>
> wrote:
>>
>> A simple example is a system where they want the balancer to run as
>> quickly as possible at low traffic times to balancer data and don't
>> care about the impact of (user) reads. This can be a big deal when you
>> are already under-provisioned and need to add a new shard to handle
>> users more effectively -- it has come up a few times.
>>
>> On Thu, May 10, 2012 at 7:33 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>> > Scott,
>> >
>> >> While it might be paramount for some situations/use-cases to
>> >> complete all
>> >> reads before a move starts, it might not be for others (which I
>> > have seen).
>> >
>> > Can you please explain the "might not be for others" part? Can you give
>> > me
>> > an example?
>> >
>> >
>> > On Thu, May 10, 2012 at 7:19 AM, Scott Hernandez
>> > <scotthernandez-***@public.gmane.org>
>> > wrote:
>> >>
>> >> There are lots of ways to look at this with some competing goals.
>> >> While it might be paramount for some situations/use-cases to complete
>> >> all reads before a move starts, it might not be for others (which I
>> >> have seen). This fix will allow reads (and other writes) to get
>> >> processes while chunks moves happen, period. The issue of locking, and
>> >> contentions on those locks, is very important and there are lots of
>> >> improvements being made which help across the boards.
>> >>
>> >> Feel free to add comments or create new issues with your ideas of how
>> >> your system would like things to best behave:
>> >> http://jira.mongodb.org/browse/server
>> >>
>> >> On Wed, May 9, 2012 at 7:46 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> wrote:
>> >> > Scott,
>> >> >
>> >> > The below are heuristics, and not guarantees. What you want from a DB
>> >> > stand-point is the latter.
>> >> >
>> >> > The delay between moves, won't provide a guarantee, unless you make
>> >> > sure
>> >> > that
>> >> > (1) the next iteration of moveChunk, does not start unless all
>> >> > pending
>> >> > reads
>> >> > complete.
>> >> > (2) Each iteration takes less than 1second.
>> >> >
>> >> > Thoughts?
>> >> >
>> >> >
>> >> > On Wed, May 9, 2012 at 6:18 PM, Scott Hernandez
>> >> > <scotthernandez-***@public.gmane.org>
>> >> > wrote:
>> >> >>
>> >> >> The logic there is to do each document write, during the migrate, in
>> >> >> a
>> >> >> separate lock so other operations can be interleaved. We have seen
>> >> >> this change dramatically reduce the constant lock percentage and
>> >> >> allow
>> >> >> reads, and other writes through during a initial move.
>> >> >>
>> >> >> There is already a delay between moves which doesn't seem to be your
>> >> >> problem.
>> >> >>
>> >> >> On Wed, May 9, 2012 at 5:37 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> >> wrote:
>> >> >> > Scott,
>> >> >> >
>> >> >> > I tried to understand what's going on there. But with a quick
>> >> >> > glance,
>> >> >> > I
>> >> >> > assume, a lock was being held longer than necessary and was fixed.
>> >> >> > It's
>> >> >> > unclear how this can help me. The only way to reliably make sure
>> >> >> > the
>> >> >> > site
>> >> >> > keeps running while a moveChunk is executing, is if, you yield
>> >> >> > every
>> >> >> > second
>> >> >> > within moveChunk and make sure all pending reads complete, before
>> >> >> > the
>> >> >> > next
>> >> >> > moveChunk period starts.
>> >> >> >
>> >> >> > Are you guys doing it, or planning to do it? Is there any other
>> >> >> > high-level
>> >> >> > way to guarantee the reads can be answered within a couple of
>> >> >> > seconds?
>> >> >> >
>> >> >> > Thanks
>> >> >> >
>> >> >> >
>> >> >> > On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez
>> >> >> > <scotthernandez-***@public.gmane.org>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Okay, looking at the receiving shard mongo55, and donor shard
>> >> >> >> mongo43,
>> >> >> >> it look like the fix in 2.0.6 (due out in the next two weeks of
>> >> >> >> so)
>> >> >> >> will help you out a lot.
>> >> >> >>
>> >> >> >> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S
>> >> >> >> <myprasanna-***@public.gmane.org>
>> >> >> >> wrote:
>> >> >> >> > 4:37 the balancer has started and disk shot up.
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S
>> >> >> >> > <myprasanna-***@public.gmane.org>
>> >> >> >> > wrote:
>> >> >> >> >>
>> >> >> >> >> Hey Scott,
>> >> >> >> >>
>> >> >> >> >> I'm going to do this now. Will start it and stop it quickly
>> >> >> >> >> and
>> >> >> >> >> give
>> >> >> >> >> you
>> >> >> >> >> the exact times.
>> >> >> >> >>
>> >> >> >> >> Thanks,
>> >> >> >> >> -Prasanna
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
>> >> >> >> >> <scotthernandez-***@public.gmane.org>
>> >> >> >> >> wrote:
>> >> >> >> >>>
>> >> >> >> >>> Those values are in blocks (512 bytes), not MB.
>> >> >> >> >>>
>> >> >> >> >>> We have an update in the upcoming 2.0.6 stable release which
>> >> >> >> >>> will
>> >> >> >> >>> help
>> >> >> >> >>> with migrations a bit.
>> >> >> >> >>> https://jira.mongodb.org/browse/SERVER-5303
>> >> >> >> >>> You
>> >> >> >> >>> can click on the version like to see other important fixes.
>> >> >> >> >>>
>> >> >> >> >>> Can you do a few migrates tonight so we can see the
>> >> >> >> >>> stats/behavior
>> >> >> >> >>> in
>> >> >> >> >>> MMS? Please let  us know when you do this so we can take a
>> >> >> >> >>> look
>> >> >> >> >>> at
>> >> >> >> >>> the
>> >> >> >> >>> data to see if this fix will help.
>> >> >> >> >>>
>> >> >> >> >>>
>> >> >> >> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S
>> >> >> >> >>> <myprasanna-***@public.gmane.org>
>> >> >> >> >>> wrote:
>> >> >> >> >>> > Hey Dan,
>> >> >> >> >>> >
>> >> >> >> >>> > Read ahead settings are the defaults (256
>> >> >> >> >>> > MB): http://pastebin.com/MHc8s2fq
>> >> >> >> >>> > Let me know if you need something else.
>> >> >> >> >>> >
>> >> >> >> >>> > @Elliot,
>> >> >> >> >>> >
>> >> >> >> >>> > I spoke with some of my friends at foursquare and it turns
>> >> >> >> >>> > out
>> >> >> >> >>> > they
>> >> >> >> >>> > face the
>> >> >> >> >>> > same issue as well. I've had my other friends who run
>> >> >> >> >>> > companies >
>> >> >> >> >>> > $100M
>> >> >> >> >>> > market cap (whom I'd rather not name in this list), tell me
>> >> >> >> >>> > about
>> >> >> >> >>> > their
>> >> >> >> >>> > mongo problems in the past few days. This is pretty serious
>> >> >> >> >>> > stuff
>> >> >> >> >>> > and
>> >> >> >> >>> > will
>> >> >> >> >>> > affect if someone wants to choose mongo as their primary
>> >> >> >> >>> > store
>> >> >> >> >>> > or
>> >> >> >> >>> > not.
>> >> >> >> >>> > To
>> >> >> >> >>> > not have sharding is one thing, but to promise it and being
>> >> >> >> >>> > stuck
>> >> >> >> >>> > in
>> >> >> >> >>> > production is another.
>> >> >> >> >>> >
>> >> >> >> >>> > I really thank you guys and appreciate the responsiveness
>> >> >> >> >>> > so
>> >> >> >> >>> > far.
>> >> >> >> >>> > Happy
>> >> >> >> >>> > to
>> >> >> >> >>> > work together and hope we can resolve this.
>> >> >> >> >>> >
>> >> >> >> >>> > Thanks so much,
>> >> >> >> >>> > -Prasanna
>> >> >> >> >>> >
>> >> >> >> >>> >
>> >> >> >> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <dan-***@public.gmane.orgm>
>> >> >> >> >>> > wrote:
>> >> >> >> >>> >>
>> >> >> >> >>> >> Looks like the docs in that collection
>> >> >> >> >>> >> (lal_production.facebook_names)
>> >> >> >> >>> >> are
>> >> >> >> >>> >> small and likely distributed throughout the disk.
>> >> >> >> >>> >> Can you check your disk readahead setting?
>> >> >> >> >>> >> Please run and post the results of blockev --report.
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S
>> >> >> >> >>> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >>> >> wrote:
>> >> >> >> >>> >>>
>> >> >> >> >>> >>> Yes, indeed.
>> >> >> >> >>> >>>
>> >> >> >> >>> >>> Username: arnold-***@public.gmane.org
>> >> >> >> >>> >>> Machine: mongo55
>> >> >> >> >>> >>>
>> >> >> >> >>> >>>
>> >> >> >> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz
>> >> >> >> >>> >>> <eliot-***@public.gmane.org>
>> >> >> >> >>> >>> wrote:
>> >> >> >> >>> >>>>
>> >> >> >> >>> >>>> Are these machines in MMS?
>> >> >> >> >>> >>>> Hard to see much in the pasted stats.
>> >> >> >> >>> >>>>
>> >> >> >> >>> >>>>
>> >> >> >> >>> >>>>
>> >> >> >> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
>> >> >> >> >>> >>>> <myprasanna-***@public.gmane.org>
>> >> >> >> >>> >>>> wrote:
>> >> >> >> >>> >>>> > Raid 10 on 8 disks.
>> >> >> >> >>> >>>> >
>> >> >> >> >>> >>>> > MongoDB shell version: 2.0.4
>> >> >> >> >>> >>>> > Tue May  8 09:02:29 [initandlisten] db version v2.0.4,
>> >> >> >> >>> >>>> > pdfile
>> >> >> >> >>> >>>> > version
>> >> >> >> >>> >>>> > 4.5
>> >> >> >> >>> >>>> >
>> >> >> >> >>> >>>> >
>> >> >> >> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
>> >> >> >> >>> >>>> > <eliot-***@public.gmane.org>
>> >> >> >> >>> >>>> > wrote:
>> >> >> >> >>> >>>> >>
>> >> >> >> >>> >>>> >> What version of mongo and what kind of ebs raid?
>> >> >> >> >>> >>>> >>
>> >> >> >> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
>> >> >> >> >>> >>>> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >>> >>>> >> wrote:
>> >> >> >> >>> >>>> >> > And yes, disabling the balancer, brings the write
>> >> >> >> >>> >>>> >> > rate
>> >> >> >> >>> >>>> >> > back
>> >> >> >> >>> >>>> >> > to
>> >> >> >> >>> >>>> >> > zero.
>> >> >> >> >>> >>>> >> > Although after 4 minutes!
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat
>> >> >> >> >>> >>>> >> > wrote:
>> >> >> >> >>> >>>> >> >>
>> >> >> >> >>> >>>> >> >> I don't think it goes into deadlock. Can you try
>> >> >> >> >>> >>>> >> >> disabling
>> >> >> >> >>> >>>> >> >> the
>> >> >> >> >>> >>>> >> >> balancer
>> >> >> >> >>> >>>> >> >> and see whether the disk write rate going down or
>> >> >> >> >>> >>>> >> >> not.
>> >> >> >> >>> >>>> >> >> ________________________________
>> >> >> >> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>> >> >> >> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>> >> >> >> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>> >> >> >> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>> >> >> >> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>> >> >> >> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>;
>> >> >> >> >>> >>>> >> >> Hrushikesh
>> >> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>> >> >> >> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>> >> >> >> >>> >>>> >> >>
>> >> >> >> >>> >>>> >> >> Hey guys,
>> >> >> >> >>> >>>> >> >>
>> >> >> >> >>> >>>> >> >> We sharded mongo as per the docs. The issue is the
>> >> >> >> >>> >>>> >> >> balancer
>> >> >> >> >>> >>>> >> >> is
>> >> >> >> >>> >>>> >> >> not
>> >> >> >> >>> >>>> >> >> being
>> >> >> >> >>> >>>> >> >> very clever and not throttling at all. Now we are
>> >> >> >> >>> >>>> >> >> stuck
>> >> >> >> >>> >>>> >> >> with a
>> >> >> >> >>> >>>> >> >> half-sharded
>> >> >> >> >>> >>>> >> >> setup and we can't turn on the balancer because
>> >> >> >> >>> >>>> >> >> it'd
>> >> >> >> >>> >>>> >> >> bring
>> >> >> >> >>> >>>> >> >> the
>> >> >> >> >>> >>>> >> >> site
>> >> >> >> >>> >>>> >> >> down.
>> >> >> >> >>> >>>> >> >> All read queries timeout when the disk write rate
>> >> >> >> >>> >>>> >> >> is
>> >> >> >> >>> >>>> >> >> 100%.
>> >> >> >> >>> >>>> >> >>
>> >> >> >> >>> >>>> >> >> Is there a way out of this mess at all? From a
>> >> >> >> >>> >>>> >> >> high-level,
>> >> >> >> >>> >>>> >> >> mongo
>> >> >> >> >>> >>>> >> >> could
>> >> >> >> >>> >>>> >> >> have handled this by any of the following means:
>> >> >> >> >>> >>>> >> >>
>> >> >> >> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
>> >> >> >> >>> >>>> >> >> 2. Throttle and replicate only as long as the disk
>> >> >> >> >>> >>>> >> >> util
>> >> >> >> >>> >>>> >> >> is
>> >> >> >> >>> >>>> >> >> <
>> >> >> >> >>> >>>> >> >> 50%
>> >> >> >> >>> >>>> >> >> or so.
>> >> >> >> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads and
>> >> >> >> >>> >>>> >> >> writes
>> >> >> >> >>> >>>> >> >> come
>> >> >> >> >>> >>>> >> >> in-between,
>> >> >> >> >>> >>>> >> >> consider them as higher priority, so that a
>> >> >> >> >>> >>>> >> >> read/write
>> >> >> >> >>> >>>> >> >> is
>> >> >> >> >>> >>>> >> >> never
>> >> >> >> >>> >>>> >> >> waiting
>> >> >> >> >>> >>>> >> >> for
>> >> >> >> >>> >>>> >> >> more than a second.
>> >> >> >> >>> >>>> >> >>
>> >> >> >> >>> >>>> >> >> Have you guys heard of this problem before?
>> >> >> >> >>> >>>> >> >> Because
>> >> >> >> >>> >>>> >> >> this
>> >> >> >> >>> >>>> >> >> really
>> >> >> >> >>> >>>> >> >> make
>> >> >> >> >>> >>>> >> >> sharding impossible. I'm not sure what others are
>> >> >> >> >>> >>>> >> >> doing
>> >> >> >> >>> >>>> >> >> in
>> >> >> >> >>> >>>> >> >> production
>> >> >> >> >>> >>>> >> >> to
>> >> >> >> >>> >>>> >> >> work around this.
>> >> >> >> >>> >>>> >> >>
>> >> >> >> >>> >>>> >> >> Thanks,
>> >> >> >> >>> >>>> >> >> -Prasanna
>> >> >> >> >>> >>>> >> >> Co-Founder & CTO, lal.com
>> >> >> >> >>> >>>> >> >>
>> >> >> >> >>> >>>> >> >> --
>> >> >> >> >>> >>>> >> >> You received this message because you are
>> >> >> >> >>> >>>> >> >> subscribed
>> >> >> >> >>> >>>> >> >> to
>> >> >> >> >>> >>>> >> >> the
>> >> >> >> >>> >>>> >> >> Google
>> >> >> >> >>> >>>> >> >> Groups
>> >> >> >> >>> >>>> >> >> "mongodb-user" group.
>> >> >> >> >>> >>>> >> >> To post to this group, send email to
>> >> >> >> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> >>>> >> >> To unsubscribe from this group, send email to
>> >> >> >> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> >>>> >> >> For more options, visit this group at
>> >> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> > --
>> >> >> >> >>> >>>> >> > You received this message because you are
>> >> >> >> >>> >>>> >> > subscribed
>> >> >> >> >>> >>>> >> > to
>> >> >> >> >>> >>>> >> > the
>> >> >> >> >>> >>>> >> > Google
>> >> >> >> >>> >>>> >> > Groups
>> >> >> >> >>> >>>> >> > "mongodb-user" group.
>> >> >> >> >>> >>>> >> > To view this discussion on the web visit
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >> >> >>> >>>> >> >
>> >> >> >> >>> >>>> >> > To post to this group, send email to
>> >> >> >> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to
>> >> >> >> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> >>>> >> > For more options, visit this group at
>> >> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>> >>
>> >> >> >> >>> >>>> >> --
>> >> >> >> >>> >>>> >> You received this message because you are subscribed
>> >> >> >> >>> >>>> >> to
>> >> >> >> >>> >>>> >> the
>> >> >> >> >>> >>>> >> Google
>> >> >> >> >>> >>>> >> Groups
>> >> >> >> >>> >>>> >> "mongodb-user" group.
>> >> >> >> >>> >>>> >> To post to this group, send email to
>> >> >> >> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to
>> >> >> >> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> >>>> >> For more options, visit this group at
>> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>> >>
>> >> >> >> >>> >>>> >
>> >> >> >> >>> >>>> > --
>> >> >> >> >>> >>>> > You received this message because you are subscribed
>> >> >> >> >>> >>>> > to
>> >> >> >> >>> >>>> > the
>> >> >> >> >>> >>>> > Google
>> >> >> >> >>> >>>> > Groups
>> >> >> >> >>> >>>> > "mongodb-user" group.
>> >> >> >> >>> >>>> > To post to this group, send email to
>> >> >> >> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> >>>> > To unsubscribe from this group, send email to
>> >> >> >> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> >>>> > For more options, visit this group at
>> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>>
>> >> >> >> >>> >>>> --
>> >> >> >> >>> >>>> You received this message because you are subscribed to
>> >> >> >> >>> >>>> the
>> >> >> >> >>> >>>> Google
>> >> >> >> >>> >>>> Groups "mongodb-user" group.
>> >> >> >> >>> >>>> To post to this group, send email to
>> >> >> >> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> >>>> To unsubscribe from this group, send email to
>> >> >> >> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> >>>> For more options, visit this group at
>> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>>>
>> >> >> >> >>> >>>
>> >> >> >> >>> >>> --
>> >> >> >> >>> >>> You received this message because you are subscribed to
>> >> >> >> >>> >>> the
>> >> >> >> >>> >>> Google
>> >> >> >> >>> >>> Groups
>> >> >> >> >>> >>> "mongodb-user" group.
>> >> >> >> >>> >>> To post to this group, send email to
>> >> >> >> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> >>> To unsubscribe from this group, send email to
>> >> >> >> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> >>> For more options, visit this group at
>> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >>
>> >> >> >> >>> >>
>> >> >> >> >>> >> --
>> >> >> >> >>> >> You received this message because you are subscribed to
>> >> >> >> >>> >> the
>> >> >> >> >>> >> Google
>> >> >> >> >>> >> Groups
>> >> >> >> >>> >> "mongodb-user" group.
>> >> >> >> >>> >> To post to this group, send email to
>> >> >> >> >>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> >> To unsubscribe from this group, send email to
>> >> >> >> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> >> For more options, visit this group at
>> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>> >
>> >> >> >> >>> >
>> >> >> >> >>> > --
>> >> >> >> >>> > You received this message because you are subscribed to the
>> >> >> >> >>> > Google
>> >> >> >> >>> > Groups
>> >> >> >> >>> > "mongodb-user" group.
>> >> >> >> >>> > To post to this group, send email to
>> >> >> >> >>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> > To unsubscribe from this group, send email to
>> >> >> >> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> > For more options, visit this group at
>> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>>
>> >> >> >> >>> --
>> >> >> >> >>> You received this message because you are subscribed to the
>> >> >> >> >>> Google
>> >> >> >> >>> Groups
>> >> >> >> >>> "mongodb-user" group.
>> >> >> >> >>> To post to this group, send email to
>> >> >> >> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> To unsubscribe from this group, send email to
>> >> >> >> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >>> For more options, visit this group at
>> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>>
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> > Google
>> >> >> >> > Groups
>> >> >> >> > "mongodb-user" group.
>> >> >> >> > To post to this group, send email to
>> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> > To unsubscribe from this group, send email to
>> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> > For more options, visit this group at
>> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >> --
>> >> >> >> You received this message because you are subscribed to the
>> >> >> >> Google
>> >> >> >> Groups
>> >> >> >> "mongodb-user" group.
>> >> >> >> To post to this group, send email to
>> >> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> To unsubscribe from this group, send email to
>> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> For more options, visit this group at
>> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "mongodb-user" group.
>> >> >> > To post to this group, send email to
>> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > To unsubscribe from this group, send email to
>> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "mongodb-user" group.
>> >> >> To post to this group, send email to mongodb-user-/***@public.gmane.org.
>> >> >> To unsubscribe from this group, send email to
>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > To unsubscribe from this group, send email to
>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> To unsubscribe from this group, send email to
>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > To unsubscribe from this group, send email to
>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Shi Shei
2012-05-10 18:13:27 UTC
Permalink
Put me in the list as well. We are suffering from exactly the same issues
that Prasanna is describing.
Mongo pretends to be fast with commodity hardware when you use sharding but
even with sharding and high performance hardware we don't see it.

Have a look at our MMS (group name "idealo", all hosts containing
"offerstore"). Profiling is enabled for queries taking longer than 3
seconds - and there are a lot of them!
We followed and have mostly applied 10gens production rules but the results
are still invisible.
Sorry to say but with such expensive hardware, mongo performance is not
that good as expected.

If you have some pointers, we'll gladly follow your instructions. However,
just waiting for the next and still next mongo release is quite de**
motivating.

On Thursday, May 10, 2012 5:04:33 PM UTC+2, Scott Hernandez wrote:
>
> You seem to looking at this as a binary result. Slow reads do not mean
> downtime. There are many use-cases where many times performance
> degradation during maintenance windows is acceptable but downtime
> (stopping reads) is not.
>
> We are working with everyone to make sure they can get the best
> performance and uptime for their systems. If you have more specific
> areas where you are seeing issue, please create a new thread/post or
> jira issue with the problem (or vote/watch for existing ones).
>
> Nobody thinks the system is perfect as is (how could it be when there
> are so many different use cases), and there are lots of improvements,
> fixes and changes going in to address these areas for improvement.
>
>
> On Thu, May 10, 2012 at 7:44 AM, Prasanna .S <myprasanna-***@public.gmane.org>
> wrote:
> > That use case can easily be supported by just shutting off the reads
> from
> > the application. Basically they are anyways taking downtime -- that
> doesn't
> > take special support, does it?
> >
> > In the normal case, don't you think the reads are the highest priority?
> Come
> > on, this is production.. And I'm telling you, your top clients have this
> > problem. Whats worse for a DB than downtime?
> >
> >
> > On Thu, May 10, 2012 at 7:37 AM, Scott Hernandez <
> scotthernandez-***@public.gmane.org>
> > wrote:
> >>
> >> A simple example is a system where they want the balancer to run as
> >> quickly as possible at low traffic times to balancer data and don't
> >> care about the impact of (user) reads. This can be a big deal when you
> >> are already under-provisioned and need to add a new shard to handle
> >> users more effectively -- it has come up a few times.
> >>
> >> On Thu, May 10, 2012 at 7:33 AM, Prasanna .S <myprasanna-***@public.gmane.org>
> wrote:
> >> > Scott,
> >> >
> >> >> While it might be paramount for some situations/use-cases to
> >> >> complete all
> >> >> reads before a move starts, it might not be for others (which I
> >> > have seen).
> >> >
> >> > Can you please explain the "might not be for others" part? Can you
> give
> >> > me
> >> > an example?
> >> >
> >> >
> >> > On Thu, May 10, 2012 at 7:19 AM, Scott Hernandez
> >> > <scotthernandez-***@public.gmane.org>
> >> > wrote:
> >> >>
> >> >> There are lots of ways to look at this with some competing goals.
> >> >> While it might be paramount for some situations/use-cases to
> complete
> >> >> all reads before a move starts, it might not be for others (which I
> >> >> have seen). This fix will allow reads (and other writes) to get
> >> >> processes while chunks moves happen, period. The issue of locking,
> and
> >> >> contentions on those locks, is very important and there are lots of
> >> >> improvements being made which help across the boards.
> >> >>
> >> >> Feel free to add comments or create new issues with your ideas of
> how
> >> >> your system would like things to best behave:
> >> >> http://jira.mongodb.org/browse/server
> >> >>
> >> >> On Wed, May 9, 2012 at 7:46 PM, Prasanna .S <myprasanna-***@public.gmane.org>
> >> >> wrote:
> >> >> > Scott,
> >> >> >
> >> >> > The below are heuristics, and not guarantees. What you want from a
> DB
> >> >> > stand-point is the latter.
> >> >> >
> >> >> > The delay between moves, won't provide a guarantee, unless you
> make
> >> >> > sure
> >> >> > that
> >> >> > (1) the next iteration of moveChunk, does not start unless all
> >> >> > pending
> >> >> > reads
> >> >> > complete.
> >> >> > (2) Each iteration takes less than 1second.
> >> >> >
> >> >> > Thoughts?
> >> >> >
> >> >> >
> >> >> > On Wed, May 9, 2012 at 6:18 PM, Scott Hernandez
> >> >> > <scotthernandez-***@public.gmane.org>
> >> >> > wrote:
> >> >> >>
> >> >> >> The logic there is to do each document write, during the migrate,
> in
> >> >> >> a
> >> >> >> separate lock so other operations can be interleaved. We have
> seen
> >> >> >> this change dramatically reduce the constant lock percentage and
> >> >> >> allow
> >> >> >> reads, and other writes through during a initial move.
> >> >> >>
> >> >> >> There is already a delay between moves which doesn't seem to be
> your
> >> >> >> problem.
> >> >> >>
> >> >> >> On Wed, May 9, 2012 at 5:37 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>
> >> >> >> wrote:
> >> >> >> > Scott,
> >> >> >> >
> >> >> >> > I tried to understand what's going on there. But with a quick
> >> >> >> > glance,
> >> >> >> > I
> >> >> >> > assume, a lock was being held longer than necessary and was
> fixed.
> >> >> >> > It's
> >> >> >> > unclear how this can help me. The only way to reliably make
> sure
> >> >> >> > the
> >> >> >> > site
> >> >> >> > keeps running while a moveChunk is executing, is if, you yield
> >> >> >> > every
> >> >> >> > second
> >> >> >> > within moveChunk and make sure all pending reads complete,
> before
> >> >> >> > the
> >> >> >> > next
> >> >> >> > moveChunk period starts.
> >> >> >> >
> >> >> >> > Are you guys doing it, or planning to do it? Is there any other
> >> >> >> > high-level
> >> >> >> > way to guarantee the reads can be answered within a couple of
> >> >> >> > seconds?
> >> >> >> >
> >> >> >> > Thanks
> >> >> >> >
> >> >> >> >
> >> >> >> > On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez
> >> >> >> > <scotthernandez-***@public.gmane.org>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> Okay, looking at the receiving shard mongo55, and donor shard
> >> >> >> >> mongo43,
> >> >> >> >> it look like the fix in 2.0.6 (due out in the next two weeks
> of
> >> >> >> >> so)
> >> >> >> >> will help you out a lot.
> >> >> >> >>
> >> >> >> >> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S
> >> >> >> >> <myprasanna-***@public.gmane.org>
> >> >> >> >> wrote:
> >> >> >> >> > 4:37 the balancer has started and disk shot up.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S
> >> >> >> >> > <myprasanna-***@public.gmane.org>
> >> >> >> >> > wrote:
> >> >> >> >> >>
> >> >> >> >> >> Hey Scott,
> >> >> >> >> >>
> >> >> >> >> >> I'm going to do this now. Will start it and stop it quickly
> >> >> >> >> >> and
> >> >> >> >> >> give
> >> >> >> >> >> you
> >> >> >> >> >> the exact times.
> >> >> >> >> >>
> >> >> >> >> >> Thanks,
> >> >> >> >> >> -Prasanna
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
> >> >> >> >> >> <scotthernandez-***@public.gmane.org>
> >> >> >> >> >> wrote:
> >> >> >> >> >>>
> >> >> >> >> >>> Those values are in blocks (512 bytes), not MB.
> >> >> >> >> >>>
> >> >> >> >> >>> We have an update in the upcoming 2.0.6 stable release
> which
> >> >> >> >> >>> will
> >> >> >> >> >>> help
> >> >> >> >> >>> with migrations a bit.
> >> >> >> >> >>> https://jira.mongodb.org/browse/SERVER-5303
> >> >> >> >> >>> You
> >> >> >> >> >>> can click on the version like to see other important
> fixes.
> >> >> >> >> >>>
> >> >> >> >> >>> Can you do a few migrates tonight so we can see the
> >> >> >> >> >>> stats/behavior
> >> >> >> >> >>> in
> >> >> >> >> >>> MMS? Please let us know when you do this so we can take a
> >> >> >> >> >>> look
> >> >> >> >> >>> at
> >> >> >> >> >>> the
> >> >> >> >> >>> data to see if this fix will help.
> >> >> >> >> >>>
> >> >> >> >> >>>
> >> >> >> >> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S
> >> >> >> >> >>> <myprasanna-***@public.gmane.org>
> >> >> >> >> >>> wrote:
> >> >> >> >> >>> > Hey Dan,
> >> >> >> >> >>> >
> >> >> >> >> >>> > Read ahead settings are the defaults (256
> >> >> >> >> >>> > MB): http://pastebin.com/MHc8s2fq
> >> >> >> >> >>> > Let me know if you need something else.
> >> >> >> >> >>> >
> >> >> >> >> >>> > @Elliot,
> >> >> >> >> >>> >
> >> >> >> >> >>> > I spoke with some of my friends at foursquare and it
> turns
> >> >> >> >> >>> > out
> >> >> >> >> >>> > they
> >> >> >> >> >>> > face the
> >> >> >> >> >>> > same issue as well. I've had my other friends who run
> >> >> >> >> >>> > companies >
> >> >> >> >> >>> > $100M
> >> >> >> >> >>> > market cap (whom I'd rather not name in this list), tell
> me
> >> >> >> >> >>> > about
> >> >> >> >> >>> > their
> >> >> >> >> >>> > mongo problems in the past few days. This is pretty
> serious
> >> >> >> >> >>> > stuff
> >> >> >> >> >>> > and
> >> >> >> >> >>> > will
> >> >> >> >> >>> > affect if someone wants to choose mongo as their primary
> >> >> >> >> >>> > store
> >> >> >> >> >>> > or
> >> >> >> >> >>> > not.
> >> >> >> >> >>> > To
> >> >> >> >> >>> > not have sharding is one thing, but to promise it and
> being
> >> >> >> >> >>> > stuck
> >> >> >> >> >>> > in
> >> >> >> >> >>> > production is another.
> >> >> >> >> >>> >
> >> >> >> >> >>> > I really thank you guys and appreciate the
> responsiveness
> >> >> >> >> >>> > so
> >> >> >> >> >>> > far.
> >> >> >> >> >>> > Happy
> >> >> >> >> >>> > to
> >> >> >> >> >>> > work together and hope we can resolve this.
> >> >> >> >> >>> >
> >> >> >> >> >>> > Thanks so much,
> >> >> >> >> >>> > -Prasanna
> >> >> >> >> >>> >
> >> >> >> >> >>> >
> >> >> >> >> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <
> dan-***@public.gmane.org>
> >> >> >> >> >>> > wrote:
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> Looks like the docs in that collection
> >> >> >> >> >>> >> (lal_production.facebook_names)
> >> >> >> >> >>> >> are
> >> >> >> >> >>> >> small and likely distributed throughout the disk.
> >> >> >> >> >>> >> Can you check your disk readahead setting?
> >> >> >> >> >>> >> Please run and post the results of blockev --report.
> >> >> >> >> >>> >>
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S
> >> >> >> >> >>> >> <myprasanna-***@public.gmane.org>
> >> >> >> >> >>> >> wrote:
> >> >> >> >> >>> >>>
> >> >> >> >> >>> >>> Yes, indeed.
> >> >> >> >> >>> >>>
> >> >> >> >> >>> >>> Username: arnold-***@public.gmane.org
> >> >> >> >> >>> >>> Machine: mongo55
> >> >> >> >> >>> >>>
> >> >> >> >> >>> >>>
> >> >> >> >> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz
> >> >> >> >> >>> >>> <eliot-***@public.gmane.org>
> >> >> >> >> >>> >>> wrote:
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>> Are these machines in MMS?
> >> >> >> >> >>> >>>> Hard to see much in the pasted stats.
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
> >> >> >> >> >>> >>>> <myprasanna-***@public.gmane.org>
> >> >> >> >> >>> >>>> wrote:
> >> >> >> >> >>> >>>> > Raid 10 on 8 disks.
> >> >> >> >> >>> >>>> >
> >> >> >> >> >>> >>>> > MongoDB shell version: 2.0.4
> >> >> >> >> >>> >>>> > Tue May 8 09:02:29 [initandlisten] db version
> v2.0.4,
> >> >> >> >> >>> >>>> > pdfile
> >> >> >> >> >>> >>>> > version
> >> >> >> >> >>> >>>> > 4.5
> >> >> >> >> >>> >>>> >
> >> >> >> >> >>> >>>> >
> >> >> >> >> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
> >> >> >> >> >>> >>>> > <eliot-***@public.gmane.org>
> >> >> >> >> >>> >>>> > wrote:
> >> >> >> >> >>> >>>> >>
> >> >> >> >> >>> >>>> >> What version of mongo and what kind of ebs raid?
> >> >> >> >> >>> >>>> >>
> >> >> >> >> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
> >> >> >> >> >>> >>>> >> <myprasanna-***@public.gmane.org>
> >> >> >> >> >>> >>>> >> wrote:
> >> >> >> >> >>> >>>> >> > And yes, disabling the balancer, brings the
> write
> >> >> >> >> >>> >>>> >> > rate
> >> >> >> >> >>> >>>> >> > back
> >> >> >> >> >>> >>>> >> > to
> >> >> >> >> >>> >>>> >> > zero.
> >> >> >> >> >>> >>>> >> > Although after 4 minutes!
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat
> >> >> >> >> >>> >>>> >> > wrote:
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> I don't think it goes into deadlock. Can you
> try
> >> >> >> >> >>> >>>> >> >> disabling
> >> >> >> >> >>> >>>> >> >> the
> >> >> >> >> >>> >>>> >> >> balancer
> >> >> >> >> >>> >>>> >> >> and see whether the disk write rate going down
> or
> >> >> >> >> >>> >>>> >> >> not.
> >> >> >> >> >>> >>>> >> >> ________________________________
> >> >> >> >> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
> >> >> >> >> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
> >> >> >> >> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
> >> >> >> >> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
> >> >> >> >> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
> >> >> >> >> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>;
> >> >> >> >> >>> >>>> >> >> Hrushikesh
> >> >> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
> >> >> >> >> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> Hey guys,
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> We sharded mongo as per the docs. The issue is
> the
> >> >> >> >> >>> >>>> >> >> balancer
> >> >> >> >> >>> >>>> >> >> is
> >> >> >> >> >>> >>>> >> >> not
> >> >> >> >> >>> >>>> >> >> being
> >> >> >> >> >>> >>>> >> >> very clever and not throttling at all. Now we
> are
> >> >> >> >> >>> >>>> >> >> stuck
> >> >> >> >> >>> >>>> >> >> with a
> >> >> >> >> >>> >>>> >> >> half-sharded
> >> >> >> >> >>> >>>> >> >> setup and we can't turn on the balancer because
> >> >> >> >> >>> >>>> >> >> it'd
> >> >> >> >> >>> >>>> >> >> bring
> >> >> >> >> >>> >>>> >> >> the
> >> >> >> >> >>> >>>> >> >> site
> >> >> >> >> >>> >>>> >> >> down.
> >> >> >> >> >>> >>>> >> >> All read queries timeout when the disk write
> rate
> >> >> >> >> >>> >>>> >> >> is
> >> >> >> >> >>> >>>> >> >> 100%.
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> Is there a way out of this mess at all? From a
> >> >> >> >> >>> >>>> >> >> high-level,
> >> >> >> >> >>> >>>> >> >> mongo
> >> >> >> >> >>> >>>> >> >> could
> >> >> >> >> >>> >>>> >> >> have handled this by any of the following
> means:
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
> >> >> >> >> >>> >>>> >> >> 2. Throttle and replicate only as long as the
> disk
> >> >> >> >> >>> >>>> >> >> util
> >> >> >> >> >>> >>>> >> >> is
> >> >> >> >> >>> >>>> >> >> <
> >> >> >> >> >>> >>>> >> >> 50%
> >> >> >> >> >>> >>>> >> >> or so.
> >> >> >> >> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads
> and
> >> >> >> >> >>> >>>> >> >> writes
> >> >> >> >> >>> >>>> >> >> come
> >> >> >> >> >>> >>>> >> >> in-between,
> >> >> >> >> >>> >>>> >> >> consider them as higher priority, so that a
> >> >> >> >> >>> >>>> >> >> read/write
> >> >> >> >> >>> >>>> >> >> is
> >> >> >> >> >>> >>>> >> >> never
> >> >> >> >> >>> >>>> >> >> waiting
> >> >> >> >> >>> >>>> >> >> for
> >> >> >> >> >>> >>>> >> >> more than a second.
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> Have you guys heard of this problem before?
> >> >> >> >> >>> >>>> >> >> Because
> >> >> >> >> >>> >>>> >> >> this
> >> >> >> >> >>> >>>> >> >> really
> >> >> >> >> >>> >>>> >> >> make
> >> >> >> >> >>> >>>> >> >> sharding impossible. I'm not sure what others
> are
> >> >> >> >> >>> >>>> >> >> doing
> >> >> >> >> >>> >>>> >> >> in
> >> >> >> >> >>> >>>> >> >> production
> >> >> >> >> >>> >>>> >> >> to
> >> >> >> >> >>> >>>> >> >> work around this.
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> Thanks,
> >> >> >> >> >>> >>>> >> >> -Prasanna
> >> >> >> >> >>> >>>> >> >> Co-Founder & CTO, lal.com
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> --
> >> >> >> >> >>> >>>> >> >> You received this message because you are
> >> >> >> >> >>> >>>> >> >> subscribed
> >> >> >> >> >>> >>>> >> >> to
> >> >> >> >> >>> >>>> >> >> the
> >> >> >> >> >>> >>>> >> >> Google
> >> >> >> >> >>> >>>> >> >> Groups
> >> >> >> >> >>> >>>> >> >> "mongodb-user" group.
> >> >> >> >> >>> >>>> >> >> To post to this group, send email to
> >> >> >> >> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> >> To unsubscribe from this group, send email to
> >> >> >> >> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> >> For more options, visit this group at
> >> >> >> >> >>> >>>> >> >>
> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> > --
> >> >> >> >> >>> >>>> >> > You received this message because you are
> >> >> >> >> >>> >>>> >> > subscribed
> >> >> >> >> >>> >>>> >> > to
> >> >> >> >> >>> >>>> >> > the
> >> >> >> >> >>> >>>> >> > Google
> >> >> >> >> >>> >>>> >> > Groups
> >> >> >> >> >>> >>>> >> > "mongodb-user" group.
> >> >> >> >> >>> >>>> >> > To view this discussion on the web visit
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> >
> https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> > To post to this group, send email to
> >> >> >> >> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to
> >> >> >> >> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> > For more options, visit this group at
> >> >> >> >> >>> >>>> >> >
> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >>>> >>
> >> >> >> >> >>> >>>> >> --
> >> >> >> >> >>> >>>> >> You received this message because you are
> subscribed
> >> >> >> >> >>> >>>> >> to
> >> >> >> >> >>> >>>> >> the
> >> >> >> >> >>> >>>> >> Google
> >> >> >> >> >>> >>>> >> Groups
> >> >> >> >> >>> >>>> >> "mongodb-user" group.
> >> >> >> >> >>> >>>> >> To post to this group, send email to
> >> >> >> >> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to
> >> >> >> >> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> For more options, visit this group at
> >> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>
> >> >> >> >> >>> >>>> >>
> >> >> >> >> >>> >>>> >
> >> >> >> >> >>> >>>> > --
> >> >> >> >> >>> >>>> > You received this message because you are
> subscribed
> >> >> >> >> >>> >>>> > to
> >> >> >> >> >>> >>>> > the
> >> >> >> >> >>> >>>> > Google
> >> >> >> >> >>> >>>> > Groups
> >> >> >> >> >>> >>>> > "mongodb-user" group.
> >> >> >> >> >>> >>>> > To post to this group, send email to
> >> >> >> >> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> > To unsubscribe from this group, send email to
> >> >> >> >> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> > For more options, visit this group at
> >> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>> --
> >> >> >> >> >>> >>>> You received this message because you are subscribed
> to
> >> >> >> >> >>> >>>> the
> >> >> >> >> >>> >>>> Google
> >> >> >> >> >>> >>>> Groups "mongodb-user" group.
> >> >> >> >> >>> >>>> To post to this group, send email to
> >> >> >> >> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> To unsubscribe from this group, send email to
> >> >> >> >> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> For more options, visit this group at
> >> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>
> >> >> >> >> >>> >>> --
> >> >> >> >> >>> >>> You received this message because you are subscribed
> to
> >> >> >> >> >>> >>> the
> >> >> >> >> >>> >>> Google
> >> >> >> >> >>> >>> Groups
> >> >> >> >> >>> >>> "mongodb-user" group.
> >> >> >> >> >>> >>> To post to this group, send email to
> >> >> >> >> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>> To unsubscribe from this group, send email to
> >> >> >> >> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>> For more options, visit this group at
> >> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >>
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> --
> >> >> >> >> >>> >> You received this message because you are subscribed to
> >> >> >> >> >>> >> the
> >> >> >> >> >>> >> Google
> >> >> >> >> >>> >> Groups
> >> >> >> >> >>> >> "mongodb-user" group.
> >> >> >> >> >>> >> To post to this group, send email to
> >> >> >> >> >>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >> To unsubscribe from this group, send email to
> >> >> >> >> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >> For more options, visit this group at
> >> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >
> >> >> >> >> >>> >
> >> >> >> >> >>> > --
> >> >> >> >> >>> > You received this message because you are subscribed to
> the
> >> >> >> >> >>> > Google
> >> >> >> >> >>> > Groups
> >> >> >> >> >>> > "mongodb-user" group.
> >> >> >> >> >>> > To post to this group, send email to
> >> >> >> >> >>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> > To unsubscribe from this group, send email to
> >> >> >> >> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> > For more options, visit this group at
> >> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>>
> >> >> >> >> >>> --
> >> >> >> >> >>> You received this message because you are subscribed to
> the
> >> >> >> >> >>> Google
> >> >> >> >> >>> Groups
> >> >> >> >> >>> "mongodb-user" group.
> >> >> >> >> >>> To post to this group, send email to
> >> >> >> >> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> To unsubscribe from this group, send email to
> >> >> >> >> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> For more options, visit this group at
> >> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>>
> >> >> >> >> >>
> >> >> >> >> >
> >> >> >> >> > --
> >> >> >> >> > You received this message because you are subscribed to the
> >> >> >> >> > Google
> >> >> >> >> > Groups
> >> >> >> >> > "mongodb-user" group.
> >> >> >> >> > To post to this group, send email to
> >> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> > To unsubscribe from this group, send email to
> >> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> > For more options, visit this group at
> >> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >>
> >> >> >> >> --
> >> >> >> >> You received this message because you are subscribed to the
> >> >> >> >> Google
> >> >> >> >> Groups
> >> >> >> >> "mongodb-user" group.
> >> >> >> >> To post to this group, send email to
> >> >> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> To unsubscribe from this group, send email to
> >> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> For more options, visit this group at
> >> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >>
> >> >> >> >
> >> >> >> > --
> >> >> >> > You received this message because you are subscribed to the
> Google
> >> >> >> > Groups
> >> >> >> > "mongodb-user" group.
> >> >> >> > To post to this group, send email to
> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> > To unsubscribe from this group, send email to
> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> > For more options, visit this group at
> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>
> >> >> >> --
> >> >> >> You received this message because you are subscribed to the
> Google
> >> >> >> Groups
> >> >> >> "mongodb-user" group.
> >> >> >> To post to this group, send email to
> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> To unsubscribe from this group, send email to
> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> For more options, visit this group at
> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>
> >> >> >
> >> >> > --
> >> >> > You received this message because you are subscribed to the Google
> >> >> > Groups
> >> >> > "mongodb-user" group.
> >> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>
> >> >> > To unsubscribe from this group, send email to
> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> > For more options, visit this group at
> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >>
> >> >> --
> >> >> You received this message because you are subscribed to the Google
> >> >> Groups
> >> >> "mongodb-user" group.
> >> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> To unsubscribe from this group, send email to
> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> For more options, visit this group at
> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >>
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups
> >> > "mongodb-user" group.
> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> > To unsubscribe from this group, send email to
> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> > For more options, visit this group at
> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "mongodb-user" group.
> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> To unsubscribe from this group, send email to
> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> For more options, visit this group at
> >> http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > To unsubscribe from this group, send email to
> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > For more options, visit this group at
> > http://groups.google.com/group/mongodb-user?hl=en.
>

On Thursday, May 10, 2012 5:04:33 PM UTC+2, Scott Hernandez wrote:
>
> You seem to looking at this as a binary result. Slow reads do not mean
> downtime. There are many use-cases where many times performance
> degradation during maintenance windows is acceptable but downtime
> (stopping reads) is not.
>
> We are working with everyone to make sure they can get the best
> performance and uptime for their systems. If you have more specific
> areas where you are seeing issue, please create a new thread/post or
> jira issue with the problem (or vote/watch for existing ones).
>
> Nobody thinks the system is perfect as is (how could it be when there
> are so many different use cases), and there are lots of improvements,
> fixes and changes going in to address these areas for improvement.
>
>
> On Thu, May 10, 2012 at 7:44 AM, Prasanna .S <myprasanna-***@public.gmane.org>
> wrote:
> > That use case can easily be supported by just shutting off the reads
> from
> > the application. Basically they are anyways taking downtime -- that
> doesn't
> > take special support, does it?
> >
> > In the normal case, don't you think the reads are the highest priority?
> Come
> > on, this is production.. And I'm telling you, your top clients have this
> > problem. Whats worse for a DB than downtime?
> >
> >
> > On Thu, May 10, 2012 at 7:37 AM, Scott Hernandez <
> scotthernandez-***@public.gmane.org>
> > wrote:
> >>
> >> A simple example is a system where they want the balancer to run as
> >> quickly as possible at low traffic times to balancer data and don't
> >> care about the impact of (user) reads. This can be a big deal when you
> >> are already under-provisioned and need to add a new shard to handle
> >> users more effectively -- it has come up a few times.
> >>
> >> On Thu, May 10, 2012 at 7:33 AM, Prasanna .S <myprasanna-***@public.gmane.org>
> wrote:
> >> > Scott,
> >> >
> >> >> While it might be paramount for some situations/use-cases to
> >> >> complete all
> >> >> reads before a move starts, it might not be for others (which I
> >> > have seen).
> >> >
> >> > Can you please explain the "might not be for others" part? Can you
> give
> >> > me
> >> > an example?
> >> >
> >> >
> >> > On Thu, May 10, 2012 at 7:19 AM, Scott Hernandez
> >> > <scotthernandez-***@public.gmane.org>
> >> > wrote:
> >> >>
> >> >> There are lots of ways to look at this with some competing goals.
> >> >> While it might be paramount for some situations/use-cases to
> complete
> >> >> all reads before a move starts, it might not be for others (which I
> >> >> have seen). This fix will allow reads (and other writes) to get
> >> >> processes while chunks moves happen, period. The issue of locking,
> and
> >> >> contentions on those locks, is very important and there are lots of
> >> >> improvements being made which help across the boards.
> >> >>
> >> >> Feel free to add comments or create new issues with your ideas of
> how
> >> >> your system would like things to best behave:
> >> >> http://jira.mongodb.org/browse/server
> >> >>
> >> >> On Wed, May 9, 2012 at 7:46 PM, Prasanna .S <myprasanna-***@public.gmane.org>
> >> >> wrote:
> >> >> > Scott,
> >> >> >
> >> >> > The below are heuristics, and not guarantees. What you want from a
> DB
> >> >> > stand-point is the latter.
> >> >> >
> >> >> > The delay between moves, won't provide a guarantee, unless you
> make
> >> >> > sure
> >> >> > that
> >> >> > (1) the next iteration of moveChunk, does not start unless all
> >> >> > pending
> >> >> > reads
> >> >> > complete.
> >> >> > (2) Each iteration takes less than 1second.
> >> >> >
> >> >> > Thoughts?
> >> >> >
> >> >> >
> >> >> > On Wed, May 9, 2012 at 6:18 PM, Scott Hernandez
> >> >> > <scotthernandez-***@public.gmane.org>
> >> >> > wrote:
> >> >> >>
> >> >> >> The logic there is to do each document write, during the migrate,
> in
> >> >> >> a
> >> >> >> separate lock so other operations can be interleaved. We have
> seen
> >> >> >> this change dramatically reduce the constant lock percentage and
> >> >> >> allow
> >> >> >> reads, and other writes through during a initial move.
> >> >> >>
> >> >> >> There is already a delay between moves which doesn't seem to be
> your
> >> >> >> problem.
> >> >> >>
> >> >> >> On Wed, May 9, 2012 at 5:37 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>
> >> >> >> wrote:
> >> >> >> > Scott,
> >> >> >> >
> >> >> >> > I tried to understand what's going on there. But with a quick
> >> >> >> > glance,
> >> >> >> > I
> >> >> >> > assume, a lock was being held longer than necessary and was
> fixed.
> >> >> >> > It's
> >> >> >> > unclear how this can help me. The only way to reliably make
> sure
> >> >> >> > the
> >> >> >> > site
> >> >> >> > keeps running while a moveChunk is executing, is if, you yield
> >> >> >> > every
> >> >> >> > second
> >> >> >> > within moveChunk and make sure all pending reads complete,
> before
> >> >> >> > the
> >> >> >> > next
> >> >> >> > moveChunk period starts.
> >> >> >> >
> >> >> >> > Are you guys doing it, or planning to do it? Is there any other
> >> >> >> > high-level
> >> >> >> > way to guarantee the reads can be answered within a couple of
> >> >> >> > seconds?
> >> >> >> >
> >> >> >> > Thanks
> >> >> >> >
> >> >> >> >
> >> >> >> > On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez
> >> >> >> > <scotthernandez-***@public.gmane.org>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> Okay, looking at the receiving shard mongo55, and donor shard
> >> >> >> >> mongo43,
> >> >> >> >> it look like the fix in 2.0.6 (due out in the next two weeks
> of
> >> >> >> >> so)
> >> >> >> >> will help you out a lot.
> >> >> >> >>
> >> >> >> >> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S
> >> >> >> >> <myprasanna-***@public.gmane.org>
> >> >> >> >> wrote:
> >> >> >> >> > 4:37 the balancer has started and disk shot up.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S
> >> >> >> >> > <myprasanna-***@public.gmane.org>
> >> >> >> >> > wrote:
> >> >> >> >> >>
> >> >> >> >> >> Hey Scott,
> >> >> >> >> >>
> >> >> >> >> >> I'm going to do this now. Will start it and stop it quickly
> >> >> >> >> >> and
> >> >> >> >> >> give
> >> >> >> >> >> you
> >> >> >> >> >> the exact times.
> >> >> >> >> >>
> >> >> >> >> >> Thanks,
> >> >> >> >> >> -Prasanna
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
> >> >> >> >> >> <scotthernandez-***@public.gmane.org>
> >> >> >> >> >> wrote:
> >> >> >> >> >>>
> >> >> >> >> >>> Those values are in blocks (512 bytes), not MB.
> >> >> >> >> >>>
> >> >> >> >> >>> We have an update in the upcoming 2.0.6 stable release
> which
> >> >> >> >> >>> will
> >> >> >> >> >>> help
> >> >> >> >> >>> with migrations a bit.
> >> >> >> >> >>> https://jira.mongodb.org/browse/SERVER-5303
> >> >> >> >> >>> You
> >> >> >> >> >>> can click on the version like to see other important
> fixes.
> >> >> >> >> >>>
> >> >> >> >> >>> Can you do a few migrates tonight so we can see the
> >> >> >> >> >>> stats/behavior
> >> >> >> >> >>> in
> >> >> >> >> >>> MMS? Please let us know when you do this so we can take a
> >> >> >> >> >>> look
> >> >> >> >> >>> at
> >> >> >> >> >>> the
> >> >> >> >> >>> data to see if this fix will help.
> >> >> >> >> >>>
> >> >> >> >> >>>
> >> >> >> >> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S
> >> >> >> >> >>> <myprasanna-***@public.gmane.org>
> >> >> >> >> >>> wrote:
> >> >> >> >> >>> > Hey Dan,
> >> >> >> >> >>> >
> >> >> >> >> >>> > Read ahead settings are the defaults (256
> >> >> >> >> >>> > MB): http://pastebin.com/MHc8s2fq
> >> >> >> >> >>> > Let me know if you need something else.
> >> >> >> >> >>> >
> >> >> >> >> >>> > @Elliot,
> >> >> >> >> >>> >
> >> >> >> >> >>> > I spoke with some of my friends at foursquare and it
> turns
> >> >> >> >> >>> > out
> >> >> >> >> >>> > they
> >> >> >> >> >>> > face the
> >> >> >> >> >>> > same issue as well. I've had my other friends who run
> >> >> >> >> >>> > companies >
> >> >> >> >> >>> > $100M
> >> >> >> >> >>> > market cap (whom I'd rather not name in this list), tell
> me
> >> >> >> >> >>> > about
> >> >> >> >> >>> > their
> >> >> >> >> >>> > mongo problems in the past few days. This is pretty
> serious
> >> >> >> >> >>> > stuff
> >> >> >> >> >>> > and
> >> >> >> >> >>> > will
> >> >> >> >> >>> > affect if someone wants to choose mongo as their primary
> >> >> >> >> >>> > store
> >> >> >> >> >>> > or
> >> >> >> >> >>> > not.
> >> >> >> >> >>> > To
> >> >> >> >> >>> > not have sharding is one thing, but to promise it and
> being
> >> >> >> >> >>> > stuck
> >> >> >> >> >>> > in
> >> >> >> >> >>> > production is another.
> >> >> >> >> >>> >
> >> >> >> >> >>> > I really thank you guys and appreciate the
> responsiveness
> >> >> >> >> >>> > so
> >> >> >> >> >>> > far.
> >> >> >> >> >>> > Happy
> >> >> >> >> >>> > to
> >> >> >> >> >>> > work together and hope we can resolve this.
> >> >> >> >> >>> >
> >> >> >> >> >>> > Thanks so much,
> >> >> >> >> >>> > -Prasanna
> >> >> >> >> >>> >
> >> >> >> >> >>> >
> >> >> >> >> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <
> dan-***@public.gmane.org>
> >> >> >> >> >>> > wrote:
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> Looks like the docs in that collection
> >> >> >> >> >>> >> (lal_production.facebook_names)
> >> >> >> >> >>> >> are
> >> >> >> >> >>> >> small and likely distributed throughout the disk.
> >> >> >> >> >>> >> Can you check your disk readahead setting?
> >> >> >> >> >>> >> Please run and post the results of blockev --report.
> >> >> >> >> >>> >>
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S
> >> >> >> >> >>> >> <myprasanna-***@public.gmane.org>
> >> >> >> >> >>> >> wrote:
> >> >> >> >> >>> >>>
> >> >> >> >> >>> >>> Yes, indeed.
> >> >> >> >> >>> >>>
> >> >> >> >> >>> >>> Username: arnold-***@public.gmane.org
> >> >> >> >> >>> >>> Machine: mongo55
> >> >> >> >> >>> >>>
> >> >> >> >> >>> >>>
> >> >> >> >> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz
> >> >> >> >> >>> >>> <eliot-***@public.gmane.org>
> >> >> >> >> >>> >>> wrote:
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>> Are these machines in MMS?
> >> >> >> >> >>> >>>> Hard to see much in the pasted stats.
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
> >> >> >> >> >>> >>>> <myprasanna-***@public.gmane.org>
> >> >> >> >> >>> >>>> wrote:
> >> >> >> >> >>> >>>> > Raid 10 on 8 disks.
> >> >> >> >> >>> >>>> >
> >> >> >> >> >>> >>>> > MongoDB shell version: 2.0.4
> >> >> >> >> >>> >>>> > Tue May 8 09:02:29 [initandlisten] db version
> v2.0.4,
> >> >> >> >> >>> >>>> > pdfile
> >> >> >> >> >>> >>>> > version
> >> >> >> >> >>> >>>> > 4.5
> >> >> >> >> >>> >>>> >
> >> >> >> >> >>> >>>> >
> >> >> >> >> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
> >> >> >> >> >>> >>>> > <eliot-***@public.gmane.org>
> >> >> >> >> >>> >>>> > wrote:
> >> >> >> >> >>> >>>> >>
> >> >> >> >> >>> >>>> >> What version of mongo and what kind of ebs raid?
> >> >> >> >> >>> >>>> >>
> >> >> >> >> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
> >> >> >> >> >>> >>>> >> <myprasanna-***@public.gmane.org>
> >> >> >> >> >>> >>>> >> wrote:
> >> >> >> >> >>> >>>> >> > And yes, disabling the balancer, brings the
> write
> >> >> >> >> >>> >>>> >> > rate
> >> >> >> >> >>> >>>> >> > back
> >> >> >> >> >>> >>>> >> > to
> >> >> >> >> >>> >>>> >> > zero.
> >> >> >> >> >>> >>>> >> > Although after 4 minutes!
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat
> >> >> >> >> >>> >>>> >> > wrote:
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> I don't think it goes into deadlock. Can you
> try
> >> >> >> >> >>> >>>> >> >> disabling
> >> >> >> >> >>> >>>> >> >> the
> >> >> >> >> >>> >>>> >> >> balancer
> >> >> >> >> >>> >>>> >> >> and see whether the disk write rate going down
> or
> >> >> >> >> >>> >>>> >> >> not.
> >> >> >> >> >>> >>>> >> >> ________________________________
> >> >> >> >> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
> >> >> >> >> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
> >> >> >> >> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
> >> >> >> >> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
> >> >> >> >> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
> >> >> >> >> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>;
> >> >> >> >> >>> >>>> >> >> Hrushikesh
> >> >> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
> >> >> >> >> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> Hey guys,
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> We sharded mongo as per the docs. The issue is
> the
> >> >> >> >> >>> >>>> >> >> balancer
> >> >> >> >> >>> >>>> >> >> is
> >> >> >> >> >>> >>>> >> >> not
> >> >> >> >> >>> >>>> >> >> being
> >> >> >> >> >>> >>>> >> >> very clever and not throttling at all. Now we
> are
> >> >> >> >> >>> >>>> >> >> stuck
> >> >> >> >> >>> >>>> >> >> with a
> >> >> >> >> >>> >>>> >> >> half-sharded
> >> >> >> >> >>> >>>> >> >> setup and we can't turn on the balancer because
> >> >> >> >> >>> >>>> >> >> it'd
> >> >> >> >> >>> >>>> >> >> bring
> >> >> >> >> >>> >>>> >> >> the
> >> >> >> >> >>> >>>> >> >> site
> >> >> >> >> >>> >>>> >> >> down.
> >> >> >> >> >>> >>>> >> >> All read queries timeout when the disk write
> rate
> >> >> >> >> >>> >>>> >> >> is
> >> >> >> >> >>> >>>> >> >> 100%.
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> Is there a way out of this mess at all? From a
> >> >> >> >> >>> >>>> >> >> high-level,
> >> >> >> >> >>> >>>> >> >> mongo
> >> >> >> >> >>> >>>> >> >> could
> >> >> >> >> >>> >>>> >> >> have handled this by any of the following
> means:
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
> >> >> >> >> >>> >>>> >> >> 2. Throttle and replicate only as long as the
> disk
> >> >> >> >> >>> >>>> >> >> util
> >> >> >> >> >>> >>>> >> >> is
> >> >> >> >> >>> >>>> >> >> <
> >> >> >> >> >>> >>>> >> >> 50%
> >> >> >> >> >>> >>>> >> >> or so.
> >> >> >> >> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads
> and
> >> >> >> >> >>> >>>> >> >> writes
> >> >> >> >> >>> >>>> >> >> come
> >> >> >> >> >>> >>>> >> >> in-between,
> >> >> >> >> >>> >>>> >> >> consider them as higher priority, so that a
> >> >> >> >> >>> >>>> >> >> read/write
> >> >> >> >> >>> >>>> >> >> is
> >> >> >> >> >>> >>>> >> >> never
> >> >> >> >> >>> >>>> >> >> waiting
> >> >> >> >> >>> >>>> >> >> for
> >> >> >> >> >>> >>>> >> >> more than a second.
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> Have you guys heard of this problem before?
> >> >> >> >> >>> >>>> >> >> Because
> >> >> >> >> >>> >>>> >> >> this
> >> >> >> >> >>> >>>> >> >> really
> >> >> >> >> >>> >>>> >> >> make
> >> >> >> >> >>> >>>> >> >> sharding impossible. I'm not sure what others
> are
> >> >> >> >> >>> >>>> >> >> doing
> >> >> >> >> >>> >>>> >> >> in
> >> >> >> >> >>> >>>> >> >> production
> >> >> >> >> >>> >>>> >> >> to
> >> >> >> >> >>> >>>> >> >> work around this.
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> Thanks,
> >> >> >> >> >>> >>>> >> >> -Prasanna
> >> >> >> >> >>> >>>> >> >> Co-Founder & CTO, lal.com
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> --
> >> >> >> >> >>> >>>> >> >> You received this message because you are
> >> >> >> >> >>> >>>> >> >> subscribed
> >> >> >> >> >>> >>>> >> >> to
> >> >> >> >> >>> >>>> >> >> the
> >> >> >> >> >>> >>>> >> >> Google
> >> >> >> >> >>> >>>> >> >> Groups
> >> >> >> >> >>> >>>> >> >> "mongodb-user" group.
> >> >> >> >> >>> >>>> >> >> To post to this group, send email to
> >> >> >> >> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> >> To unsubscribe from this group, send email to
> >> >> >> >> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> >> For more options, visit this group at
> >> >> >> >> >>> >>>> >> >>
> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> > --
> >> >> >> >> >>> >>>> >> > You received this message because you are
> >> >> >> >> >>> >>>> >> > subscribed
> >> >> >> >> >>> >>>> >> > to
> >> >> >> >> >>> >>>> >> > the
> >> >> >> >> >>> >>>> >> > Google
> >> >> >> >> >>> >>>> >> > Groups
> >> >> >> >> >>> >>>> >> > "mongodb-user" group.
> >> >> >> >> >>> >>>> >> > To view this discussion on the web visit
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> >
> https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> > To post to this group, send email to
> >> >> >> >> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to
> >> >> >> >> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> > For more options, visit this group at
> >> >> >> >> >>> >>>> >> >
> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >>>> >>
> >> >> >> >> >>> >>>> >> --
> >> >> >> >> >>> >>>> >> You received this message because you are
> subscribed
> >> >> >> >> >>> >>>> >> to
> >> >> >> >> >>> >>>> >> the
> >> >> >> >> >>> >>>> >> Google
> >> >> >> >> >>> >>>> >> Groups
> >> >> >> >> >>> >>>> >> "mongodb-user" group.
> >> >> >> >> >>> >>>> >> To post to this group, send email to
> >> >> >> >> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to
> >> >> >> >> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> For more options, visit this group at
> >> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>
> >> >> >> >> >>> >>>> >>
> >> >> >> >> >>> >>>> >
> >> >> >> >> >>> >>>> > --
> >> >> >> >> >>> >>>> > You received this message because you are
> subscribed
> >> >> >> >> >>> >>>> > to
> >> >> >> >> >>> >>>> > the
> >> >> >> >> >>> >>>> > Google
> >> >> >> >> >>> >>>> > Groups
> >> >> >> >> >>> >>>> > "mongodb-user" group.
> >> >> >> >> >>> >>>> > To post to this group, send email to
> >> >> >> >> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> > To unsubscribe from this group, send email to
> >> >> >> >> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> > For more options, visit this group at
> >> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>> --
> >> >> >> >> >>> >>>> You received this message because you are subscribed
> to
> >> >> >> >> >>> >>>> the
> >> >> >> >> >>> >>>> Google
> >> >> >> >> >>> >>>> Groups "mongodb-user" group.
> >> >> >> >> >>> >>>> To post to this group, send email to
> >> >> >> >> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> To unsubscribe from this group, send email to
> >> >> >> >> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> For more options, visit this group at
> >> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>
> >> >> >> >> >>> >>> --
> >> >> >> >> >>> >>> You received this message because you are subscribed
> to
> >> >> >> >> >>> >>> the
> >> >> >> >> >>> >>> Google
> >> >> >> >> >>> >>> Groups
> >> >> >> >> >>> >>> "mongodb-user" group.
> >> >> >> >> >>> >>> To post to this group, send email to
> >> >> >> >> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>> To unsubscribe from this group, send email to
> >> >> >> >> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>> For more options, visit this group at
> >> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >>
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> --
> >> >> >> >> >>> >> You received this message because you are subscribed to
> >> >> >> >> >>> >> the
> >> >> >> >> >>> >> Google
> >> >> >> >> >>> >> Groups
> >> >> >> >> >>> >> "mongodb-user" group.
> >> >> >> >> >>> >> To post to this group, send email to
> >> >> >> >> >>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >> To unsubscribe from this group, send email to
> >> >> >> >> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >> For more options, visit this group at
> >> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >
> >> >> >> >> >>> >
> >> >> >> >> >>> > --
> >> >> >> >> >>> > You received this message because you are subscribed to
> the
> >> >> >> >> >>> > Google
> >> >> >> >> >>> > Groups
> >> >> >> >> >>> > "mongodb-user" group.
> >> >> >> >> >>> > To post to this group, send email to
> >> >> >> >> >>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> > To unsubscribe from this group, send email to
> >> >> >> >> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> > For more options, visit this group at
> >> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>>
> >> >> >> >> >>> --
> >> >> >> >> >>> You received this message because you are subscribed to
> the
> >> >> >> >> >>> Google
> >> >> >> >> >>> Groups
> >> >> >> >> >>> "mongodb-user" group.
> >> >> >> >> >>> To post to this group, send email to
> >> >> >> >> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> To unsubscribe from this group, send email to
> >> >> >> >> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> For more options, visit this group at
> >> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>>
> >> >> >> >> >>
> >> >> >> >> >
> >> >> >> >> > --
> >> >> >> >> > You received this message because you are subscribed to the
> >> >> >> >> > Google
> >> >> >> >> > Groups
> >> >> >> >> > "mongodb-user" group.
> >> >> >> >> > To post to this group, send email to
> >> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> > To unsubscribe from this group, send email to
> >> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> > For more options, visit this group at
> >> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >>
> >> >> >> >> --
> >> >> >> >> You received this message because you are subscribed to the
> >> >> >> >> Google
> >> >> >> >> Groups
> >> >> >> >> "mongodb-user" group.
> >> >> >> >> To post to this group, send email to
> >> >> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> To unsubscribe from this group, send email to
> >> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> For more options, visit this group at
> >> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >>
> >> >> >> >
> >> >> >> > --
> >> >> >> > You received this message because you are subscribed to the
> Google
> >> >> >> > Groups
> >> >> >> > "mongodb-user" group.
> >> >> >> > To post to this group, send email to
> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> > To unsubscribe from this group, send email to
> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> > For more options, visit this group at
> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>
> >> >> >> --
> >> >> >> You received this message because you are subscribed to the
> Google
> >> >> >> Groups
> >> >> >> "mongodb-user" group.
> >> >> >> To post to this group, send email to
> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> To unsubscribe from this group, send email to
> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> For more options, visit this group at
> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>
> >> >> >
> >> >> > --
> >> >> > You received this message because you are subscribed to the Google
> >> >> > Groups
> >> >> > "mongodb-user" group.
> >> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>
> >> >> > To unsubscribe from this group, send email to
> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> > For more options, visit this group at
> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >>
> >> >> --
> >> >> You received this message because you are subscribed to the Google
> >> >> Groups
> >> >> "mongodb-user" group.
> >> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> To unsubscribe from this group, send email to
> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> For more options, visit this group at
> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >>
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups
> >> > "mongodb-user" group.
> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> > To unsubscribe from this group, send email to
> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> > For more options, visit this group at
> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "mongodb-user" group.
> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> To unsubscribe from this group, send email to
> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> For more options, visit this group at
> >> http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > To unsubscribe from this group, send email to
> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > For more options, visit this group at
> > http://groups.google.com/group/mongodb-user?hl=en.
>

On Thursday, May 10, 2012 5:04:33 PM UTC+2, Scott Hernandez wrote:
>
> You seem to looking at this as a binary result. Slow reads do not mean
> downtime. There are many use-cases where many times performance
> degradation during maintenance windows is acceptable but downtime
> (stopping reads) is not.
>
> We are working with everyone to make sure they can get the best
> performance and uptime for their systems. If you have more specific
> areas where you are seeing issue, please create a new thread/post or
> jira issue with the problem (or vote/watch for existing ones).
>
> Nobody thinks the system is perfect as is (how could it be when there
> are so many different use cases), and there are lots of improvements,
> fixes and changes going in to address these areas for improvement.
>
>
> On Thu, May 10, 2012 at 7:44 AM, Prasanna .S <myprasanna-***@public.gmane.org>
> wrote:
> > That use case can easily be supported by just shutting off the reads
> from
> > the application. Basically they are anyways taking downtime -- that
> doesn't
> > take special support, does it?
> >
> > In the normal case, don't you think the reads are the highest priority?
> Come
> > on, this is production.. And I'm telling you, your top clients have this
> > problem. Whats worse for a DB than downtime?
> >
> >
> > On Thu, May 10, 2012 at 7:37 AM, Scott Hernandez <
> scotthernandez-***@public.gmane.org>
> > wrote:
> >>
> >> A simple example is a system where they want the balancer to run as
> >> quickly as possible at low traffic times to balancer data and don't
> >> care about the impact of (user) reads. This can be a big deal when you
> >> are already under-provisioned and need to add a new shard to handle
> >> users more effectively -- it has come up a few times.
> >>
> >> On Thu, May 10, 2012 at 7:33 AM, Prasanna .S <myprasanna-***@public.gmane.org>
> wrote:
> >> > Scott,
> >> >
> >> >> While it might be paramount for some situations/use-cases to
> >> >> complete all
> >> >> reads before a move starts, it might not be for others (which I
> >> > have seen).
> >> >
> >> > Can you please explain the "might not be for others" part? Can you
> give
> >> > me
> >> > an example?
> >> >
> >> >
> >> > On Thu, May 10, 2012 at 7:19 AM, Scott Hernandez
> >> > <scotthernandez-***@public.gmane.org>
> >> > wrote:
> >> >>
> >> >> There are lots of ways to look at this with some competing goals.
> >> >> While it might be paramount for some situations/use-cases to
> complete
> >> >> all reads before a move starts, it might not be for others (which I
> >> >> have seen). This fix will allow reads (and other writes) to get
> >> >> processes while chunks moves happen, period. The issue of locking,
> and
> >> >> contentions on those locks, is very important and there are lots of
> >> >> improvements being made which help across the boards.
> >> >>
> >> >> Feel free to add comments or create new issues with your ideas of
> how
> >> >> your system would like things to best behave:
> >> >> http://jira.mongodb.org/browse/server
> >> >>
> >> >> On Wed, May 9, 2012 at 7:46 PM, Prasanna .S <myprasanna-***@public.gmane.org>
> >> >> wrote:
> >> >> > Scott,
> >> >> >
> >> >> > The below are heuristics, and not guarantees. What you want from a
> DB
> >> >> > stand-point is the latter.
> >> >> >
> >> >> > The delay between moves, won't provide a guarantee, unless you
> make
> >> >> > sure
> >> >> > that
> >> >> > (1) the next iteration of moveChunk, does not start unless all
> >> >> > pending
> >> >> > reads
> >> >> > complete.
> >> >> > (2) Each iteration takes less than 1second.
> >> >> >
> >> >> > Thoughts?
> >> >> >
> >> >> >
> >> >> > On Wed, May 9, 2012 at 6:18 PM, Scott Hernandez
> >> >> > <scotthernandez-***@public.gmane.org>
> >> >> > wrote:
> >> >> >>
> >> >> >> The logic there is to do each document write, during the migrate,
> in
> >> >> >> a
> >> >> >> separate lock so other operations can be interleaved. We have
> seen
> >> >> >> this change dramatically reduce the constant lock percentage and
> >> >> >> allow
> >> >> >> reads, and other writes through during a initial move.
> >> >> >>
> >> >> >> There is already a delay between moves which doesn't seem to be
> your
> >> >> >> problem.
> >> >> >>
> >> >> >> On Wed, May 9, 2012 at 5:37 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>
> >> >> >> wrote:
> >> >> >> > Scott,
> >> >> >> >
> >> >> >> > I tried to understand what's going on there. But with a quick
> >> >> >> > glance,
> >> >> >> > I
> >> >> >> > assume, a lock was being held longer than necessary and was
> fixed.
> >> >> >> > It's
> >> >> >> > unclear how this can help me. The only way to reliably make
> sure
> >> >> >> > the
> >> >> >> > site
> >> >> >> > keeps running while a moveChunk is executing, is if, you yield
> >> >> >> > every
> >> >> >> > second
> >> >> >> > within moveChunk and make sure all pending reads complete,
> before
> >> >> >> > the
> >> >> >> > next
> >> >> >> > moveChunk period starts.
> >> >> >> >
> >> >> >> > Are you guys doing it, or planning to do it? Is there any other
> >> >> >> > high-level
> >> >> >> > way to guarantee the reads can be answered within a couple of
> >> >> >> > seconds?
> >> >> >> >
> >> >> >> > Thanks
> >> >> >> >
> >> >> >> >
> >> >> >> > On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez
> >> >> >> > <scotthernandez-***@public.gmane.org>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> Okay, looking at the receiving shard mongo55, and donor shard
> >> >> >> >> mongo43,
> >> >> >> >> it look like the fix in 2.0.6 (due out in the next two weeks
> of
> >> >> >> >> so)
> >> >> >> >> will help you out a lot.
> >> >> >> >>
> >> >> >> >> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S
> >> >> >> >> <myprasanna-***@public.gmane.org>
> >> >> >> >> wrote:
> >> >> >> >> > 4:37 the balancer has started and disk shot up.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S
> >> >> >> >> > <myprasanna-***@public.gmane.org>
> >> >> >> >> > wrote:
> >> >> >> >> >>
> >> >> >> >> >> Hey Scott,
> >> >> >> >> >>
> >> >> >> >> >> I'm going to do this now. Will start it and stop it quickly
> >> >> >> >> >> and
> >> >> >> >> >> give
> >> >> >> >> >> you
> >> >> >> >> >> the exact times.
> >> >> >> >> >>
> >> >> >> >> >> Thanks,
> >> >> >> >> >> -Prasanna
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
> >> >> >> >> >> <scotthernandez-***@public.gmane.org>
> >> >> >> >> >> wrote:
> >> >> >> >> >>>
> >> >> >> >> >>> Those values are in blocks (512 bytes), not MB.
> >> >> >> >> >>>
> >> >> >> >> >>> We have an update in the upcoming 2.0.6 stable release
> which
> >> >> >> >> >>> will
> >> >> >> >> >>> help
> >> >> >> >> >>> with migrations a bit.
> >> >> >> >> >>> https://jira.mongodb.org/browse/SERVER-5303
> >> >> >> >> >>> You
> >> >> >> >> >>> can click on the version like to see other important
> fixes.
> >> >> >> >> >>>
> >> >> >> >> >>> Can you do a few migrates tonight so we can see the
> >> >> >> >> >>> stats/behavior
> >> >> >> >> >>> in
> >> >> >> >> >>> MMS? Please let us know when you do this so we can take a
> >> >> >> >> >>> look
> >> >> >> >> >>> at
> >> >> >> >> >>> the
> >> >> >> >> >>> data to see if this fix will help.
> >> >> >> >> >>>
> >> >> >> >> >>>
> >> >> >> >> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S
> >> >> >> >> >>> <myprasanna-***@public.gmane.org>
> >> >> >> >> >>> wrote:
> >> >> >> >> >>> > Hey Dan,
> >> >> >> >> >>> >
> >> >> >> >> >>> > Read ahead settings are the defaults (256
> >> >> >> >> >>> > MB): http://pastebin.com/MHc8s2fq
> >> >> >> >> >>> > Let me know if you need something else.
> >> >> >> >> >>> >
> >> >> >> >> >>> > @Elliot,
> >> >> >> >> >>> >
> >> >> >> >> >>> > I spoke with some of my friends at foursquare and it
> turns
> >> >> >> >> >>> > out
> >> >> >> >> >>> > they
> >> >> >> >> >>> > face the
> >> >> >> >> >>> > same issue as well. I've had my other friends who run
> >> >> >> >> >>> > companies >
> >> >> >> >> >>> > $100M
> >> >> >> >> >>> > market cap (whom I'd rather not name in this list), tell
> me
> >> >> >> >> >>> > about
> >> >> >> >> >>> > their
> >> >> >> >> >>> > mongo problems in the past few days. This is pretty
> serious
> >> >> >> >> >>> > stuff
> >> >> >> >> >>> > and
> >> >> >> >> >>> > will
> >> >> >> >> >>> > affect if someone wants to choose mongo as their primary
> >> >> >> >> >>> > store
> >> >> >> >> >>> > or
> >> >> >> >> >>> > not.
> >> >> >> >> >>> > To
> >> >> >> >> >>> > not have sharding is one thing, but to promise it and
> being
> >> >> >> >> >>> > stuck
> >> >> >> >> >>> > in
> >> >> >> >> >>> > production is another.
> >> >> >> >> >>> >
> >> >> >> >> >>> > I really thank you guys and appreciate the
> responsiveness
> >> >> >> >> >>> > so
> >> >> >> >> >>> > far.
> >> >> >> >> >>> > Happy
> >> >> >> >> >>> > to
> >> >> >> >> >>> > work together and hope we can resolve this.
> >> >> >> >> >>> >
> >> >> >> >> >>> > Thanks so much,
> >> >> >> >> >>> > -Prasanna
> >> >> >> >> >>> >
> >> >> >> >> >>> >
> >> >> >> >> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette <
> dan-***@public.gmane.org>
> >> >> >> >> >>> > wrote:
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> Looks like the docs in that collection
> >> >> >> >> >>> >> (lal_production.facebook_names)
> >> >> >> >> >>> >> are
> >> >> >> >> >>> >> small and likely distributed throughout the disk.
> >> >> >> >> >>> >> Can you check your disk readahead setting?
> >> >> >> >> >>> >> Please run and post the results of blockev --report.
> >> >> >> >> >>> >>
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S
> >> >> >> >> >>> >> <myprasanna-***@public.gmane.org>
> >> >> >> >> >>> >> wrote:
> >> >> >> >> >>> >>>
> >> >> >> >> >>> >>> Yes, indeed.
> >> >> >> >> >>> >>>
> >> >> >> >> >>> >>> Username: arnold-***@public.gmane.org
> >> >> >> >> >>> >>> Machine: mongo55
> >> >> >> >> >>> >>>
> >> >> >> >> >>> >>>
> >> >> >> >> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz
> >> >> >> >> >>> >>> <eliot-***@public.gmane.org>
> >> >> >> >> >>> >>> wrote:
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>> Are these machines in MMS?
> >> >> >> >> >>> >>>> Hard to see much in the pasted stats.
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
> >> >> >> >> >>> >>>> <myprasanna-***@public.gmane.org>
> >> >> >> >> >>> >>>> wrote:
> >> >> >> >> >>> >>>> > Raid 10 on 8 disks.
> >> >> >> >> >>> >>>> >
> >> >> >> >> >>> >>>> > MongoDB shell version: 2.0.4
> >> >> >> >> >>> >>>> > Tue May 8 09:02:29 [initandlisten] db version
> v2.0.4,
> >> >> >> >> >>> >>>> > pdfile
> >> >> >> >> >>> >>>> > version
> >> >> >> >> >>> >>>> > 4.5
> >> >> >> >> >>> >>>> >
> >> >> >> >> >>> >>>> >
> >> >> >> >> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
> >> >> >> >> >>> >>>> > <eliot-***@public.gmane.org>
> >> >> >> >> >>> >>>> > wrote:
> >> >> >> >> >>> >>>> >>
> >> >> >> >> >>> >>>> >> What version of mongo and what kind of ebs raid?
> >> >> >> >> >>> >>>> >>
> >> >> >> >> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
> >> >> >> >> >>> >>>> >> <myprasanna-***@public.gmane.org>
> >> >> >> >> >>> >>>> >> wrote:
> >> >> >> >> >>> >>>> >> > And yes, disabling the balancer, brings the
> write
> >> >> >> >> >>> >>>> >> > rate
> >> >> >> >> >>> >>>> >> > back
> >> >> >> >> >>> >>>> >> > to
> >> >> >> >> >>> >>>> >> > zero.
> >> >> >> >> >>> >>>> >> > Although after 4 minutes!
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat
> >> >> >> >> >>> >>>> >> > wrote:
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> I don't think it goes into deadlock. Can you
> try
> >> >> >> >> >>> >>>> >> >> disabling
> >> >> >> >> >>> >>>> >> >> the
> >> >> >> >> >>> >>>> >> >> balancer
> >> >> >> >> >>> >>>> >> >> and see whether the disk write rate going down
> or
> >> >> >> >> >>> >>>> >> >> not.
> >> >> >> >> >>> >>>> >> >> ________________________________
> >> >> >> >> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
> >> >> >> >> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
> >> >> >> >> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
> >> >> >> >> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
> >> >> >> >> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
> >> >> >> >> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>;
> >> >> >> >> >>> >>>> >> >> Hrushikesh
> >> >> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
> >> >> >> >> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> Hey guys,
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> We sharded mongo as per the docs. The issue is
> the
> >> >> >> >> >>> >>>> >> >> balancer
> >> >> >> >> >>> >>>> >> >> is
> >> >> >> >> >>> >>>> >> >> not
> >> >> >> >> >>> >>>> >> >> being
> >> >> >> >> >>> >>>> >> >> very clever and not throttling at all. Now we
> are
> >> >> >> >> >>> >>>> >> >> stuck
> >> >> >> >> >>> >>>> >> >> with a
> >> >> >> >> >>> >>>> >> >> half-sharded
> >> >> >> >> >>> >>>> >> >> setup and we can't turn on the balancer because
> >> >> >> >> >>> >>>> >> >> it'd
> >> >> >> >> >>> >>>> >> >> bring
> >> >> >> >> >>> >>>> >> >> the
> >> >> >> >> >>> >>>> >> >> site
> >> >> >> >> >>> >>>> >> >> down.
> >> >> >> >> >>> >>>> >> >> All read queries timeout when the disk write
> rate
> >> >> >> >> >>> >>>> >> >> is
> >> >> >> >> >>> >>>> >> >> 100%.
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> Is there a way out of this mess at all? From a
> >> >> >> >> >>> >>>> >> >> high-level,
> >> >> >> >> >>> >>>> >> >> mongo
> >> >> >> >> >>> >>>> >> >> could
> >> >> >> >> >>> >>>> >> >> have handled this by any of the following
> means:
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
> >> >> >> >> >>> >>>> >> >> 2. Throttle and replicate only as long as the
> disk
> >> >> >> >> >>> >>>> >> >> util
> >> >> >> >> >>> >>>> >> >> is
> >> >> >> >> >>> >>>> >> >> <
> >> >> >> >> >>> >>>> >> >> 50%
> >> >> >> >> >>> >>>> >> >> or so.
> >> >> >> >> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads
> and
> >> >> >> >> >>> >>>> >> >> writes
> >> >> >> >> >>> >>>> >> >> come
> >> >> >> >> >>> >>>> >> >> in-between,
> >> >> >> >> >>> >>>> >> >> consider them as higher priority, so that a
> >> >> >> >> >>> >>>> >> >> read/write
> >> >> >> >> >>> >>>> >> >> is
> >> >> >> >> >>> >>>> >> >> never
> >> >> >> >> >>> >>>> >> >> waiting
> >> >> >> >> >>> >>>> >> >> for
> >> >> >> >> >>> >>>> >> >> more than a second.
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> Have you guys heard of this problem before?
> >> >> >> >> >>> >>>> >> >> Because
> >> >> >> >> >>> >>>> >> >> this
> >> >> >> >> >>> >>>> >> >> really
> >> >> >> >> >>> >>>> >> >> make
> >> >> >> >> >>> >>>> >> >> sharding impossible. I'm not sure what others
> are
> >> >> >> >> >>> >>>> >> >> doing
> >> >> >> >> >>> >>>> >> >> in
> >> >> >> >> >>> >>>> >> >> production
> >> >> >> >> >>> >>>> >> >> to
> >> >> >> >> >>> >>>> >> >> work around this.
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> Thanks,
> >> >> >> >> >>> >>>> >> >> -Prasanna
> >> >> >> >> >>> >>>> >> >> Co-Founder & CTO, lal.com
> >> >> >> >> >>> >>>> >> >>
> >> >> >> >> >>> >>>> >> >> --
> >> >> >> >> >>> >>>> >> >> You received this message because you are
> >> >> >> >> >>> >>>> >> >> subscribed
> >> >> >> >> >>> >>>> >> >> to
> >> >> >> >> >>> >>>> >> >> the
> >> >> >> >> >>> >>>> >> >> Google
> >> >> >> >> >>> >>>> >> >> Groups
> >> >> >> >> >>> >>>> >> >> "mongodb-user" group.
> >> >> >> >> >>> >>>> >> >> To post to this group, send email to
> >> >> >> >> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> >> To unsubscribe from this group, send email to
> >> >> >> >> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> >> For more options, visit this group at
> >> >> >> >> >>> >>>> >> >>
> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> > --
> >> >> >> >> >>> >>>> >> > You received this message because you are
> >> >> >> >> >>> >>>> >> > subscribed
> >> >> >> >> >>> >>>> >> > to
> >> >> >> >> >>> >>>> >> > the
> >> >> >> >> >>> >>>> >> > Google
> >> >> >> >> >>> >>>> >> > Groups
> >> >> >> >> >>> >>>> >> > "mongodb-user" group.
> >> >> >> >> >>> >>>> >> > To view this discussion on the web visit
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> >
> https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
> >> >> >> >> >>> >>>> >> >
> >> >> >> >> >>> >>>> >> > To post to this group, send email to
> >> >> >> >> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to
> >> >> >> >> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> > For more options, visit this group at
> >> >> >> >> >>> >>>> >> >
> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >>>> >>
> >> >> >> >> >>> >>>> >> --
> >> >> >> >> >>> >>>> >> You received this message because you are
> subscribed
> >> >> >> >> >>> >>>> >> to
> >> >> >> >> >>> >>>> >> the
> >> >> >> >> >>> >>>> >> Google
> >> >> >> >> >>> >>>> >> Groups
> >> >> >> >> >>> >>>> >> "mongodb-user" group.
> >> >> >> >> >>> >>>> >> To post to this group, send email to
> >> >> >> >> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to
> >> >> >> >> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> >> For more options, visit this group at
> >> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>
> >> >> >> >> >>> >>>> >>
> >> >> >> >> >>> >>>> >
> >> >> >> >> >>> >>>> > --
> >> >> >> >> >>> >>>> > You received this message because you are
> subscribed
> >> >> >> >> >>> >>>> > to
> >> >> >> >> >>> >>>> > the
> >> >> >> >> >>> >>>> > Google
> >> >> >> >> >>> >>>> > Groups
> >> >> >> >> >>> >>>> > "mongodb-user" group.
> >> >> >> >> >>> >>>> > To post to this group, send email to
> >> >> >> >> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> > To unsubscribe from this group, send email to
> >> >> >> >> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> > For more options, visit this group at
> >> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>> --
> >> >> >> >> >>> >>>> You received this message because you are subscribed
> to
> >> >> >> >> >>> >>>> the
> >> >> >> >> >>> >>>> Google
> >> >> >> >> >>> >>>> Groups "mongodb-user" group.
> >> >> >> >> >>> >>>> To post to this group, send email to
> >> >> >> >> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> To unsubscribe from this group, send email to
> >> >> >> >> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>>> For more options, visit this group at
> >> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >>>>
> >> >> >> >> >>> >>>
> >> >> >> >> >>> >>> --
> >> >> >> >> >>> >>> You received this message because you are subscribed
> to
> >> >> >> >> >>> >>> the
> >> >> >> >> >>> >>> Google
> >> >> >> >> >>> >>> Groups
> >> >> >> >> >>> >>> "mongodb-user" group.
> >> >> >> >> >>> >>> To post to this group, send email to
> >> >> >> >> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>> To unsubscribe from this group, send email to
> >> >> >> >> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >>> For more options, visit this group at
> >> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >>
> >> >> >> >> >>> >>
> >> >> >> >> >>> >> --
> >> >> >> >> >>> >> You received this message because you are subscribed to
> >> >> >> >> >>> >> the
> >> >> >> >> >>> >> Google
> >> >> >> >> >>> >> Groups
> >> >> >> >> >>> >> "mongodb-user" group.
> >> >> >> >> >>> >> To post to this group, send email to
> >> >> >> >> >>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >> To unsubscribe from this group, send email to
> >> >> >> >> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> >> For more options, visit this group at
> >> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>> >
> >> >> >> >> >>> >
> >> >> >> >> >>> > --
> >> >> >> >> >>> > You received this message because you are subscribed to
> the
> >> >> >> >> >>> > Google
> >> >> >> >> >>> > Groups
> >> >> >> >> >>> > "mongodb-user" group.
> >> >> >> >> >>> > To post to this group, send email to
> >> >> >> >> >>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> > To unsubscribe from this group, send email to
> >> >> >> >> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> > For more options, visit this group at
> >> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>>
> >> >> >> >> >>> --
> >> >> >> >> >>> You received this message because you are subscribed to
> the
> >> >> >> >> >>> Google
> >> >> >> >> >>> Groups
> >> >> >> >> >>> "mongodb-user" group.
> >> >> >> >> >>> To post to this group, send email to
> >> >> >> >> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> To unsubscribe from this group, send email to
> >> >> >> >> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> >>> For more options, visit this group at
> >> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >> >>>
> >> >> >> >> >>
> >> >> >> >> >
> >> >> >> >> > --
> >> >> >> >> > You received this message because you are subscribed to the
> >> >> >> >> > Google
> >> >> >> >> > Groups
> >> >> >> >> > "mongodb-user" group.
> >> >> >> >> > To post to this group, send email to
> >> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> > To unsubscribe from this group, send email to
> >> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> > For more options, visit this group at
> >> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >>
> >> >> >> >> --
> >> >> >> >> You received this message because you are subscribed to the
> >> >> >> >> Google
> >> >> >> >> Groups
> >> >> >> >> "mongodb-user" group.
> >> >> >> >> To post to this group, send email to
> >> >> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> To unsubscribe from this group, send email to
> >> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> >> For more options, visit this group at
> >> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >> >>
> >> >> >> >
> >> >> >> > --
> >> >> >> > You received this message because you are subscribed to the
> Google
> >> >> >> > Groups
> >> >> >> > "mongodb-user" group.
> >> >> >> > To post to this group, send email to
> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> > To unsubscribe from this group, send email to
> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> > For more options, visit this group at
> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>
> >> >> >> --
> >> >> >> You received this message because you are subscribed to the
> Google
> >> >> >> Groups
> >> >> >> "mongodb-user" group.
> >> >> >> To post to this group, send email to
> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> To unsubscribe from this group, send email to
> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> >> For more options, visit this group at
> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >> >>
> >> >> >
> >> >> > --
> >> >> > You received this message because you are subscribed to the Google
> >> >> > Groups
> >> >> > "mongodb-user" group.
> >> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>
> >> >> > To unsubscribe from this group, send email to
> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> > For more options, visit this group at
> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >> >>
> >> >> --
> >> >> You received this message because you are subscribed to the Google
> >> >> Groups
> >> >> "mongodb-user" group.
> >> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> To unsubscribe from this group, send email to
> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> For more options, visit this group at
> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >>
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups
> >> > "mongodb-user" group.
> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> > To unsubscribe from this group, send email to
> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> > For more options, visit this group at
> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "mongodb-user" group.
> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> To unsubscribe from this group, send email to
> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> For more options, visit this group at
> >> http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > To unsubscribe from this group, send email to
> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > For more options, visit this group at
> > http://groups.google.com/group/mongodb-user?hl=en.
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To view this discussion on the web visit https://groups.google.com/d/msg/mongodb-user/-/YRlv9SXrV6QJ.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Scott Hernandez
2012-05-10 19:00:17 UTC
Permalink
Please give 2.0.6 a try when it comes out in a week or two. If you
want to test now you can use the dev/unstable 2.1.1 release.

On Thu, May 10, 2012 at 11:13 AM, Shi Shei <QTRAURFUICIN-geNee64TY+gS+***@public.gmane.org> wrote:
> Put me in the list as well. We are suffering from exactly the same issues
> that Prasanna is describing.
> Mongo pretends to be fast with commodity hardware when you use sharding but
> even with sharding and high performance hardware we don't see it.
>
> Have a look at our MMS (group name "idealo", all hosts containing
> "offerstore"). Profiling is enabled for queries taking longer than 3 seconds
> - and there are a lot of them!
> We followed and have mostly applied 10gens production rules but the results
> are still invisible.
> Sorry to say but with such expensive hardware, mongo performance is not that
> good as expected.
>
> If you have some pointers, we'll gladly follow your instructions. However,
> just waiting for the next and still next mongo release is quite
> demotivating.
>
> On Thursday, May 10, 2012 5:04:33 PM UTC+2, Scott Hernandez wrote:
>>
>> You seem to looking at this as a binary result. Slow reads do not mean
>> downtime. There are many use-cases where many times performance
>> degradation during maintenance windows is acceptable but downtime
>> (stopping reads) is not.
>>
>> We are working with everyone to make sure they can get the best
>> performance and uptime for their systems. If you have more specific
>> areas where you are seeing issue, please create a new thread/post or
>> jira issue with the problem (or vote/watch for existing ones).
>>
>> Nobody thinks the system is perfect as is (how could it be when there
>> are so many different use cases), and there are lots of improvements,
>> fixes and changes going in to address these areas for improvement.
>>
>>
>> On Thu, May 10, 2012 at 7:44 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>> > That use case can easily be supported by just shutting off the reads
>> > from
>> > the application. Basically they are anyways taking downtime -- that
>> > doesn't
>> > take special support, does it?
>> >
>> > In the normal case, don't you think the reads are the highest priority?
>> > Come
>> > on, this is production.. And I'm telling you, your top clients have this
>> > problem. Whats worse for a DB than downtime?
>> >
>> >
>> > On Thu, May 10, 2012 at 7:37 AM, Scott Hernandez
>> > <scotthernandez-***@public.gmane.org>
>> > wrote:
>> >>
>> >> A simple example is a system where they want the balancer to run as
>> >> quickly as possible at low traffic times to balancer data and don't
>> >> care about the impact of (user) reads. This can be a big deal when you
>> >> are already under-provisioned and need to add a new shard to handle
>> >> users more effectively -- it has come up a few times.
>> >>
>> >> On Thu, May 10, 2012 at 7:33 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> wrote:
>> >> > Scott,
>> >> >
>> >> >> While it might be paramount for some situations/use-cases to
>> >> >> complete all
>> >> >> reads before a move starts, it might not be for others (which I
>> >> > have seen).
>> >> >
>> >> > Can you please explain the "might not be for others" part? Can you
>> >> > give
>> >> > me
>> >> > an example?
>> >> >
>> >> >
>> >> > On Thu, May 10, 2012 at 7:19 AM, Scott Hernandez
>> >> > <scotthernandez-***@public.gmane.org>
>> >> > wrote:
>> >> >>
>> >> >> There are lots of ways to look at this with some competing goals.
>> >> >> While it might be paramount for some situations/use-cases to
>> >> >> complete
>> >> >> all reads before a move starts, it might not be for others (which I
>> >> >> have seen). This fix will allow reads (and other writes) to get
>> >> >> processes while chunks moves happen, period. The issue of locking,
>> >> >> and
>> >> >> contentions on those locks, is very important and there are lots of
>> >> >> improvements being made which help across the boards.
>> >> >>
>> >> >> Feel free to add comments or create new issues with your ideas of
>> >> >> how
>> >> >> your system would like things to best behave:
>> >> >> http://jira.mongodb.org/browse/server
>> >> >>
>> >> >> On Wed, May 9, 2012 at 7:46 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> >> wrote:
>> >> >> > Scott,
>> >> >> >
>> >> >> > The below are heuristics, and not guarantees. What you want from a
>> >> >> > DB
>> >> >> > stand-point is the latter.
>> >> >> >
>> >> >> > The delay between moves, won't provide a guarantee, unless you
>> >> >> > make
>> >> >> > sure
>> >> >> > that
>> >> >> > (1) the next iteration of moveChunk, does not start unless all
>> >> >> > pending
>> >> >> > reads
>> >> >> > complete.
>> >> >> > (2) Each iteration takes less than 1second.
>> >> >> >
>> >> >> > Thoughts?
>> >> >> >
>> >> >> >
>> >> >> > On Wed, May 9, 2012 at 6:18 PM, Scott Hernandez
>> >> >> > <scotthernandez-***@public.gmane.org>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> The logic there is to do each document write, during the migrate,
>> >> >> >> in
>> >> >> >> a
>> >> >> >> separate lock so other operations can be interleaved. We have
>> >> >> >> seen
>> >> >> >> this change dramatically reduce the constant lock percentage and
>> >> >> >> allow
>> >> >> >> reads, and other writes through during a initial move.
>> >> >> >>
>> >> >> >> There is already a delay between moves which doesn't seem to be
>> >> >> >> your
>> >> >> >> problem.
>> >> >> >>
>> >> >> >> On Wed, May 9, 2012 at 5:37 PM, Prasanna .S
>> >> >> >> <myprasanna-***@public.gmane.org>
>> >> >> >> wrote:
>> >> >> >> > Scott,
>> >> >> >> >
>> >> >> >> > I tried to understand what's going on there. But with a quick
>> >> >> >> > glance,
>> >> >> >> > I
>> >> >> >> > assume, a lock was being held longer than necessary and was
>> >> >> >> > fixed.
>> >> >> >> > It's
>> >> >> >> > unclear how this can help me. The only way to reliably make
>> >> >> >> > sure
>> >> >> >> > the
>> >> >> >> > site
>> >> >> >> > keeps running while a moveChunk is executing, is if, you yield
>> >> >> >> > every
>> >> >> >> > second
>> >> >> >> > within moveChunk and make sure all pending reads complete,
>> >> >> >> > before
>> >> >> >> > the
>> >> >> >> > next
>> >> >> >> > moveChunk period starts.
>> >> >> >> >
>> >> >> >> > Are you guys doing it, or planning to do it? Is there any other
>> >> >> >> > high-level
>> >> >> >> > way to guarantee the reads can be answered within a couple of
>> >> >> >> > seconds?
>> >> >> >> >
>> >> >> >> > Thanks
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez
>> >> >> >> > <scotthernandez-***@public.gmane.org>
>> >> >> >> > wrote:
>> >> >> >> >>
>> >> >> >> >> Okay, looking at the receiving shard mongo55, and donor shard
>> >> >> >> >> mongo43,
>> >> >> >> >> it look like the fix in 2.0.6 (due out in the next two weeks
>> >> >> >> >> of
>> >> >> >> >> so)
>> >> >> >> >> will help you out a lot.
>> >> >> >> >>
>> >> >> >> >> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S
>> >> >> >> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> wrote:
>> >> >> >> >> > 4:37 the balancer has started and disk shot up.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S
>> >> >> >> >> > <myprasanna-***@public.gmane.org>
>> >> >> >> >> > wrote:
>> >> >> >> >> >>
>> >> >> >> >> >> Hey Scott,
>> >> >> >> >> >>
>> >> >> >> >> >> I'm going to do this now. Will start it and stop it quickly
>> >> >> >> >> >> and
>> >> >> >> >> >> give
>> >> >> >> >> >> you
>> >> >> >> >> >> the exact times.
>> >> >> >> >> >>
>> >> >> >> >> >> Thanks,
>> >> >> >> >> >> -Prasanna
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
>> >> >> >> >> >> <scotthernandez-***@public.gmane.org>
>> >> >> >> >> >> wrote:
>> >> >> >> >> >>>
>> >> >> >> >> >>> Those values are in blocks (512 bytes), not MB.
>> >> >> >> >> >>>
>> >> >> >> >> >>> We have an update in the upcoming 2.0.6 stable release
>> >> >> >> >> >>> which
>> >> >> >> >> >>> will
>> >> >> >> >> >>> help
>> >> >> >> >> >>> with migrations a bit.
>> >> >> >> >> >>> https://jira.mongodb.org/browse/SERVER-5303
>> >> >> >> >> >>> You
>> >> >> >> >> >>> can click on the version like to see other important
>> >> >> >> >> >>> fixes.
>> >> >> >> >> >>>
>> >> >> >> >> >>> Can you do a few migrates tonight so we can see the
>> >> >> >> >> >>> stats/behavior
>> >> >> >> >> >>> in
>> >> >> >> >> >>> MMS? Please let  us know when you do this so we can take a
>> >> >> >> >> >>> look
>> >> >> >> >> >>> at
>> >> >> >> >> >>> the
>> >> >> >> >> >>> data to see if this fix will help.
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S
>> >> >> >> >> >>> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> wrote:
>> >> >> >> >> >>> > Hey Dan,
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > Read ahead settings are the defaults (256
>> >> >> >> >> >>> > MB): http://pastebin.com/MHc8s2fq
>> >> >> >> >> >>> > Let me know if you need something else.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > @Elliot,
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > I spoke with some of my friends at foursquare and it
>> >> >> >> >> >>> > turns
>> >> >> >> >> >>> > out
>> >> >> >> >> >>> > they
>> >> >> >> >> >>> > face the
>> >> >> >> >> >>> > same issue as well. I've had my other friends who run
>> >> >> >> >> >>> > companies >
>> >> >> >> >> >>> > $100M
>> >> >> >> >> >>> > market cap (whom I'd rather not name in this list), tell
>> >> >> >> >> >>> > me
>> >> >> >> >> >>> > about
>> >> >> >> >> >>> > their
>> >> >> >> >> >>> > mongo problems in the past few days. This is pretty
>> >> >> >> >> >>> > serious
>> >> >> >> >> >>> > stuff
>> >> >> >> >> >>> > and
>> >> >> >> >> >>> > will
>> >> >> >> >> >>> > affect if someone wants to choose mongo as their primary
>> >> >> >> >> >>> > store
>> >> >> >> >> >>> > or
>> >> >> >> >> >>> > not.
>> >> >> >> >> >>> > To
>> >> >> >> >> >>> > not have sharding is one thing, but to promise it and
>> >> >> >> >> >>> > being
>> >> >> >> >> >>> > stuck
>> >> >> >> >> >>> > in
>> >> >> >> >> >>> > production is another.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > I really thank you guys and appreciate the
>> >> >> >> >> >>> > responsiveness
>> >> >> >> >> >>> > so
>> >> >> >> >> >>> > far.
>> >> >> >> >> >>> > Happy
>> >> >> >> >> >>> > to
>> >> >> >> >> >>> > work together and hope we can resolve this.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > Thanks so much,
>> >> >> >> >> >>> > -Prasanna
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette
>> >> >> >> >> >>> > <dan-***@public.gmane.org>
>> >> >> >> >> >>> > wrote:
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> Looks like the docs in that collection
>> >> >> >> >> >>> >> (lal_production.facebook_names)
>> >> >> >> >> >>> >> are
>> >> >> >> >> >>> >> small and likely distributed throughout the disk.
>> >> >> >> >> >>> >> Can you check your disk readahead setting?
>> >> >> >> >> >>> >> Please run and post the results of blockev --report.
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S
>> >> >> >> >> >>> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >> wrote:
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> Yes, indeed.
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> Username: arnold-***@public.gmane.org
>> >> >> >> >> >>> >>> Machine: mongo55
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz
>> >> >> >> >> >>> >>> <eliot-***@public.gmane.org>
>> >> >> >> >> >>> >>> wrote:
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> Are these machines in MMS?
>> >> >> >> >> >>> >>>> Hard to see much in the pasted stats.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
>> >> >> >> >> >>> >>>> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> wrote:
>> >> >> >> >> >>> >>>> > Raid 10 on 8 disks.
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > MongoDB shell version: 2.0.4
>> >> >> >> >> >>> >>>> > Tue May  8 09:02:29 [initandlisten] db version
>> >> >> >> >> >>> >>>> > v2.0.4,
>> >> >> >> >> >>> >>>> > pdfile
>> >> >> >> >> >>> >>>> > version
>> >> >> >> >> >>> >>>> > 4.5
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
>> >> >> >> >> >>> >>>> > <eliot-***@public.gmane.org>
>> >> >> >> >> >>> >>>> > wrote:
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> What version of mongo and what kind of ebs raid?
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
>> >> >> >> >> >>> >>>> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> wrote:
>> >> >> >> >> >>> >>>> >> > And yes, disabling the balancer, brings the
>> >> >> >> >> >>> >>>> >> > write
>> >> >> >> >> >>> >>>> >> > rate
>> >> >> >> >> >>> >>>> >> > back
>> >> >> >> >> >>> >>>> >> > to
>> >> >> >> >> >>> >>>> >> > zero.
>> >> >> >> >> >>> >>>> >> > Although after 4 minutes!
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat
>> >> >> >> >> >>> >>>> >> > wrote:
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> I don't think it goes into deadlock. Can you
>> >> >> >> >> >>> >>>> >> >> try
>> >> >> >> >> >>> >>>> >> >> disabling
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> balancer
>> >> >> >> >> >>> >>>> >> >> and see whether the disk write rate going down
>> >> >> >> >> >>> >>>> >> >> or
>> >> >> >> >> >>> >>>> >> >> not.
>> >> >> >> >> >>> >>>> >> >> ________________________________
>> >> >> >> >> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>> >> >> >> >> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>;
>> >> >> >> >> >>> >>>> >> >> Hrushikesh
>> >> >> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Hey guys,
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> We sharded mongo as per the docs. The issue is
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> balancer
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> not
>> >> >> >> >> >>> >>>> >> >> being
>> >> >> >> >> >>> >>>> >> >> very clever and not throttling at all. Now we
>> >> >> >> >> >>> >>>> >> >> are
>> >> >> >> >> >>> >>>> >> >> stuck
>> >> >> >> >> >>> >>>> >> >> with a
>> >> >> >> >> >>> >>>> >> >> half-sharded
>> >> >> >> >> >>> >>>> >> >> setup and we can't turn on the balancer because
>> >> >> >> >> >>> >>>> >> >> it'd
>> >> >> >> >> >>> >>>> >> >> bring
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> site
>> >> >> >> >> >>> >>>> >> >> down.
>> >> >> >> >> >>> >>>> >> >> All read queries timeout when the disk write
>> >> >> >> >> >>> >>>> >> >> rate
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> 100%.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Is there a way out of this mess at all? From a
>> >> >> >> >> >>> >>>> >> >> high-level,
>> >> >> >> >> >>> >>>> >> >> mongo
>> >> >> >> >> >>> >>>> >> >> could
>> >> >> >> >> >>> >>>> >> >> have handled this by any of the following
>> >> >> >> >> >>> >>>> >> >> means:
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
>> >> >> >> >> >>> >>>> >> >> 2. Throttle and replicate only as long as the
>> >> >> >> >> >>> >>>> >> >> disk
>> >> >> >> >> >>> >>>> >> >> util
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> <
>> >> >> >> >> >>> >>>> >> >> 50%
>> >> >> >> >> >>> >>>> >> >> or so.
>> >> >> >> >> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads
>> >> >> >> >> >>> >>>> >> >> and
>> >> >> >> >> >>> >>>> >> >> writes
>> >> >> >> >> >>> >>>> >> >> come
>> >> >> >> >> >>> >>>> >> >> in-between,
>> >> >> >> >> >>> >>>> >> >> consider them as higher priority, so that a
>> >> >> >> >> >>> >>>> >> >> read/write
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> never
>> >> >> >> >> >>> >>>> >> >> waiting
>> >> >> >> >> >>> >>>> >> >> for
>> >> >> >> >> >>> >>>> >> >> more than a second.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Have you guys heard of this problem before?
>> >> >> >> >> >>> >>>> >> >> Because
>> >> >> >> >> >>> >>>> >> >> this
>> >> >> >> >> >>> >>>> >> >> really
>> >> >> >> >> >>> >>>> >> >> make
>> >> >> >> >> >>> >>>> >> >> sharding impossible. I'm not sure what others
>> >> >> >> >> >>> >>>> >> >> are
>> >> >> >> >> >>> >>>> >> >> doing
>> >> >> >> >> >>> >>>> >> >> in
>> >> >> >> >> >>> >>>> >> >> production
>> >> >> >> >> >>> >>>> >> >> to
>> >> >> >> >> >>> >>>> >> >> work around this.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Thanks,
>> >> >> >> >> >>> >>>> >> >> -Prasanna
>> >> >> >> >> >>> >>>> >> >> Co-Founder & CTO, lal.com
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> --
>> >> >> >> >> >>> >>>> >> >> You received this message because you are
>> >> >> >> >> >>> >>>> >> >> subscribed
>> >> >> >> >> >>> >>>> >> >> to
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> Google
>> >> >> >> >> >>> >>>> >> >> Groups
>> >> >> >> >> >>> >>>> >> >> "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> >> To post to this group, send email to
>> >> >> >> >> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > --
>> >> >> >> >> >>> >>>> >> > You received this message because you are
>> >> >> >> >> >>> >>>> >> > subscribed
>> >> >> >> >> >>> >>>> >> > to
>> >> >> >> >> >>> >>>> >> > the
>> >> >> >> >> >>> >>>> >> > Google
>> >> >> >> >> >>> >>>> >> > Groups
>> >> >> >> >> >>> >>>> >> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> > To view this discussion on the web visit
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > To post to this group, send email to
>> >> >> >> >> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> > For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> --
>> >> >> >> >> >>> >>>> >> You received this message because you are
>> >> >> >> >> >>> >>>> >> subscribed
>> >> >> >> >> >>> >>>> >> to
>> >> >> >> >> >>> >>>> >> the
>> >> >> >> >> >>> >>>> >> Google
>> >> >> >> >> >>> >>>> >> Groups
>> >> >> >> >> >>> >>>> >> "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> To post to this group, send email to
>> >> >> >> >> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > --
>> >> >> >> >> >>> >>>> > You received this message because you are
>> >> >> >> >> >>> >>>> > subscribed
>> >> >> >> >> >>> >>>> > to
>> >> >> >> >> >>> >>>> > the
>> >> >> >> >> >>> >>>> > Google
>> >> >> >> >> >>> >>>> > Groups
>> >> >> >> >> >>> >>>> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> > To post to this group, send email to
>> >> >> >> >> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> > For more options, visit this group at
>> >> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> --
>> >> >> >> >> >>> >>>> You received this message because you are subscribed
>> >> >> >> >> >>> >>>> to
>> >> >> >> >> >>> >>>> the
>> >> >> >> >> >>> >>>> Google
>> >> >> >> >> >>> >>>> Groups "mongodb-user" group.
>> >> >> >> >> >>> >>>> To post to this group, send email to
>> >> >> >> >> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> For more options, visit this group at
>> >> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> --
>> >> >> >> >> >>> >>> You received this message because you are subscribed
>> >> >> >> >> >>> >>> to
>> >> >> >> >> >>> >>> the
>> >> >> >> >> >>> >>> Google
>> >> >> >> >> >>> >>> Groups
>> >> >> >> >> >>> >>> "mongodb-user" group.
>> >> >> >> >> >>> >>> To post to this group, send email to
>> >> >> >> >> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>> For more options, visit this group at
>> >> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> --
>> >> >> >> >> >>> >> You received this message because you are subscribed to
>> >> >> >> >> >>> >> the
>> >> >> >> >> >>> >> Google
>> >> >> >> >> >>> >> Groups
>> >> >> >> >> >>> >> "mongodb-user" group.
>> >> >> >> >> >>> >> To post to this group, send email to
>> >> >> >> >> >>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >> For more options, visit this group at
>> >> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > --
>> >> >> >> >> >>> > You received this message because you are subscribed to
>> >> >> >> >> >>> > the
>> >> >> >> >> >>> > Google
>> >> >> >> >> >>> > Groups
>> >> >> >> >> >>> > "mongodb-user" group.
>> >> >> >> >> >>> > To post to this group, send email to
>> >> >> >> >> >>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> > For more options, visit this group at
>> >> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>> --
>> >> >> >> >> >>> You received this message because you are subscribed to
>> >> >> >> >> >>> the
>> >> >> >> >> >>> Google
>> >> >> >> >> >>> Groups
>> >> >> >> >> >>> "mongodb-user" group.
>> >> >> >> >> >>> To post to this group, send email to
>> >> >> >> >> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> For more options, visit this group at
>> >> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>
>> >> >> >> >> >
>> >> >> >> >> > --
>> >> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> >> > Google
>> >> >> >> >> > Groups
>> >> >> >> >> > "mongodb-user" group.
>> >> >> >> >> > To post to this group, send email to
>> >> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> > To unsubscribe from this group, send email to
>> >> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> > For more options, visit this group at
>> >> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >> --
>> >> >> >> >> You received this message because you are subscribed to the
>> >> >> >> >> Google
>> >> >> >> >> Groups
>> >> >> >> >> "mongodb-user" group.
>> >> >> >> >> To post to this group, send email to
>> >> >> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> To unsubscribe from this group, send email to
>> >> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> For more options, visit this group at
>> >> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> > Google
>> >> >> >> > Groups
>> >> >> >> > "mongodb-user" group.
>> >> >> >> > To post to this group, send email to
>> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> > To unsubscribe from this group, send email to
>> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> > For more options, visit this group at
>> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >> --
>> >> >> >> You received this message because you are subscribed to the
>> >> >> >> Google
>> >> >> >> Groups
>> >> >> >> "mongodb-user" group.
>> >> >> >> To post to this group, send email to
>> >> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> To unsubscribe from this group, send email to
>> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> For more options, visit this group at
>> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "mongodb-user" group.
>> >> >> > To post to this group, send email to
>> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > To unsubscribe from this group, send email to
>> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "mongodb-user" group.
>> >> >> To post to this group, send email to mongodb-user-/***@public.gmane.org.
>> >> >> To unsubscribe from this group, send email to
>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > To unsubscribe from this group, send email to
>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> To unsubscribe from this group, send email to
>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > To unsubscribe from this group, send email to
>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
>
> On Thursday, May 10, 2012 5:04:33 PM UTC+2, Scott Hernandez wrote:
>>
>> You seem to looking at this as a binary result. Slow reads do not mean
>> downtime. There are many use-cases where many times performance
>> degradation during maintenance windows is acceptable but downtime
>> (stopping reads) is not.
>>
>> We are working with everyone to make sure they can get the best
>> performance and uptime for their systems. If you have more specific
>> areas where you are seeing issue, please create a new thread/post or
>> jira issue with the problem (or vote/watch for existing ones).
>>
>> Nobody thinks the system is perfect as is (how could it be when there
>> are so many different use cases), and there are lots of improvements,
>> fixes and changes going in to address these areas for improvement.
>>
>>
>> On Thu, May 10, 2012 at 7:44 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>> > That use case can easily be supported by just shutting off the reads
>> > from
>> > the application. Basically they are anyways taking downtime -- that
>> > doesn't
>> > take special support, does it?
>> >
>> > In the normal case, don't you think the reads are the highest priority?
>> > Come
>> > on, this is production.. And I'm telling you, your top clients have this
>> > problem. Whats worse for a DB than downtime?
>> >
>> >
>> > On Thu, May 10, 2012 at 7:37 AM, Scott Hernandez
>> > <scotthernandez-***@public.gmane.org>
>> > wrote:
>> >>
>> >> A simple example is a system where they want the balancer to run as
>> >> quickly as possible at low traffic times to balancer data and don't
>> >> care about the impact of (user) reads. This can be a big deal when you
>> >> are already under-provisioned and need to add a new shard to handle
>> >> users more effectively -- it has come up a few times.
>> >>
>> >> On Thu, May 10, 2012 at 7:33 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> wrote:
>> >> > Scott,
>> >> >
>> >> >> While it might be paramount for some situations/use-cases to
>> >> >> complete all
>> >> >> reads before a move starts, it might not be for others (which I
>> >> > have seen).
>> >> >
>> >> > Can you please explain the "might not be for others" part? Can you
>> >> > give
>> >> > me
>> >> > an example?
>> >> >
>> >> >
>> >> > On Thu, May 10, 2012 at 7:19 AM, Scott Hernandez
>> >> > <scotthernandez-***@public.gmane.org>
>> >> > wrote:
>> >> >>
>> >> >> There are lots of ways to look at this with some competing goals.
>> >> >> While it might be paramount for some situations/use-cases to
>> >> >> complete
>> >> >> all reads before a move starts, it might not be for others (which I
>> >> >> have seen). This fix will allow reads (and other writes) to get
>> >> >> processes while chunks moves happen, period. The issue of locking,
>> >> >> and
>> >> >> contentions on those locks, is very important and there are lots of
>> >> >> improvements being made which help across the boards.
>> >> >>
>> >> >> Feel free to add comments or create new issues with your ideas of
>> >> >> how
>> >> >> your system would like things to best behave:
>> >> >> http://jira.mongodb.org/browse/server
>> >> >>
>> >> >> On Wed, May 9, 2012 at 7:46 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> >> wrote:
>> >> >> > Scott,
>> >> >> >
>> >> >> > The below are heuristics, and not guarantees. What you want from a
>> >> >> > DB
>> >> >> > stand-point is the latter.
>> >> >> >
>> >> >> > The delay between moves, won't provide a guarantee, unless you
>> >> >> > make
>> >> >> > sure
>> >> >> > that
>> >> >> > (1) the next iteration of moveChunk, does not start unless all
>> >> >> > pending
>> >> >> > reads
>> >> >> > complete.
>> >> >> > (2) Each iteration takes less than 1second.
>> >> >> >
>> >> >> > Thoughts?
>> >> >> >
>> >> >> >
>> >> >> > On Wed, May 9, 2012 at 6:18 PM, Scott Hernandez
>> >> >> > <scotthernandez-***@public.gmane.org>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> The logic there is to do each document write, during the migrate,
>> >> >> >> in
>> >> >> >> a
>> >> >> >> separate lock so other operations can be interleaved. We have
>> >> >> >> seen
>> >> >> >> this change dramatically reduce the constant lock percentage and
>> >> >> >> allow
>> >> >> >> reads, and other writes through during a initial move.
>> >> >> >>
>> >> >> >> There is already a delay between moves which doesn't seem to be
>> >> >> >> your
>> >> >> >> problem.
>> >> >> >>
>> >> >> >> On Wed, May 9, 2012 at 5:37 PM, Prasanna .S
>> >> >> >> <myprasanna-***@public.gmane.org>
>> >> >> >> wrote:
>> >> >> >> > Scott,
>> >> >> >> >
>> >> >> >> > I tried to understand what's going on there. But with a quick
>> >> >> >> > glance,
>> >> >> >> > I
>> >> >> >> > assume, a lock was being held longer than necessary and was
>> >> >> >> > fixed.
>> >> >> >> > It's
>> >> >> >> > unclear how this can help me. The only way to reliably make
>> >> >> >> > sure
>> >> >> >> > the
>> >> >> >> > site
>> >> >> >> > keeps running while a moveChunk is executing, is if, you yield
>> >> >> >> > every
>> >> >> >> > second
>> >> >> >> > within moveChunk and make sure all pending reads complete,
>> >> >> >> > before
>> >> >> >> > the
>> >> >> >> > next
>> >> >> >> > moveChunk period starts.
>> >> >> >> >
>> >> >> >> > Are you guys doing it, or planning to do it? Is there any other
>> >> >> >> > high-level
>> >> >> >> > way to guarantee the reads can be answered within a couple of
>> >> >> >> > seconds?
>> >> >> >> >
>> >> >> >> > Thanks
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez
>> >> >> >> > <scotthernandez-***@public.gmane.org>
>> >> >> >> > wrote:
>> >> >> >> >>
>> >> >> >> >> Okay, looking at the receiving shard mongo55, and donor shard
>> >> >> >> >> mongo43,
>> >> >> >> >> it look like the fix in 2.0.6 (due out in the next two weeks
>> >> >> >> >> of
>> >> >> >> >> so)
>> >> >> >> >> will help you out a lot.
>> >> >> >> >>
>> >> >> >> >> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S
>> >> >> >> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> wrote:
>> >> >> >> >> > 4:37 the balancer has started and disk shot up.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S
>> >> >> >> >> > <myprasanna-***@public.gmane.org>
>> >> >> >> >> > wrote:
>> >> >> >> >> >>
>> >> >> >> >> >> Hey Scott,
>> >> >> >> >> >>
>> >> >> >> >> >> I'm going to do this now. Will start it and stop it quickly
>> >> >> >> >> >> and
>> >> >> >> >> >> give
>> >> >> >> >> >> you
>> >> >> >> >> >> the exact times.
>> >> >> >> >> >>
>> >> >> >> >> >> Thanks,
>> >> >> >> >> >> -Prasanna
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
>> >> >> >> >> >> <scotthernandez-***@public.gmane.org>
>> >> >> >> >> >> wrote:
>> >> >> >> >> >>>
>> >> >> >> >> >>> Those values are in blocks (512 bytes), not MB.
>> >> >> >> >> >>>
>> >> >> >> >> >>> We have an update in the upcoming 2.0.6 stable release
>> >> >> >> >> >>> which
>> >> >> >> >> >>> will
>> >> >> >> >> >>> help
>> >> >> >> >> >>> with migrations a bit.
>> >> >> >> >> >>> https://jira.mongodb.org/browse/SERVER-5303
>> >> >> >> >> >>> You
>> >> >> >> >> >>> can click on the version like to see other important
>> >> >> >> >> >>> fixes.
>> >> >> >> >> >>>
>> >> >> >> >> >>> Can you do a few migrates tonight so we can see the
>> >> >> >> >> >>> stats/behavior
>> >> >> >> >> >>> in
>> >> >> >> >> >>> MMS? Please let  us know when you do this so we can take a
>> >> >> >> >> >>> look
>> >> >> >> >> >>> at
>> >> >> >> >> >>> the
>> >> >> >> >> >>> data to see if this fix will help.
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S
>> >> >> >> >> >>> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> wrote:
>> >> >> >> >> >>> > Hey Dan,
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > Read ahead settings are the defaults (256
>> >> >> >> >> >>> > MB): http://pastebin.com/MHc8s2fq
>> >> >> >> >> >>> > Let me know if you need something else.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > @Elliot,
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > I spoke with some of my friends at foursquare and it
>> >> >> >> >> >>> > turns
>> >> >> >> >> >>> > out
>> >> >> >> >> >>> > they
>> >> >> >> >> >>> > face the
>> >> >> >> >> >>> > same issue as well. I've had my other friends who run
>> >> >> >> >> >>> > companies >
>> >> >> >> >> >>> > $100M
>> >> >> >> >> >>> > market cap (whom I'd rather not name in this list), tell
>> >> >> >> >> >>> > me
>> >> >> >> >> >>> > about
>> >> >> >> >> >>> > their
>> >> >> >> >> >>> > mongo problems in the past few days. This is pretty
>> >> >> >> >> >>> > serious
>> >> >> >> >> >>> > stuff
>> >> >> >> >> >>> > and
>> >> >> >> >> >>> > will
>> >> >> >> >> >>> > affect if someone wants to choose mongo as their primary
>> >> >> >> >> >>> > store
>> >> >> >> >> >>> > or
>> >> >> >> >> >>> > not.
>> >> >> >> >> >>> > To
>> >> >> >> >> >>> > not have sharding is one thing, but to promise it and
>> >> >> >> >> >>> > being
>> >> >> >> >> >>> > stuck
>> >> >> >> >> >>> > in
>> >> >> >> >> >>> > production is another.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > I really thank you guys and appreciate the
>> >> >> >> >> >>> > responsiveness
>> >> >> >> >> >>> > so
>> >> >> >> >> >>> > far.
>> >> >> >> >> >>> > Happy
>> >> >> >> >> >>> > to
>> >> >> >> >> >>> > work together and hope we can resolve this.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > Thanks so much,
>> >> >> >> >> >>> > -Prasanna
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette
>> >> >> >> >> >>> > <dan-***@public.gmane.org>
>> >> >> >> >> >>> > wrote:
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> Looks like the docs in that collection
>> >> >> >> >> >>> >> (lal_production.facebook_names)
>> >> >> >> >> >>> >> are
>> >> >> >> >> >>> >> small and likely distributed throughout the disk.
>> >> >> >> >> >>> >> Can you check your disk readahead setting?
>> >> >> >> >> >>> >> Please run and post the results of blockev --report.
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S
>> >> >> >> >> >>> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >> wrote:
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> Yes, indeed.
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> Username: arnold-***@public.gmane.org
>> >> >> >> >> >>> >>> Machine: mongo55
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz
>> >> >> >> >> >>> >>> <eliot-***@public.gmane.org>
>> >> >> >> >> >>> >>> wrote:
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> Are these machines in MMS?
>> >> >> >> >> >>> >>>> Hard to see much in the pasted stats.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
>> >> >> >> >> >>> >>>> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> wrote:
>> >> >> >> >> >>> >>>> > Raid 10 on 8 disks.
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > MongoDB shell version: 2.0.4
>> >> >> >> >> >>> >>>> > Tue May  8 09:02:29 [initandlisten] db version
>> >> >> >> >> >>> >>>> > v2.0.4,
>> >> >> >> >> >>> >>>> > pdfile
>> >> >> >> >> >>> >>>> > version
>> >> >> >> >> >>> >>>> > 4.5
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
>> >> >> >> >> >>> >>>> > <eliot-***@public.gmane.org>
>> >> >> >> >> >>> >>>> > wrote:
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> What version of mongo and what kind of ebs raid?
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
>> >> >> >> >> >>> >>>> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> wrote:
>> >> >> >> >> >>> >>>> >> > And yes, disabling the balancer, brings the
>> >> >> >> >> >>> >>>> >> > write
>> >> >> >> >> >>> >>>> >> > rate
>> >> >> >> >> >>> >>>> >> > back
>> >> >> >> >> >>> >>>> >> > to
>> >> >> >> >> >>> >>>> >> > zero.
>> >> >> >> >> >>> >>>> >> > Although after 4 minutes!
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat
>> >> >> >> >> >>> >>>> >> > wrote:
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> I don't think it goes into deadlock. Can you
>> >> >> >> >> >>> >>>> >> >> try
>> >> >> >> >> >>> >>>> >> >> disabling
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> balancer
>> >> >> >> >> >>> >>>> >> >> and see whether the disk write rate going down
>> >> >> >> >> >>> >>>> >> >> or
>> >> >> >> >> >>> >>>> >> >> not.
>> >> >> >> >> >>> >>>> >> >> ________________________________
>> >> >> >> >> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>> >> >> >> >> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>;
>> >> >> >> >> >>> >>>> >> >> Hrushikesh
>> >> >> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Hey guys,
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> We sharded mongo as per the docs. The issue is
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> balancer
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> not
>> >> >> >> >> >>> >>>> >> >> being
>> >> >> >> >> >>> >>>> >> >> very clever and not throttling at all. Now we
>> >> >> >> >> >>> >>>> >> >> are
>> >> >> >> >> >>> >>>> >> >> stuck
>> >> >> >> >> >>> >>>> >> >> with a
>> >> >> >> >> >>> >>>> >> >> half-sharded
>> >> >> >> >> >>> >>>> >> >> setup and we can't turn on the balancer because
>> >> >> >> >> >>> >>>> >> >> it'd
>> >> >> >> >> >>> >>>> >> >> bring
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> site
>> >> >> >> >> >>> >>>> >> >> down.
>> >> >> >> >> >>> >>>> >> >> All read queries timeout when the disk write
>> >> >> >> >> >>> >>>> >> >> rate
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> 100%.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Is there a way out of this mess at all? From a
>> >> >> >> >> >>> >>>> >> >> high-level,
>> >> >> >> >> >>> >>>> >> >> mongo
>> >> >> >> >> >>> >>>> >> >> could
>> >> >> >> >> >>> >>>> >> >> have handled this by any of the following
>> >> >> >> >> >>> >>>> >> >> means:
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
>> >> >> >> >> >>> >>>> >> >> 2. Throttle and replicate only as long as the
>> >> >> >> >> >>> >>>> >> >> disk
>> >> >> >> >> >>> >>>> >> >> util
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> <
>> >> >> >> >> >>> >>>> >> >> 50%
>> >> >> >> >> >>> >>>> >> >> or so.
>> >> >> >> >> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads
>> >> >> >> >> >>> >>>> >> >> and
>> >> >> >> >> >>> >>>> >> >> writes
>> >> >> >> >> >>> >>>> >> >> come
>> >> >> >> >> >>> >>>> >> >> in-between,
>> >> >> >> >> >>> >>>> >> >> consider them as higher priority, so that a
>> >> >> >> >> >>> >>>> >> >> read/write
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> never
>> >> >> >> >> >>> >>>> >> >> waiting
>> >> >> >> >> >>> >>>> >> >> for
>> >> >> >> >> >>> >>>> >> >> more than a second.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Have you guys heard of this problem before?
>> >> >> >> >> >>> >>>> >> >> Because
>> >> >> >> >> >>> >>>> >> >> this
>> >> >> >> >> >>> >>>> >> >> really
>> >> >> >> >> >>> >>>> >> >> make
>> >> >> >> >> >>> >>>> >> >> sharding impossible. I'm not sure what others
>> >> >> >> >> >>> >>>> >> >> are
>> >> >> >> >> >>> >>>> >> >> doing
>> >> >> >> >> >>> >>>> >> >> in
>> >> >> >> >> >>> >>>> >> >> production
>> >> >> >> >> >>> >>>> >> >> to
>> >> >> >> >> >>> >>>> >> >> work around this.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Thanks,
>> >> >> >> >> >>> >>>> >> >> -Prasanna
>> >> >> >> >> >>> >>>> >> >> Co-Founder & CTO, lal.com
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> --
>> >> >> >> >> >>> >>>> >> >> You received this message because you are
>> >> >> >> >> >>> >>>> >> >> subscribed
>> >> >> >> >> >>> >>>> >> >> to
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> Google
>> >> >> >> >> >>> >>>> >> >> Groups
>> >> >> >> >> >>> >>>> >> >> "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> >> To post to this group, send email to
>> >> >> >> >> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > --
>> >> >> >> >> >>> >>>> >> > You received this message because you are
>> >> >> >> >> >>> >>>> >> > subscribed
>> >> >> >> >> >>> >>>> >> > to
>> >> >> >> >> >>> >>>> >> > the
>> >> >> >> >> >>> >>>> >> > Google
>> >> >> >> >> >>> >>>> >> > Groups
>> >> >> >> >> >>> >>>> >> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> > To view this discussion on the web visit
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > To post to this group, send email to
>> >> >> >> >> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> > For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> --
>> >> >> >> >> >>> >>>> >> You received this message because you are
>> >> >> >> >> >>> >>>> >> subscribed
>> >> >> >> >> >>> >>>> >> to
>> >> >> >> >> >>> >>>> >> the
>> >> >> >> >> >>> >>>> >> Google
>> >> >> >> >> >>> >>>> >> Groups
>> >> >> >> >> >>> >>>> >> "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> To post to this group, send email to
>> >> >> >> >> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > --
>> >> >> >> >> >>> >>>> > You received this message because you are
>> >> >> >> >> >>> >>>> > subscribed
>> >> >> >> >> >>> >>>> > to
>> >> >> >> >> >>> >>>> > the
>> >> >> >> >> >>> >>>> > Google
>> >> >> >> >> >>> >>>> > Groups
>> >> >> >> >> >>> >>>> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> > To post to this group, send email to
>> >> >> >> >> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> > For more options, visit this group at
>> >> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> --
>> >> >> >> >> >>> >>>> You received this message because you are subscribed
>> >> >> >> >> >>> >>>> to
>> >> >> >> >> >>> >>>> the
>> >> >> >> >> >>> >>>> Google
>> >> >> >> >> >>> >>>> Groups "mongodb-user" group.
>> >> >> >> >> >>> >>>> To post to this group, send email to
>> >> >> >> >> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> For more options, visit this group at
>> >> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> --
>> >> >> >> >> >>> >>> You received this message because you are subscribed
>> >> >> >> >> >>> >>> to
>> >> >> >> >> >>> >>> the
>> >> >> >> >> >>> >>> Google
>> >> >> >> >> >>> >>> Groups
>> >> >> >> >> >>> >>> "mongodb-user" group.
>> >> >> >> >> >>> >>> To post to this group, send email to
>> >> >> >> >> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>> For more options, visit this group at
>> >> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> --
>> >> >> >> >> >>> >> You received this message because you are subscribed to
>> >> >> >> >> >>> >> the
>> >> >> >> >> >>> >> Google
>> >> >> >> >> >>> >> Groups
>> >> >> >> >> >>> >> "mongodb-user" group.
>> >> >> >> >> >>> >> To post to this group, send email to
>> >> >> >> >> >>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >> For more options, visit this group at
>> >> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > --
>> >> >> >> >> >>> > You received this message because you are subscribed to
>> >> >> >> >> >>> > the
>> >> >> >> >> >>> > Google
>> >> >> >> >> >>> > Groups
>> >> >> >> >> >>> > "mongodb-user" group.
>> >> >> >> >> >>> > To post to this group, send email to
>> >> >> >> >> >>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> > For more options, visit this group at
>> >> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>> --
>> >> >> >> >> >>> You received this message because you are subscribed to
>> >> >> >> >> >>> the
>> >> >> >> >> >>> Google
>> >> >> >> >> >>> Groups
>> >> >> >> >> >>> "mongodb-user" group.
>> >> >> >> >> >>> To post to this group, send email to
>> >> >> >> >> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> For more options, visit this group at
>> >> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>
>> >> >> >> >> >
>> >> >> >> >> > --
>> >> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> >> > Google
>> >> >> >> >> > Groups
>> >> >> >> >> > "mongodb-user" group.
>> >> >> >> >> > To post to this group, send email to
>> >> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> > To unsubscribe from this group, send email to
>> >> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> > For more options, visit this group at
>> >> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >> --
>> >> >> >> >> You received this message because you are subscribed to the
>> >> >> >> >> Google
>> >> >> >> >> Groups
>> >> >> >> >> "mongodb-user" group.
>> >> >> >> >> To post to this group, send email to
>> >> >> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> To unsubscribe from this group, send email to
>> >> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> For more options, visit this group at
>> >> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> > Google
>> >> >> >> > Groups
>> >> >> >> > "mongodb-user" group.
>> >> >> >> > To post to this group, send email to
>> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> > To unsubscribe from this group, send email to
>> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> > For more options, visit this group at
>> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >> --
>> >> >> >> You received this message because you are subscribed to the
>> >> >> >> Google
>> >> >> >> Groups
>> >> >> >> "mongodb-user" group.
>> >> >> >> To post to this group, send email to
>> >> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> To unsubscribe from this group, send email to
>> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> For more options, visit this group at
>> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "mongodb-user" group.
>> >> >> > To post to this group, send email to
>> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > To unsubscribe from this group, send email to
>> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "mongodb-user" group.
>> >> >> To post to this group, send email to mongodb-user-/***@public.gmane.org.
>> >> >> To unsubscribe from this group, send email to
>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > To unsubscribe from this group, send email to
>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> To unsubscribe from this group, send email to
>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > To unsubscribe from this group, send email to
>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
>
> On Thursday, May 10, 2012 5:04:33 PM UTC+2, Scott Hernandez wrote:
>>
>> You seem to looking at this as a binary result. Slow reads do not mean
>> downtime. There are many use-cases where many times performance
>> degradation during maintenance windows is acceptable but downtime
>> (stopping reads) is not.
>>
>> We are working with everyone to make sure they can get the best
>> performance and uptime for their systems. If you have more specific
>> areas where you are seeing issue, please create a new thread/post or
>> jira issue with the problem (or vote/watch for existing ones).
>>
>> Nobody thinks the system is perfect as is (how could it be when there
>> are so many different use cases), and there are lots of improvements,
>> fixes and changes going in to address these areas for improvement.
>>
>>
>> On Thu, May 10, 2012 at 7:44 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>> > That use case can easily be supported by just shutting off the reads
>> > from
>> > the application. Basically they are anyways taking downtime -- that
>> > doesn't
>> > take special support, does it?
>> >
>> > In the normal case, don't you think the reads are the highest priority?
>> > Come
>> > on, this is production.. And I'm telling you, your top clients have this
>> > problem. Whats worse for a DB than downtime?
>> >
>> >
>> > On Thu, May 10, 2012 at 7:37 AM, Scott Hernandez
>> > <scotthernandez-***@public.gmane.org>
>> > wrote:
>> >>
>> >> A simple example is a system where they want the balancer to run as
>> >> quickly as possible at low traffic times to balancer data and don't
>> >> care about the impact of (user) reads. This can be a big deal when you
>> >> are already under-provisioned and need to add a new shard to handle
>> >> users more effectively -- it has come up a few times.
>> >>
>> >> On Thu, May 10, 2012 at 7:33 AM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> wrote:
>> >> > Scott,
>> >> >
>> >> >> While it might be paramount for some situations/use-cases to
>> >> >> complete all
>> >> >> reads before a move starts, it might not be for others (which I
>> >> > have seen).
>> >> >
>> >> > Can you please explain the "might not be for others" part? Can you
>> >> > give
>> >> > me
>> >> > an example?
>> >> >
>> >> >
>> >> > On Thu, May 10, 2012 at 7:19 AM, Scott Hernandez
>> >> > <scotthernandez-***@public.gmane.org>
>> >> > wrote:
>> >> >>
>> >> >> There are lots of ways to look at this with some competing goals.
>> >> >> While it might be paramount for some situations/use-cases to
>> >> >> complete
>> >> >> all reads before a move starts, it might not be for others (which I
>> >> >> have seen). This fix will allow reads (and other writes) to get
>> >> >> processes while chunks moves happen, period. The issue of locking,
>> >> >> and
>> >> >> contentions on those locks, is very important and there are lots of
>> >> >> improvements being made which help across the boards.
>> >> >>
>> >> >> Feel free to add comments or create new issues with your ideas of
>> >> >> how
>> >> >> your system would like things to best behave:
>> >> >> http://jira.mongodb.org/browse/server
>> >> >>
>> >> >> On Wed, May 9, 2012 at 7:46 PM, Prasanna .S <myprasanna-***@public.gmane.org>
>> >> >> wrote:
>> >> >> > Scott,
>> >> >> >
>> >> >> > The below are heuristics, and not guarantees. What you want from a
>> >> >> > DB
>> >> >> > stand-point is the latter.
>> >> >> >
>> >> >> > The delay between moves, won't provide a guarantee, unless you
>> >> >> > make
>> >> >> > sure
>> >> >> > that
>> >> >> > (1) the next iteration of moveChunk, does not start unless all
>> >> >> > pending
>> >> >> > reads
>> >> >> > complete.
>> >> >> > (2) Each iteration takes less than 1second.
>> >> >> >
>> >> >> > Thoughts?
>> >> >> >
>> >> >> >
>> >> >> > On Wed, May 9, 2012 at 6:18 PM, Scott Hernandez
>> >> >> > <scotthernandez-***@public.gmane.org>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> The logic there is to do each document write, during the migrate,
>> >> >> >> in
>> >> >> >> a
>> >> >> >> separate lock so other operations can be interleaved. We have
>> >> >> >> seen
>> >> >> >> this change dramatically reduce the constant lock percentage and
>> >> >> >> allow
>> >> >> >> reads, and other writes through during a initial move.
>> >> >> >>
>> >> >> >> There is already a delay between moves which doesn't seem to be
>> >> >> >> your
>> >> >> >> problem.
>> >> >> >>
>> >> >> >> On Wed, May 9, 2012 at 5:37 PM, Prasanna .S
>> >> >> >> <myprasanna-***@public.gmane.org>
>> >> >> >> wrote:
>> >> >> >> > Scott,
>> >> >> >> >
>> >> >> >> > I tried to understand what's going on there. But with a quick
>> >> >> >> > glance,
>> >> >> >> > I
>> >> >> >> > assume, a lock was being held longer than necessary and was
>> >> >> >> > fixed.
>> >> >> >> > It's
>> >> >> >> > unclear how this can help me. The only way to reliably make
>> >> >> >> > sure
>> >> >> >> > the
>> >> >> >> > site
>> >> >> >> > keeps running while a moveChunk is executing, is if, you yield
>> >> >> >> > every
>> >> >> >> > second
>> >> >> >> > within moveChunk and make sure all pending reads complete,
>> >> >> >> > before
>> >> >> >> > the
>> >> >> >> > next
>> >> >> >> > moveChunk period starts.
>> >> >> >> >
>> >> >> >> > Are you guys doing it, or planning to do it? Is there any other
>> >> >> >> > high-level
>> >> >> >> > way to guarantee the reads can be answered within a couple of
>> >> >> >> > seconds?
>> >> >> >> >
>> >> >> >> > Thanks
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > On Wed, May 9, 2012 at 5:17 PM, Scott Hernandez
>> >> >> >> > <scotthernandez-***@public.gmane.org>
>> >> >> >> > wrote:
>> >> >> >> >>
>> >> >> >> >> Okay, looking at the receiving shard mongo55, and donor shard
>> >> >> >> >> mongo43,
>> >> >> >> >> it look like the fix in 2.0.6 (due out in the next two weeks
>> >> >> >> >> of
>> >> >> >> >> so)
>> >> >> >> >> will help you out a lot.
>> >> >> >> >>
>> >> >> >> >> On Wed, May 9, 2012 at 4:37 PM, Prasanna .S
>> >> >> >> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> wrote:
>> >> >> >> >> > 4:37 the balancer has started and disk shot up.
>> >> >> >> >> >
>> >> >> >> >> >
>> >> >> >> >> > On Wed, May 9, 2012 at 4:30 PM, Prasanna .S
>> >> >> >> >> > <myprasanna-***@public.gmane.org>
>> >> >> >> >> > wrote:
>> >> >> >> >> >>
>> >> >> >> >> >> Hey Scott,
>> >> >> >> >> >>
>> >> >> >> >> >> I'm going to do this now. Will start it and stop it quickly
>> >> >> >> >> >> and
>> >> >> >> >> >> give
>> >> >> >> >> >> you
>> >> >> >> >> >> the exact times.
>> >> >> >> >> >>
>> >> >> >> >> >> Thanks,
>> >> >> >> >> >> -Prasanna
>> >> >> >> >> >>
>> >> >> >> >> >>
>> >> >> >> >> >> On Wed, May 9, 2012 at 3:41 PM, Scott Hernandez
>> >> >> >> >> >> <scotthernandez-***@public.gmane.org>
>> >> >> >> >> >> wrote:
>> >> >> >> >> >>>
>> >> >> >> >> >>> Those values are in blocks (512 bytes), not MB.
>> >> >> >> >> >>>
>> >> >> >> >> >>> We have an update in the upcoming 2.0.6 stable release
>> >> >> >> >> >>> which
>> >> >> >> >> >>> will
>> >> >> >> >> >>> help
>> >> >> >> >> >>> with migrations a bit.
>> >> >> >> >> >>> https://jira.mongodb.org/browse/SERVER-5303
>> >> >> >> >> >>> You
>> >> >> >> >> >>> can click on the version like to see other important
>> >> >> >> >> >>> fixes.
>> >> >> >> >> >>>
>> >> >> >> >> >>> Can you do a few migrates tonight so we can see the
>> >> >> >> >> >>> stats/behavior
>> >> >> >> >> >>> in
>> >> >> >> >> >>> MMS? Please let  us know when you do this so we can take a
>> >> >> >> >> >>> look
>> >> >> >> >> >>> at
>> >> >> >> >> >>> the
>> >> >> >> >> >>> data to see if this fix will help.
>> >> >> >> >> >>>
>> >> >> >> >> >>>
>> >> >> >> >> >>> On Tue, May 8, 2012 at 11:49 AM, Prasanna .S
>> >> >> >> >> >>> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> wrote:
>> >> >> >> >> >>> > Hey Dan,
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > Read ahead settings are the defaults (256
>> >> >> >> >> >>> > MB): http://pastebin.com/MHc8s2fq
>> >> >> >> >> >>> > Let me know if you need something else.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > @Elliot,
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > I spoke with some of my friends at foursquare and it
>> >> >> >> >> >>> > turns
>> >> >> >> >> >>> > out
>> >> >> >> >> >>> > they
>> >> >> >> >> >>> > face the
>> >> >> >> >> >>> > same issue as well. I've had my other friends who run
>> >> >> >> >> >>> > companies >
>> >> >> >> >> >>> > $100M
>> >> >> >> >> >>> > market cap (whom I'd rather not name in this list), tell
>> >> >> >> >> >>> > me
>> >> >> >> >> >>> > about
>> >> >> >> >> >>> > their
>> >> >> >> >> >>> > mongo problems in the past few days. This is pretty
>> >> >> >> >> >>> > serious
>> >> >> >> >> >>> > stuff
>> >> >> >> >> >>> > and
>> >> >> >> >> >>> > will
>> >> >> >> >> >>> > affect if someone wants to choose mongo as their primary
>> >> >> >> >> >>> > store
>> >> >> >> >> >>> > or
>> >> >> >> >> >>> > not.
>> >> >> >> >> >>> > To
>> >> >> >> >> >>> > not have sharding is one thing, but to promise it and
>> >> >> >> >> >>> > being
>> >> >> >> >> >>> > stuck
>> >> >> >> >> >>> > in
>> >> >> >> >> >>> > production is another.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > I really thank you guys and appreciate the
>> >> >> >> >> >>> > responsiveness
>> >> >> >> >> >>> > so
>> >> >> >> >> >>> > far.
>> >> >> >> >> >>> > Happy
>> >> >> >> >> >>> > to
>> >> >> >> >> >>> > work together and hope we can resolve this.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > Thanks so much,
>> >> >> >> >> >>> > -Prasanna
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > On Tue, May 8, 2012 at 7:02 AM, Dan Pasette
>> >> >> >> >> >>> > <dan-***@public.gmane.org>
>> >> >> >> >> >>> > wrote:
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> Looks like the docs in that collection
>> >> >> >> >> >>> >> (lal_production.facebook_names)
>> >> >> >> >> >>> >> are
>> >> >> >> >> >>> >> small and likely distributed throughout the disk.
>> >> >> >> >> >>> >> Can you check your disk readahead setting?
>> >> >> >> >> >>> >> Please run and post the results of blockev --report.
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> On Tue, May 8, 2012 at 7:56 AM, Prasanna .S
>> >> >> >> >> >>> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >> wrote:
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> Yes, indeed.
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> Username: arnold-***@public.gmane.org
>> >> >> >> >> >>> >>> Machine: mongo55
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> On Tue, May 8, 2012 at 4:49 AM, Eliot Horowitz
>> >> >> >> >> >>> >>> <eliot-***@public.gmane.org>
>> >> >> >> >> >>> >>> wrote:
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> Are these machines in MMS?
>> >> >> >> >> >>> >>>> Hard to see much in the pasted stats.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> On Tue, May 8, 2012 at 5:03 AM, Prasanna .S
>> >> >> >> >> >>> >>>> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> wrote:
>> >> >> >> >> >>> >>>> > Raid 10 on 8 disks.
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > MongoDB shell version: 2.0.4
>> >> >> >> >> >>> >>>> > Tue May  8 09:02:29 [initandlisten] db version
>> >> >> >> >> >>> >>>> > v2.0.4,
>> >> >> >> >> >>> >>>> > pdfile
>> >> >> >> >> >>> >>>> > version
>> >> >> >> >> >>> >>>> > 4.5
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > On Tue, May 8, 2012 at 1:49 AM, Eliot Horowitz
>> >> >> >> >> >>> >>>> > <eliot-***@public.gmane.org>
>> >> >> >> >> >>> >>>> > wrote:
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> What version of mongo and what kind of ebs raid?
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> On Tue, May 8, 2012 at 4:46 AM, Prasanna
>> >> >> >> >> >>> >>>> >> <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> wrote:
>> >> >> >> >> >>> >>>> >> > And yes, disabling the balancer, brings the
>> >> >> >> >> >>> >>>> >> > write
>> >> >> >> >> >>> >>>> >> > rate
>> >> >> >> >> >>> >>>> >> > back
>> >> >> >> >> >>> >>>> >> > to
>> >> >> >> >> >>> >>>> >> > zero.
>> >> >> >> >> >>> >>>> >> > Although after 4 minutes!
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > On Monday, May 7, 2012 11:24:22 PM UTC-7, Nat
>> >> >> >> >> >>> >>>> >> > wrote:
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> I don't think it goes into deadlock. Can you
>> >> >> >> >> >>> >>>> >> >> try
>> >> >> >> >> >>> >>>> >> >> disabling
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> balancer
>> >> >> >> >> >>> >>>> >> >> and see whether the disk write rate going down
>> >> >> >> >> >>> >>>> >> >> or
>> >> >> >> >> >>> >>>> >> >> not.
>> >> >> >> >> >>> >>>> >> >> ________________________________
>> >> >> >> >> >>> >>>> >> >> From: "Prasanna .S" <myprasanna-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> Sender: mongodb-user-/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> Date: Mon, 7 May 2012 23:22:11 -0700
>> >> >> >> >> >>> >>>> >> >> To: <mongodb-user-/***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> ReplyTo: mongodb-user-/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> Cc: Surendran Mahendran<surenspost-***@public.gmane.org>;
>> >> >> >> >> >>> >>>> >> >> Hrushikesh
>> >> >> >> >> >>> >>>> >> >> Tilak<hrushikesh.tilak-***@public.gmane.org>
>> >> >> >> >> >>> >>>> >> >> Subject: [mongodb-user] Mongo sharding deadlock
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Hey guys,
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> We sharded mongo as per the docs. The issue is
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> balancer
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> not
>> >> >> >> >> >>> >>>> >> >> being
>> >> >> >> >> >>> >>>> >> >> very clever and not throttling at all. Now we
>> >> >> >> >> >>> >>>> >> >> are
>> >> >> >> >> >>> >>>> >> >> stuck
>> >> >> >> >> >>> >>>> >> >> with a
>> >> >> >> >> >>> >>>> >> >> half-sharded
>> >> >> >> >> >>> >>>> >> >> setup and we can't turn on the balancer because
>> >> >> >> >> >>> >>>> >> >> it'd
>> >> >> >> >> >>> >>>> >> >> bring
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> site
>> >> >> >> >> >>> >>>> >> >> down.
>> >> >> >> >> >>> >>>> >> >> All read queries timeout when the disk write
>> >> >> >> >> >>> >>>> >> >> rate
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> 100%.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Is there a way out of this mess at all? From a
>> >> >> >> >> >>> >>>> >> >> high-level,
>> >> >> >> >> >>> >>>> >> >> mongo
>> >> >> >> >> >>> >>>> >> >> could
>> >> >> >> >> >>> >>>> >> >> have handled this by any of the following
>> >> >> >> >> >>> >>>> >> >> means:
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> 1. Replicate from the slaves. Mysql does this.
>> >> >> >> >> >>> >>>> >> >> 2. Throttle and replicate only as long as the
>> >> >> >> >> >>> >>>> >> >> disk
>> >> >> >> >> >>> >>>> >> >> util
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> <
>> >> >> >> >> >>> >>>> >> >> 50%
>> >> >> >> >> >>> >>>> >> >> or so.
>> >> >> >> >> >>> >>>> >> >> 3. Have the moveChunk interruptible. If reads
>> >> >> >> >> >>> >>>> >> >> and
>> >> >> >> >> >>> >>>> >> >> writes
>> >> >> >> >> >>> >>>> >> >> come
>> >> >> >> >> >>> >>>> >> >> in-between,
>> >> >> >> >> >>> >>>> >> >> consider them as higher priority, so that a
>> >> >> >> >> >>> >>>> >> >> read/write
>> >> >> >> >> >>> >>>> >> >> is
>> >> >> >> >> >>> >>>> >> >> never
>> >> >> >> >> >>> >>>> >> >> waiting
>> >> >> >> >> >>> >>>> >> >> for
>> >> >> >> >> >>> >>>> >> >> more than a second.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Have you guys heard of this problem before?
>> >> >> >> >> >>> >>>> >> >> Because
>> >> >> >> >> >>> >>>> >> >> this
>> >> >> >> >> >>> >>>> >> >> really
>> >> >> >> >> >>> >>>> >> >> make
>> >> >> >> >> >>> >>>> >> >> sharding impossible. I'm not sure what others
>> >> >> >> >> >>> >>>> >> >> are
>> >> >> >> >> >>> >>>> >> >> doing
>> >> >> >> >> >>> >>>> >> >> in
>> >> >> >> >> >>> >>>> >> >> production
>> >> >> >> >> >>> >>>> >> >> to
>> >> >> >> >> >>> >>>> >> >> work around this.
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> Thanks,
>> >> >> >> >> >>> >>>> >> >> -Prasanna
>> >> >> >> >> >>> >>>> >> >> Co-Founder & CTO, lal.com
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> --
>> >> >> >> >> >>> >>>> >> >> You received this message because you are
>> >> >> >> >> >>> >>>> >> >> subscribed
>> >> >> >> >> >>> >>>> >> >> to
>> >> >> >> >> >>> >>>> >> >> the
>> >> >> >> >> >>> >>>> >> >> Google
>> >> >> >> >> >>> >>>> >> >> Groups
>> >> >> >> >> >>> >>>> >> >> "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> >> To post to this group, send email to
>> >> >> >> >> >>> >>>> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >>
>> >> >> >> >> >>> >>>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > --
>> >> >> >> >> >>> >>>> >> > You received this message because you are
>> >> >> >> >> >>> >>>> >> > subscribed
>> >> >> >> >> >>> >>>> >> > to
>> >> >> >> >> >>> >>>> >> > the
>> >> >> >> >> >>> >>>> >> > Google
>> >> >> >> >> >>> >>>> >> > Groups
>> >> >> >> >> >>> >>>> >> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> > To view this discussion on the web visit
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > https://groups.google.com/d/msg/mongodb-user/-/ePw_QsFo6TYJ.
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > To post to this group, send email to
>> >> >> >> >> >>> >>>> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> > For more options, visit this group at
>> >> >> >> >> >>> >>>> >> >
>> >> >> >> >> >>> >>>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >> --
>> >> >> >> >> >>> >>>> >> You received this message because you are
>> >> >> >> >> >>> >>>> >> subscribed
>> >> >> >> >> >>> >>>> >> to
>> >> >> >> >> >>> >>>> >> the
>> >> >> >> >> >>> >>>> >> Google
>> >> >> >> >> >>> >>>> >> Groups
>> >> >> >> >> >>> >>>> >> "mongodb-user" group.
>> >> >> >> >> >>> >>>> >> To post to this group, send email to
>> >> >> >> >> >>> >>>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> >> For more options, visit this group at
>> >> >> >> >> >>> >>>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>> >>
>> >> >> >> >> >>> >>>> >
>> >> >> >> >> >>> >>>> > --
>> >> >> >> >> >>> >>>> > You received this message because you are
>> >> >> >> >> >>> >>>> > subscribed
>> >> >> >> >> >>> >>>> > to
>> >> >> >> >> >>> >>>> > the
>> >> >> >> >> >>> >>>> > Google
>> >> >> >> >> >>> >>>> > Groups
>> >> >> >> >> >>> >>>> > "mongodb-user" group.
>> >> >> >> >> >>> >>>> > To post to this group, send email to
>> >> >> >> >> >>> >>>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> > For more options, visit this group at
>> >> >> >> >> >>> >>>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>> --
>> >> >> >> >> >>> >>>> You received this message because you are subscribed
>> >> >> >> >> >>> >>>> to
>> >> >> >> >> >>> >>>> the
>> >> >> >> >> >>> >>>> Google
>> >> >> >> >> >>> >>>> Groups "mongodb-user" group.
>> >> >> >> >> >>> >>>> To post to this group, send email to
>> >> >> >> >> >>> >>>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>>> For more options, visit this group at
>> >> >> >> >> >>> >>>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>>>
>> >> >> >> >> >>> >>>
>> >> >> >> >> >>> >>> --
>> >> >> >> >> >>> >>> You received this message because you are subscribed
>> >> >> >> >> >>> >>> to
>> >> >> >> >> >>> >>> the
>> >> >> >> >> >>> >>> Google
>> >> >> >> >> >>> >>> Groups
>> >> >> >> >> >>> >>> "mongodb-user" group.
>> >> >> >> >> >>> >>> To post to this group, send email to
>> >> >> >> >> >>> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >>> For more options, visit this group at
>> >> >> >> >> >>> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >>
>> >> >> >> >> >>> >> --
>> >> >> >> >> >>> >> You received this message because you are subscribed to
>> >> >> >> >> >>> >> the
>> >> >> >> >> >>> >> Google
>> >> >> >> >> >>> >> Groups
>> >> >> >> >> >>> >> "mongodb-user" group.
>> >> >> >> >> >>> >> To post to this group, send email to
>> >> >> >> >> >>> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >> To unsubscribe from this group, send email to
>> >> >> >> >> >>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> >> For more options, visit this group at
>> >> >> >> >> >>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>> >
>> >> >> >> >> >>> >
>> >> >> >> >> >>> > --
>> >> >> >> >> >>> > You received this message because you are subscribed to
>> >> >> >> >> >>> > the
>> >> >> >> >> >>> > Google
>> >> >> >> >> >>> > Groups
>> >> >> >> >> >>> > "mongodb-user" group.
>> >> >> >> >> >>> > To post to this group, send email to
>> >> >> >> >> >>> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> > To unsubscribe from this group, send email to
>> >> >> >> >> >>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> > For more options, visit this group at
>> >> >> >> >> >>> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>> --
>> >> >> >> >> >>> You received this message because you are subscribed to
>> >> >> >> >> >>> the
>> >> >> >> >> >>> Google
>> >> >> >> >> >>> Groups
>> >> >> >> >> >>> "mongodb-user" group.
>> >> >> >> >> >>> To post to this group, send email to
>> >> >> >> >> >>> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> To unsubscribe from this group, send email to
>> >> >> >> >> >>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> >>> For more options, visit this group at
>> >> >> >> >> >>> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >> >>>
>> >> >> >> >> >>
>> >> >> >> >> >
>> >> >> >> >> > --
>> >> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> >> > Google
>> >> >> >> >> > Groups
>> >> >> >> >> > "mongodb-user" group.
>> >> >> >> >> > To post to this group, send email to
>> >> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> > To unsubscribe from this group, send email to
>> >> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> > For more options, visit this group at
>> >> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >> --
>> >> >> >> >> You received this message because you are subscribed to the
>> >> >> >> >> Google
>> >> >> >> >> Groups
>> >> >> >> >> "mongodb-user" group.
>> >> >> >> >> To post to this group, send email to
>> >> >> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> To unsubscribe from this group, send email to
>> >> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> >> For more options, visit this group at
>> >> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >> >>
>> >> >> >> >
>> >> >> >> > --
>> >> >> >> > You received this message because you are subscribed to the
>> >> >> >> > Google
>> >> >> >> > Groups
>> >> >> >> > "mongodb-user" group.
>> >> >> >> > To post to this group, send email to
>> >> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> > To unsubscribe from this group, send email to
>> >> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> > For more options, visit this group at
>> >> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >> --
>> >> >> >> You received this message because you are subscribed to the
>> >> >> >> Google
>> >> >> >> Groups
>> >> >> >> "mongodb-user" group.
>> >> >> >> To post to this group, send email to
>> >> >> >> mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> To unsubscribe from this group, send email to
>> >> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> >> For more options, visit this group at
>> >> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >> >>
>> >> >> >
>> >> >> > --
>> >> >> > You received this message because you are subscribed to the Google
>> >> >> > Groups
>> >> >> > "mongodb-user" group.
>> >> >> > To post to this group, send email to
>> >> >> > mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > To unsubscribe from this group, send email to
>> >> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> > For more options, visit this group at
>> >> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >> --
>> >> >> You received this message because you are subscribed to the Google
>> >> >> Groups
>> >> >> "mongodb-user" group.
>> >> >> To post to this group, send email to mongodb-user-/***@public.gmane.org.
>> >> >> To unsubscribe from this group, send email to
>> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> >> For more options, visit this group at
>> >> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >> >>
>> >> >
>> >> > --
>> >> > You received this message because you are subscribed to the Google
>> >> > Groups
>> >> > "mongodb-user" group.
>> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > To unsubscribe from this group, send email to
>> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> > For more options, visit this group at
>> >> > http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> To unsubscribe from this group, send email to
>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > To unsubscribe from this group, send email to
>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/mongodb-user/-/YRlv9SXrV6QJ.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-10 22:22:37 UTC
Permalink
Scott,

> You seem to looking at this as a binary result. Slow reads do not mean
downtime.

If this were true I'd be fine. Performance degradation is fine, downtime is
not. Right now the result is binary.

So, how do we guarantee, performance degradation will not cause downtime?
You make sure the read query queue, doesn't get infinitely long.
Alternatively, you can ensure, because of moveChunk, reads wait at most 1
second longer.

My worry is, you are proposing a heuristic to reduce lock contention, but
no guarantees. I worked on a database for Microsoft Bing which runs on 300K
machines, and the diff between the two scare me. The diff is, on a long
enough timeframe, heuristics absolutely break, and guarantees do not.

I definitely do want a convincing answer to this one. If you can somehow
tell me, there won't be downtime in the future, because of this issue,
that'd be great. You have to understand though, there is right now, and
it's not pretty.

Thanks,
-Prasanna

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-10 22:58:21 UTC
Permalink
> Nobody thinks the system is perfect as is (how could it be when there
> are so many different use cases), and there are lots of improvements,
> fixes and changes going in to address these areas for improvement.

Of-course, I agree with you on this. If my queries just run 10x slower
during maintenance, I won't complain here.
Saying the same to down-time is not acceptable. At-least if downtimes
should be like 1 in a million or billion chance of happening, I can bear
with it. It happens *all* the time to *everyone* using sharding.

Let's figure out where we disagree on:

1) FourSquare and other of your top clients, run with moveChunk off during
production, which otherwise causes downtime? Yes/No
2) This is the highest priority of MongoDB as a database, to solve right
now. Yes/No

If both the answers are Yes, we both agree and I'd just make a Jira thread
and monitor when it ships. You guys have been incredibly responsive and
that's the main reason why I haven't moved away or blogged in TechCrunch
about this otherwise fatal problem.

I'm open to hearing alternate explanations: Is there a way you
can guarantee, moveChunk won't take more than 10% of disk-time and cpu time
etc. I suspect any such approaches, will end up being way more complex,
than executing things based on priorities.

Thanks,
-Prasanna


On Thu, May 10, 2012 at 3:22 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:

> Scott,
>
>
> > You seem to looking at this as a binary result. Slow reads do not mean
> downtime.
>
> If this were true I'd be fine. Performance degradation is fine, downtime
> is not. Right now the result is binary.
>
> So, how do we guarantee, performance degradation will not cause downtime?
> You make sure the read query queue, doesn't get infinitely long.
> Alternatively, you can ensure, because of moveChunk, reads wait at most 1
> second longer.
>
> My worry is, you are proposing a heuristic to reduce lock contention, but
> no guarantees. I worked on a database for Microsoft Bing which runs on 300K
> machines, and the diff between the two scare me. The diff is, on a long
> enough timeframe, heuristics absolutely break, and guarantees do not.
>
> I definitely do want a convincing answer to this one. If you can somehow
> tell me, there won't be downtime in the future, because of this issue,
> that'd be great. You have to understand though, there is right now, and
> it's not pretty.
>
> Thanks,
> -Prasanna
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Eliot Horowitz
2012-05-11 05:28:30 UTC
Permalink
> 1) FourSquare and other of your top clients, run with moveChunk off during
> production, which otherwise causes downtime? Yes/No

No, most people run with balancing on 24/7.

> 2) This is the highest priority of MongoDB as a database, to solve right
> now. Yes/No

The general issue here is overall concurrency, which is the highest priority.

As Scott said, 2.0.6 will have some significant improvements for this
case as well.
Prasanna .S
2012-05-11 13:47:38 UTC
Permalink
Eliot, Foursquare does not. MongoHQ folks recommend people not to. Bump
stopped using MongoDB. How many more examples do you want?

On Thu, May 10, 2012 at 10:28 PM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:

> > 1) FourSquare and other of your top clients, run with moveChunk off
> during
> > production, which otherwise causes downtime? Yes/No
>
> No, most people run with balancing on 24/7.
>
> > 2) This is the highest priority of MongoDB as a database, to solve right
> > now. Yes/No
>
> The general issue here is overall concurrency, which is the highest
> priority.
>
> As Scott said, 2.0.6 will have some significant improvements for this
> case as well.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Eliot Horowitz
2012-05-11 14:23:59 UTC
Permalink
I can't discuss foursquare further.
Many people (most by my count) have balancing on 24x7

There are a lot of factors that change performance, notable doc size
and disk speed.

There are definitely scenarios where balancing can cause problems
because of concurrency, and those are what we are working on.
2.0.5 had one big one and 2.0.6 will have a very large one.
2.2 will add a lot more concurrency overall, so will help tremendously
with this issue as well.

On Fri, May 11, 2012 at 9:47 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
> Eliot, Foursquare does not. MongoHQ folks recommend people not to. Bump
> stopped using MongoDB. How many more examples do you want?
>
> On Thu, May 10, 2012 at 10:28 PM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:
>>
>> > 1) FourSquare and other of your top clients, run with moveChunk off
>> > during
>> > production, which otherwise causes downtime? Yes/No
>>
>> No, most people run with balancing on 24/7.
>>
>> > 2) This is the highest priority of MongoDB as a database, to solve right
>> > now. Yes/No
>>
>> The general issue here is overall concurrency, which is the highest
>> priority.
>>
>> As Scott said, 2.0.6 will have some significant improvements for this
>> case as well.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-11 14:31:25 UTC
Permalink
> I can't discuss foursquare further.

You added them as your top customer <http://www.10gen.com/>.

> Many people (most by my count) have balancing on 24x7

Can you tell me what's different about them? The people I asked seem to be
having problems running it 24x7. All of us certainly have data fitting in
memory. Can you give me one contact?

> There are definitely scenarios where balancing can cause problems
> because of concurrency, and those are what we are working on.

On one hand I'm happy that you are working on it. On the other, I'm not
happy about the guarantees being very weak. "Improve" is a bad word to use,
when associated with DB downtime.

On Fri, May 11, 2012 at 7:23 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:

> I can't discuss foursquare further.
> Many people (most by my count) have balancing on 24x7
>
> There are a lot of factors that change performance, notable doc size
> and disk speed.
>
> There are definitely scenarios where balancing can cause problems
> because of concurrency, and those are what we are working on.
> 2.0.5 had one big one and 2.0.6 will have a very large one.
> 2.2 will add a lot more concurrency overall, so will help tremendously
> with this issue as well.
>
> On Fri, May 11, 2012 at 9:47 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
> > Eliot, Foursquare does not. MongoHQ folks recommend people not to. Bump
> > stopped using MongoDB. How many more examples do you want?
> >
> > On Thu, May 10, 2012 at 10:28 PM, Eliot Horowitz <eliot-***@public.gmane.org>
> wrote:
> >>
> >> > 1) FourSquare and other of your top clients, run with moveChunk off
> >> > during
> >> > production, which otherwise causes downtime? Yes/No
> >>
> >> No, most people run with balancing on 24/7.
> >>
> >> > 2) This is the highest priority of MongoDB as a database, to solve
> right
> >> > now. Yes/No
> >>
> >> The general issue here is overall concurrency, which is the highest
> >> priority.
> >>
> >> As Scott said, 2.0.6 will have some significant improvements for this
> >> case as well.
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "mongodb-user" group.
> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> To unsubscribe from this group, send email to
> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> For more options, visit this group at
> >> http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > To unsubscribe from this group, send email to
> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > For more options, visit this group at
> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Eliot Horowitz
2012-05-11 14:36:43 UTC
Permalink
>> I can't discuss foursquare further.
> You added them as your top customer.
And they are a client, so I cannot discuss it.

>> Many people (most by my count) have balancing on 24x7
>
> Can you tell me what's different about them? The people I asked seem to be
> having problems running it 24x7. All of us certainly have data fitting in
> memory. Can you give me one contact?

Most people running do not have all data fitting in memory.
If no one step on this thread if you contact me offline I can probably
set something up.

>> There are definitely scenarios where balancing can cause problems
>> because of concurrency, and those are what we are working on.
>
> On one hand I'm happy that you are working on it. On the other, I'm not
> happy about the guarantees being very weak. "Improve" is a bad word to use,
> when associated with DB downtime.

Guarantees are hard when we don't control hardware, etc...
There are a lot of factors, so a 100% guarantee would take a lot to get to.

>
> On Fri, May 11, 2012 at 7:23 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:
>>
>> I can't discuss foursquare further.
>> Many people (most by my count) have balancing on 24x7
>>
>> There are a lot of factors that change performance, notable doc size
>> and disk speed.
>>
>> There are definitely scenarios where balancing can cause problems
>> because of concurrency, and those are what we are working on.
>> 2.0.5 had one big one and 2.0.6 will have a very large one.
>> 2.2 will add a lot more concurrency overall, so will help tremendously
>> with this issue as well.
>>
>> On Fri, May 11, 2012 at 9:47 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>> > Eliot, Foursquare does not. MongoHQ folks recommend people not to. Bump
>> > stopped using MongoDB. How many more examples do you want?
>> >
>> > On Thu, May 10, 2012 at 10:28 PM, Eliot Horowitz <eliot-***@public.gmane.org>
>> > wrote:
>> >>
>> >> > 1) FourSquare and other of your top clients, run with moveChunk off
>> >> > during
>> >> > production, which otherwise causes downtime? Yes/No
>> >>
>> >> No, most people run with balancing on 24/7.
>> >>
>> >> > 2) This is the highest priority of MongoDB as a database, to solve
>> >> > right
>> >> > now. Yes/No
>> >>
>> >> The general issue here is overall concurrency, which is the highest
>> >> priority.
>> >>
>> >> As Scott said, 2.0.6 will have some significant improvements for this
>> >> case as well.
>> >>
>> >> --
>> >> You received this message because you are subscribed to the Google
>> >> Groups
>> >> "mongodb-user" group.
>> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> To unsubscribe from this group, send email to
>> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> >> For more options, visit this group at
>> >> http://groups.google.com/group/mongodb-user?hl=en.
>> >>
>> >
>> > --
>> > You received this message because you are subscribed to the Google
>> > Groups
>> > "mongodb-user" group.
>> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > To unsubscribe from this group, send email to
>> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> > For more options, visit this group at
>> > http://groups.google.com/group/mongodb-user?hl=en.
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-11 14:52:32 UTC
Permalink
On Fri, May 11, 2012 at 7:36 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:

> >> I can't discuss foursquare further.
> > You added them as your top customer.
> And they are a client, so I cannot discuss it.
>

If you want to take this offline, I can connect with my friends at
foursquare. As far as I know, the lead there just told me they don't run it
24x7.


>
> >> Many people (most by my count) have balancing on 24x7
> >
> > Can you tell me what's different about them? The people I asked seem to
> be
> > having problems running it 24x7. All of us certainly have data fitting in
> > memory. Can you give me one contact?
>
> Most people running do not have all data fitting in memory.
> If no one step on this thread if you contact me offline I can probably
> set something up.
>

Thanks - I just want to understand what's different. We both just disagree
on the impact of this problem.


>
> >> There are definitely scenarios where balancing can cause problems
> >> because of concurrency, and those are what we are working on.
> >
> > On one hand I'm happy that you are working on it. On the other, I'm not
> > happy about the guarantees being very weak. "Improve" is a bad word to
> use,
> > when associated with DB downtime.
>
> Guarantees are hard when we don't control hardware, etc...
> There are a lot of factors, so a 100% guarantee would take a lot to get to.
>

I think we'd agree on 3 high level ways to provide a guarantee in this
specific instance.
1. Replicate from the slaves. Mysql does this. -- Probably too complex for
you guys at this time.
2. Throttle and replicate only as long as the disk util is < 50% or so AND
cpu util is < 50%. -- Probably complex as well.
3. Have the moveChunk interruptible. If reads and writes come in-between,
consider them as higher priority, so that a read/write is never waiting for
more than a second. -- Should not be extremely hard to do.

Do you agree the above 3 ways provide a guarantee, regardless of hardware?

Sorry to be an assole here Eliot. You can google search for me, and I've
never ever debated an open source software this passionately. I really got
burnt this time in production. People who use mongo regularly said it's a
very common problem. I think, if others knew what they were getting into,
half the CTOs won't choose mongo.


>
> >
> > On Fri, May 11, 2012 at 7:23 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:
> >>
> >> I can't discuss foursquare further.
> >> Many people (most by my count) have balancing on 24x7
> >>
> >> There are a lot of factors that change performance, notable doc size
> >> and disk speed.
> >>
> >> There are definitely scenarios where balancing can cause problems
> >> because of concurrency, and those are what we are working on.
> >> 2.0.5 had one big one and 2.0.6 will have a very large one.
> >> 2.2 will add a lot more concurrency overall, so will help tremendously
> >> with this issue as well.
> >>
> >> On Fri, May 11, 2012 at 9:47 AM, Prasanna .S <myprasanna-***@public.gmane.org>
> wrote:
> >> > Eliot, Foursquare does not. MongoHQ folks recommend people not to.
> Bump
> >> > stopped using MongoDB. How many more examples do you want?
> >> >
> >> > On Thu, May 10, 2012 at 10:28 PM, Eliot Horowitz <eliot-***@public.gmane.org>
> >> > wrote:
> >> >>
> >> >> > 1) FourSquare and other of your top clients, run with moveChunk off
> >> >> > during
> >> >> > production, which otherwise causes downtime? Yes/No
> >> >>
> >> >> No, most people run with balancing on 24/7.
> >> >>
> >> >> > 2) This is the highest priority of MongoDB as a database, to solve
> >> >> > right
> >> >> > now. Yes/No
> >> >>
> >> >> The general issue here is overall concurrency, which is the highest
> >> >> priority.
> >> >>
> >> >> As Scott said, 2.0.6 will have some significant improvements for this
> >> >> case as well.
> >> >>
> >> >> --
> >> >> You received this message because you are subscribed to the Google
> >> >> Groups
> >> >> "mongodb-user" group.
> >> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> To unsubscribe from this group, send email to
> >> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> >> For more options, visit this group at
> >> >> http://groups.google.com/group/mongodb-user?hl=en.
> >> >>
> >> >
> >> > --
> >> > You received this message because you are subscribed to the Google
> >> > Groups
> >> > "mongodb-user" group.
> >> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> > To unsubscribe from this group, send email to
> >> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> > For more options, visit this group at
> >> > http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "mongodb-user" group.
> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> To unsubscribe from this group, send email to
> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> For more options, visit this group at
> >> http://groups.google.com/group/mongodb-user?hl=en.
> >>
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > To unsubscribe from this group, send email to
> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > For more options, visit this group at
> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Eliot Horowitz
2012-05-11 14:58:11 UTC
Permalink
On Fri, May 11, 2012 at 10:52 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
> I think we'd agree on 3 high level ways to provide a guarantee in this
> specific instance.
> 1. Replicate from the slaves. Mysql does this. -- Probably too complex for
> you guys at this time.

Do you mean chained replication, or something else?
We already do chained replication in certain cases.

> 2. Throttle and replicate only as long as the disk util is < 50% or so AND
> cpu util is < 50%. -- Probably complex as well.

This is tricky.
For example, some people run with disk util > 50% all the time.

> 3. Have the moveChunk interruptible. If reads and writes come in-between,
> consider them as higher priority, so that a read/write is never waiting for
> more than a second. -- Should not be extremely hard to do.

This is the direction the changes in 2.0.5 and 2.0.6 have gone.

> Do you agree the above 3 ways provide a guarantee, regardless of hardware?

No - I do not think those three things provide a guarantee.
When you add load to a system, there are a lot of factors at play, so
I would definitely not say even with those 3 things, that it could not
cause an issue.
Especially if you're operating in a system where the hardware isn't
predictable itself, there are many possible issues.

What I can guarantee is that we will keep working on it till its not an issue.

> Sorry to be an assole here Eliot. You can google search for me, and I've
> never ever debated an open source software this passionately. I really got
> burnt this time in production. People who use mongo regularly said it's a
> very common problem. I think, if others knew what they were getting into,
> half the CTOs won't choose mongo.

I'm very sorry you got burnt, I don't know the specifics of your case,
or why it happened, but certainly happy to try and understand the
specifics.
If you drill down into specifics, the causes are often very different.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-11 15:11:23 UTC
Permalink
> No - I do not think those three things provide a guarantee.

I was saying any one of the 3 provide a guarantee.
Let's take the last one alone -

a) moveChunk executes it's iteration, iff all reads are completed.
b) we guarantee that a single moveChunk iteration takes < 1s.

(b) Can be a bit tricky because you have to make sure, the iteration which
includes the time the writes are waiting in disk buffers, finish within
that 1s. But definitely not very hard.

I'm hypothesizing that this provides a complete guarantee, regardless of
hardware. Can you explain a bit, why my reasoning is flawed?

> If you drill down into specifics, the causes are often very different.

First of all, thank you so much for offering to help. I have never seen
this level of commitment and professionalism elsewhere and I really
appreciate it.

>From my basic drill-down, it was the last part of moveChunk, which was
deleting the date from the master-shard that was causing tonnes of disk
writes. It makes me think, even if you make sure the moveChunk only marks a
chunk as deleted in Primary, but doesn't physically touch the disk, just
unloads the pages from RAM, that might work too.

Very happy to co-operate with you if you want to investigate further on the
machines. Glad to provide shell level access.

On Fri, May 11, 2012 at 7:58 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:

> On Fri, May 11, 2012 at 10:52 AM, Prasanna .S <myprasanna-***@public.gmane.org>
> wrote:
> > I think we'd agree on 3 high level ways to provide a guarantee in this
> > specific instance.
> > 1. Replicate from the slaves. Mysql does this. -- Probably too complex
> for
> > you guys at this time.
>
> Do you mean chained replication, or something else?
> We already do chained replication in certain cases.
>
> > 2. Throttle and replicate only as long as the disk util is < 50% or so
> AND
> > cpu util is < 50%. -- Probably complex as well.
>
> This is tricky.
> For example, some people run with disk util > 50% all the time.
>
> > 3. Have the moveChunk interruptible. If reads and writes come in-between,
> > consider them as higher priority, so that a read/write is never waiting
> for
> > more than a second. -- Should not be extremely hard to do.
>
> This is the direction the changes in 2.0.5 and 2.0.6 have gone.
>
> > Do you agree the above 3 ways provide a guarantee, regardless of
> hardware?
>
> No - I do not think those three things provide a guarantee.
> When you add load to a system, there are a lot of factors at play, so
> I would definitely not say even with those 3 things, that it could not
> cause an issue.
> Especially if you're operating in a system where the hardware isn't
> predictable itself, there are many possible issues.
>
> What I can guarantee is that we will keep working on it till its not an
> issue.
>
> > Sorry to be an assole here Eliot. You can google search for me, and I've
> > never ever debated an open source software this passionately. I really
> got
> > burnt this time in production. People who use mongo regularly said it's a
> > very common problem. I think, if others knew what they were getting into,
> > half the CTOs won't choose mongo.
>
> I'm very sorry you got burnt, I don't know the specifics of your case,
> or why it happened, but certainly happy to try and understand the
> specifics.
> If you drill down into specifics, the causes are often very different.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Eliot Horowitz
2012-05-11 15:24:33 UTC
Permalink
> a) moveChunk executes it's iteration, iff all reads are completed.
> b) we guarantee that a single moveChunk iteration takes < 1s.

1 second is too long for some apps.
there are mongo users who have 20ms SLAs on db reads for example.

> (b) Can be a bit tricky because you have to make sure, the iteration which
> includes the time the writes are waiting in disk buffers, finish within that
> 1s. But definitely not very hard.
>
> I'm hypothesizing that this provides a complete guarantee, regardless of
> hardware. Can you explain a bit, why my reasoning is flawed?

For example, if you were on a single spinning disk, and you need to
load data that will cause a seek, and you do this a few times per
second, it could degrade read performance enough to have a significant
impact.
Now, if you're at that point you probably wait too long to shard, but
its a gray area, so wouldn't want to guarantee anything/

> From my basic drill-down, it was the last part of moveChunk, which was
> deleting the date from the master-shard that was causing tonnes of disk
> writes. It makes me think, even if you make sure the moveChunk only marks a
> chunk as deleted in Primary, but doesn't physically touch the disk, just
> unloads the pages from RAM, that might work too.

Interesting. We do have to actually delete the data so the space can
be re-used, but might be able to do it more efficiently.
The code should be yielding before actually doing anything, so kind of
surprising it caused too much load on the primary.
How active is your disk at steady state?

There is one quick change that would be interesting to try.
When we first added sharding a couple years ago, we put it a pedantic
safety measure of writing out all the data we delete after a chunk is
moved just to be sure.
I wonder how much load that is adding.
Would be trivial to remove.

Do you have a lab where you can reproduce this easily?

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-11 16:47:11 UTC
Permalink
inline...

On Fri, May 11, 2012 at 8:24 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:

> > a) moveChunk executes it's iteration, iff all reads are completed.
> > b) we guarantee that a single moveChunk iteration takes < 1s.
>
> 1 second is too long for some apps.
> there are mongo users who have 20ms SLAs on db reads for example.
>

Sure, 1s is an example time unit. If you go lower, I can't be less happier
;)


>
> > (b) Can be a bit tricky because you have to make sure, the iteration
> which
> > includes the time the writes are waiting in disk buffers, finish within
> that
> > 1s. But definitely not very hard.
> >
> > I'm hypothesizing that this provides a complete guarantee, regardless of
> > hardware. Can you explain a bit, why my reasoning is flawed?
>
> For example, if you were on a single spinning disk, and you need to
> load data that will cause a seek, and you do this a few times per
> second, it could degrade read performance enough to have a significant
> impact.

Now, if you're at that point you probably wait too long to shard, but
> its a gray area, so wouldn't want to guarantee anything/
>

Time_1: Time taken for read X to complete without replication.
Time_2: The same with replication.

Guarantee is this:

Time_2 <= Time_1 + 1 second;

i.e., The disk read queue filling up problem has nothing to do with
moveChunk. moveChunk only added one second to the otherwise downtime
causing environment.

Makes sense right?


>
> > From my basic drill-down, it was the last part of moveChunk, which was
> > deleting the date from the master-shard that was causing tonnes of disk
> > writes. It makes me think, even if you make sure the moveChunk only
> marks a
> > chunk as deleted in Primary, but doesn't physically touch the disk, just
> > unloads the pages from RAM, that might work too.
>
> Interesting. We do have to actually delete the data so the space can
> be re-used, but might be able to do it more efficiently.
> The code should be yielding before actually doing anything, so kind of
> surprising it caused too much load on the primary.
>

Disk going to 100% was the issue.


> How active is your disk at steady state?
>

< 5%; Usually 1% or less.


>
> There is one quick change that would be interesting to try.
> When we first added sharding a couple years ago, we put it a pedantic
> safety measure of writing out all the data we delete after a chunk is
> moved just to be sure.
> I wonder how much load that is adding.
> Would be trivial to remove.
>

Awesome. Nice to hear that.


>
> Do you have a lab where you can reproduce this easily?
>

I don't really have a lab. Only have the production environment as of now.
Each moveChunk gives about 4-12 minutes of downtime. If you want we can try
it sometime later in the night.

I suspect a vanilla setup, should be able to reproduce this though, since
it's pretty commonly happening across companies.


>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Jeff Lee
2012-05-11 16:51:05 UTC
Permalink
Hey Eliot,

We've been having a problem with this for a while. I can force load a
chunk into memory so that the actual move and metadata change takes less
than 20s but the removal can take 15 to 20 min during which the lock pct on
the primary goes to > 90%.

The lock yields for other operations but our average application latency
doubles during this period.

Could this be caused by use of a shard key other than _id?

The change you mentioned sounds promising...can you point me to the code?

Thanks.



On Fri, May 11, 2012 at 8:24 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:

> > From my basic drill-down, it was the last part of moveChunk, which was
> > deleting the date from the master-shard that was causing tonnes of disk
> > writes. It makes me think, even if you make sure the moveChunk only
> marks a
> > chunk as deleted in Primary, but doesn't physically touch the disk, just
> > unloads the pages from RAM, that might work too.
>
> Interesting. We do have to actually delete the data so the space can
> be re-used, but might be able to do it more efficiently.
> The code should be yielding before actually doing anything, so kind of
> surprising it caused too much load on the primary.
> How active is your disk at steady state?

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Eliot Horowitz
2012-05-11 16:57:05 UTC
Permalink
How many indexes do you both have?

Jeff, what version?

For the change, there is actually a command line arg: --noMoveParanoia

Again, sample docs or a reproducible case would be great, as there are
a lot factors.


On Fri, May 11, 2012 at 12:51 PM, Jeff Lee <jeff.lee-***@public.gmane.org> wrote:
> Hey Eliot,
>
> We've been having a problem with this for a while.  I can force load a chunk
> into memory so that the actual move and metadata change takes less than 20s
> but the removal can take 15 to 20 min during which the lock pct on the
> primary goes to > 90%.
>
> The lock yields for other operations but our average application latency
> doubles during this period.
>
> Could this be caused by use of a shard key other than _id?
>
> The change you mentioned sounds promising...can you point me to the code?
>
> Thanks.
>
>
>
> On Fri, May 11, 2012 at 8:24 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:
>>
>> > From my basic drill-down, it was the last part of moveChunk, which was
>> > deleting the date from the master-shard that was causing tonnes of disk
>> > writes. It makes me think, even if you make sure the moveChunk only
>> > marks a
>> > chunk as deleted in Primary, but doesn't physically touch the disk, just
>> > unloads the pages from RAM, that might work too.
>>
>> Interesting.  We do have to actually delete the data so the space can
>> be re-used, but might be able to do it more efficiently.
>> The code should be yielding before actually doing anything, so kind of
>> surprising it caused too much load on the primary.
>> How active is your disk at steady state?
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+***@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Jeff Lee
2012-05-11 17:09:29 UTC
Permalink
Awesome...I'll run some tests with that and gather some info for a ticket
if that doesn't doesn't do the trick.

Thanks!


On Fri, May 11, 2012 at 9:57 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:

> How many indexes do you both have?
>
> Jeff, what version?
>
> For the change, there is actually a command line arg: --noMoveParanoia
>
> Again, sample docs or a reproducible case would be great, as there are
> a lot factors.
>
>
> On Fri, May 11, 2012 at 12:51 PM, Jeff Lee <jeff.lee-***@public.gmane.org> wrote:
> > Hey Eliot,
> >
> > We've been having a problem with this for a while. I can force load a
> chunk
> > into memory so that the actual move and metadata change takes less than
> 20s
> > but the removal can take 15 to 20 min during which the lock pct on the
> > primary goes to > 90%.
> >
> > The lock yields for other operations but our average application latency
> > doubles during this period.
> >
> > Could this be caused by use of a shard key other than _id?
> >
> > The change you mentioned sounds promising...can you point me to the code?
> >
> > Thanks.
> >
> >
> >
> > On Fri, May 11, 2012 at 8:24 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:
> >>
> >> > From my basic drill-down, it was the last part of moveChunk, which was
> >> > deleting the date from the master-shard that was causing tonnes of
> disk
> >> > writes. It makes me think, even if you make sure the moveChunk only
> >> > marks a
> >> > chunk as deleted in Primary, but doesn't physically touch the disk,
> just
> >> > unloads the pages from RAM, that might work too.
> >>
> >> Interesting. We do have to actually delete the data so the space can
> >> be re-used, but might be able to do it more efficiently.
> >> The code should be yielding before actually doing anything, so kind of
> >> surprising it caused too much load on the primary.
> >> How active is your disk at steady state?
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "mongodb-user" group.
> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > To unsubscribe from this group, send email to
> > mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > For more options, visit this group at
> > http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Kenny Gorman
2012-05-11 17:30:20 UTC
Permalink
Just speaking out loud here, but what about a 'deferred delete' option?

aka: not to delete the old data post move, but rather just leave it in place and somehow mark it as 'old'. Then properly timed and throttled jobs could clean up later? I think this came up before in some discussions.

You are relying on the config image to ensure you don't get logical corruption however with this approach.

I have used that approach for large manually sharded PostgreSQL clusters with success.

-kg

On May 11, 2012, at 9:57 AM, Eliot Horowitz wrote:

> How many indexes do you both have?
>
> Jeff, what version?
>
> For the change, there is actually a command line arg: --noMoveParanoia
>
> Again, sample docs or a reproducible case would be great, as there are
> a lot factors.
>
>
> On Fri, May 11, 2012 at 12:51 PM, Jeff Lee <jeff.lee-***@public.gmane.org> wrote:
>> Hey Eliot,
>>
>> We've been having a problem with this for a while. I can force load a chunk
>> into memory so that the actual move and metadata change takes less than 20s
>> but the removal can take 15 to 20 min during which the lock pct on the
>> primary goes to > 90%.
>>
>> The lock yields for other operations but our average application latency
>> doubles during this period.
>>
>> Could this be caused by use of a shard key other than _id?
>>
>> The change you mentioned sounds promising...can you point me to the code?
>>
>> Thanks.
>>
>>
>>
>> On Fri, May 11, 2012 at 8:24 AM, Eliot Horowitz <eliot-***@public.gmane.org> wrote:
>>>
>>>> From my basic drill-down, it was the last part of moveChunk, which was
>>>> deleting the date from the master-shard that was causing tonnes of disk
>>>> writes. It makes me think, even if you make sure the moveChunk only
>>>> marks a
>>>> chunk as deleted in Primary, but doesn't physically touch the disk, just
>>>> unloads the pages from RAM, that might work too.
>>>
>>> Interesting. We do have to actually delete the data so the space can
>>> be re-used, but might be able to do it more efficiently.
>>> The code should be yielding before actually doing anything, so kind of
>>> surprising it caused too much load on the primary.
>>> How active is your disk at steady state?
>>
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "mongodb-user" group.
>> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> To unsubscribe from this group, send email to
>> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
>> For more options, visit this group at
>> http://groups.google.com/group/mongodb-user?hl=en.
>
> --
> You received this message because you are subscribed to the Google Groups "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
>
>
Adrien Mogenet
2012-05-11 20:16:02 UTC
Permalink
On Friday, May 11, 2012 7:30:20 PM UTC+2, k...-***@public.gmane.org wrote:
>
> Just speaking out loud here, but what about a 'deferred delete' option?
>

This is the choice of HBase, and probably many other products. I have to
admit that I think it's a valuable trade-off : we did not meet a lot of
situations where getting free space immediately does matter. Btw, I have to
run repairDatabase() as crontab to get free space back, so I think that
"deferred delete" can be helpful, used with a great defragmentation policy,
but I'm not sure this is the topic here...


>
> aka: not to delete the old data post move, but rather just leave it in
> place and somehow mark it as 'old'. Then properly timed and throttled jobs
> could clean up later? I think this came up before in some discussions.
>
> You are relying on the config image to ensure you don't get logical
> corruption however with this approach.
>
> I have used that approach for large manually sharded PostgreSQL clusters
> with success.
>
> -kg
>
> On May 11, 2012, at 9:57 AM, Eliot Horowitz wrote:
>
> > How many indexes do you both have?
> >
> > Jeff, what version?
> >
> > For the change, there is actually a command line arg: --noMoveParanoia
> >
> > Again, sample docs or a reproducible case would be great, as there are
> > a lot factors.
> >
> >
> > On Fri, May 11, 2012 at 12:51 PM, Jeff Lee <jeff.lee-***@public.gmane.org> wrote:
> >> Hey Eliot,
> >>
> >> We've been having a problem with this for a while. I can force load a
> chunk
> >> into memory so that the actual move and metadata change takes less than
> 20s
> >> but the removal can take 15 to 20 min during which the lock pct on the
> >> primary goes to > 90%.
> >>
> >> The lock yields for other operations but our average application
> latency
> >> doubles during this period.
> >>
> >> Could this be caused by use of a shard key other than _id?
> >>
> >> The change you mentioned sounds promising...can you point me to the
> code?
> >>
> >> Thanks.
> >>
> >>
> >>
> >> On Fri, May 11, 2012 at 8:24 AM, Eliot Horowitz <eliot-***@public.gmane.org>
> wrote:
> >>>
> >>>> From my basic drill-down, it was the last part of moveChunk, which
> was
> >>>> deleting the date from the master-shard that was causing tonnes of
> disk
> >>>> writes. It makes me think, even if you make sure the moveChunk only
> >>>> marks a
> >>>> chunk as deleted in Primary, but doesn't physically touch the disk,
> just
> >>>> unloads the pages from RAM, that might work too.
> >>>
> >>> Interesting. We do have to actually delete the data so the space can
> >>> be re-used, but might be able to do it more efficiently.
> >>> The code should be yielding before actually doing anything, so kind of
> >>> surprising it caused too much load on the primary.
> >>> How active is your disk at steady state?
> >>
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> Groups
> >> "mongodb-user" group.
> >> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> To unsubscribe from this group, send email to
> >> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> >> For more options, visit this group at
> >> http://groups.google.com/group/mongodb-user?hl=en.
> >
> > --
> > You received this message because you are subscribed to the Google
> Groups "mongodb-user" group.
> > To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> > For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
> >
> >
>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To view this discussion on the web visit https://groups.google.com/d/msg/mongodb-user/-/NNrdjFDA3XQJ.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Glenn Maynard
2012-05-11 22:50:17 UTC
Permalink
On Fri, May 11, 2012 at 10:11 AM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:

> Let's take the last one alone -
>
> a) moveChunk executes it's iteration, iff all reads are completed.
>

I think what you mean is for moveChunk to periodically yield long enough
for everything previously in the read queue to finish.

That would probably just result in your four minute chunk migrations taking
hours or days, though, which isn't really a working solution. Giving
priority to reads doesn't help if the reads end up starving out chunk
migrations.


On Fri, May 11, 2012 at 12:30 PM, Kenny Gorman <kg-***@public.gmane.org> wrote:

> Just speaking out loud here, but what about a 'deferred delete' option?
>

This won't help much if you have a constant load. You have to spend the
I/O time sooner or later.

People keep talking about workarounds--yields and delays just shift the
time around. Even if Mongo did any of those, you'd still have a problem;
it'd just have taken a bit longer to surface. The focus should probably be
on figuring out why the deletes are taking so much I/O to begin with.

--
Glenn Maynard

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-12 02:18:49 UTC
Permalink
Hey Glenn,

inline..

On Fri, May 11, 2012 at 3:50 PM, Glenn Maynard <glenn-***@public.gmane.org> wrote:

> On Fri, May 11, 2012 at 10:11 AM, Prasanna .S <myprasanna-***@public.gmane.org>wrote:
>
>> Let's take the last one alone -
>>
>> a) moveChunk executes it's iteration, iff all reads are completed.
>>
>
> I think what you mean is for moveChunk to periodically yield long enough
> for everything previously in the read queue to finish.
>
> That would probably just result in your four minute chunk migrations
> taking hours or days, though, which isn't really a working solution.
> Giving priority to reads doesn't help if the reads end up starving out
> chunk migrations.
>

At 100% capacity, you need to choose. Yes, it'll starve out chunk
migrations. The other option is to starve out reads. When one is about
keeping the site up, and the other is about a background operation, I think
the priorities are pretty clear.

Note that there can be alternate ways of providing a guarantee:

[1] You can say reads will be at 90% capacity, and 10% capacity (CPU AND
Disk) goes to chunk movements.
[2] You can say reads will always execute before chunk migrations complete.
Yes this will starve chunkMigrations, which is probably fine. Something has
to take a hit right? Note that, the other option is, read queue keeps
getting incrementally larger, which on saturation - you'll hit timeouts -
regardless of weather your timeout numbers are 100 ms or 10s.


>
>
> On Fri, May 11, 2012 at 12:30 PM, Kenny Gorman <kg-***@public.gmane.org> wrote:
>
>> Just speaking out loud here, but what about a 'deferred delete' option?
>>
>
> This won't help much if you have a constant load. You have to spend the
> I/O time sooner or later.
>

Yes, this is true. One alternate solution is to not do the IO at all from
the primary. Just mark it as ignored and not run the compaction. It'll take
extra disk space, but all databases have de-fragmentation problems. Esp,
compare this with the magnitude of the other problem - downtime.


>
> People keep talking about workarounds--yields and delays just shift the
> time around. Even if Mongo did any of those, you'd still have a problem;
> it'd just have taken a bit longer to surface. The focus should probably be
> on figuring out why the deletes are taking so much I/O to begin with.


Unfortunately, that'll make things better but not solve the problem. Let's
say you make the deletes 10x faster, you are talking about 24s of downtime
instead of 4 minutes. Throttling the background process reliably is the
only way out.


>
>
> --
> Glenn Maynard
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "mongodb-user" group.
> To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> To unsubscribe from this group, send email to
> mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
> For more options, visit this group at
> http://groups.google.com/group/mongodb-user?hl=en.
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna .S
2012-05-08 06:34:09 UTC
Permalink
More info:

- Our chunk size is 64MB.
- It takes ~4 minutes, for a single moveChunk to complete. Is that weird or
expected?
- Within the 4 minutes, the site goes down, because the reads timeout.
- We are running a EBS disk.

I can try to make the chunks smaller if that's the issue?

On Mon, May 7, 2012 at 11:22 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:

> Hey guys,
>
> We sharded mongo as per the docs. The issue is the balancer is not being
> very clever and not throttling at all. Now we are stuck with a half-sharded
> setup and we can't turn on the balancer because it'd bring the site down.
> All read queries timeout when the disk write rate is 100%.
>
> Is there a way out of this mess at all? From a high-level, mongo could
> have handled this by any of the following means:
>
> 1. Replicate from the slaves. Mysql does this.
> 2. Throttle and replicate only as long as the disk util is < 50% or so.
> 3. Have the moveChunk interruptible. If reads and writes come in-between,
> consider them as higher priority, so that a read/write is never waiting for
> more than a second.
>
> Have you guys heard of this problem before? Because this really make
> sharding impossible. I'm not sure what others are doing in production to
> work around this.
>
> Thanks,
> -Prasanna
> Co-Founder & CTO, lal.com
>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Nat
2012-05-08 07:28:37 UTC
Permalink
It shouldn't take that long. Can you show your iostat, mongostat when the
problem is happening? Also what EBS setup do you use? Did you follow the
production notes at
http://www.mongodb.org/display/DOCS/Amazon+EC2#AmazonEC2-StorageConfiguration
?



On Tuesday, 8 May 2012 14:34:09 UTC+8, Prasanna wrote:

> More info:
>
> - Our chunk size is 64MB.
> - It takes ~4 minutes, for a single moveChunk to complete. Is that weird
> or expected?
> - Within the 4 minutes, the site goes down, because the reads timeout.
> - We are running a EBS disk.
>
> I can try to make the chunks smaller if that's the issue?
>
> On Mon, May 7, 2012 at 11:22 PM, Prasanna .S <myprasanna-***@public.gmane.org> wrote:
>
>> Hey guys,
>>
>> We sharded mongo as per the docs. The issue is the balancer is not being
>> very clever and not throttling at all. Now we are stuck with a half-sharded
>> setup and we can't turn on the balancer because it'd bring the site down.
>> All read queries timeout when the disk write rate is 100%.
>>
>> Is there a way out of this mess at all? From a high-level, mongo could
>> have handled this by any of the following means:
>>
>> 1. Replicate from the slaves. Mysql does this.
>> 2. Throttle and replicate only as long as the disk util is < 50% or so.
>> 3. Have the moveChunk interruptible. If reads and writes come in-between,
>> consider them as higher priority, so that a read/write is never waiting for
>> more than a second.
>>
>> Have you guys heard of this problem before? Because this really make
>> sharding impossible. I'm not sure what others are doing in production to
>> work around this.
>>
>> Thanks,
>> -Prasanna
>> Co-Founder & CTO, lal.com
>>
>>
>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To view this discussion on the web visit https://groups.google.com/d/msg/mongodb-user/-/rj8o5zGmDeoJ.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Prasanna
2012-05-08 08:42:10 UTC
Permalink
Thanks a bunch for the response Nat.
Yes we follow the EC2 Storage configuration guide.

Here's the iostat: http://pastebin.com/nB7tbKtb
The mongostat: http://pastebin.com/qz637ViD
And the mongotop: http://pastebin.com/dTnRTMWD

Any suggestions would be much appreciated!

On Tuesday, May 8, 2012 12:28:37 AM UTC-7, Nat wrote:
>
> It shouldn't take that long. Can you show your iostat, mongostat when the
> problem is happening? Also what EBS setup do you use? Did you follow the
> production notes at
> http://www.mongodb.org/display/DOCS/Amazon+EC2#AmazonEC2-StorageConfiguration
> ?
>
>
>
> On Tuesday, 8 May 2012 14:34:09 UTC+8, Prasanna wrote:
>
>> More info:
>>
>> - Our chunk size is 64MB.
>> - It takes ~4 minutes, for a single moveChunk to complete. Is that weird
>> or expected?
>> - Within the 4 minutes, the site goes down, because the reads timeout.
>> - We are running a EBS disk.
>>
>> I can try to make the chunks smaller if that's the issue?
>>
>> On Mon, May 7, 2012 at 11:22 PM, Prasanna .S <myprasanna-***@public.gmane.org>wrote:
>>
>>> Hey guys,
>>>
>>> We sharded mongo as per the docs. The issue is the balancer is not being
>>> very clever and not throttling at all. Now we are stuck with a half-sharded
>>> setup and we can't turn on the balancer because it'd bring the site down.
>>> All read queries timeout when the disk write rate is 100%.
>>>
>>> Is there a way out of this mess at all? From a high-level, mongo could
>>> have handled this by any of the following means:
>>>
>>> 1. Replicate from the slaves. Mysql does this.
>>> 2. Throttle and replicate only as long as the disk util is < 50% or so.
>>> 3. Have the moveChunk interruptible. If reads and writes come
>>> in-between, consider them as higher priority, so that a read/write is never
>>> waiting for more than a second.
>>>
>>> Have you guys heard of this problem before? Because this really make
>>> sharding impossible. I'm not sure what others are doing in production to
>>> work around this.
>>>
>>> Thanks,
>>> -Prasanna
>>> Co-Founder & CTO, lal.com
>>>
>>>
>>

--
You received this message because you are subscribed to the Google Groups "mongodb-user" group.
To view this discussion on the web visit https://groups.google.com/d/msg/mongodb-user/-/-u9_bIdWaSwJ.
To post to this group, send email to mongodb-user-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
To unsubscribe from this group, send email to mongodb-user+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/***@public.gmane.org
For more options, visit this group at http://groups.google.com/group/mongodb-user?hl=en.
Loading...