| Message ID | 075032be4fd288074b8f699d4f4ba0179518df6f.1379344038.git.viresh.kumar@linaro.org |
|---|---|
| State | New |
| Headers | show |
On Mon, 2013-09-16 at 20:40 +0530, Viresh Kumar wrote: > Current code looks like this: > > WARN_ON(lock_policy_rwsem_write(cpu)); > update_policy_cpu(policy, new_cpu); > unlock_policy_rwsem_write(cpu); > > {lock|unlock}_policy_rwsem_write(cpu) takes/releases policy->cpu's rwsem. > Because cpu is changing with the call to update_policy_cpu(), the > unlock_policy_rwsem_write() will release the incorrect lock. > > The right solution would be to take rwsem lock/unlock for both old and new cpu. I thought possibly something like that, then wondered if threads could take the locks in different orders at the same time, leading to deadlock? So as I wasn't familiar with cpufreq I thought I'd leave it to the experts to worry about :-) This patch contains a bug, see inline comment below. Even with that fixed it still doesn't work for me. I get a lockdep warning (below). I have verified the cpu and locks are different locks, and it's not a harmless false positive because I later get the message 'cpufreq: __cpufreq_remove_dev_prepare: Failed to stop governor'. ============================================= [ INFO: possible recursive locking detected ] 3.11.0+ #4 Not tainted --------------------------------------------- swapper/0/1 is trying to acquire lock: (&per_cpu(cpu_policy_rwsem, cpu)){+.+...}, at: [<c0293c03>] update_policy_cpu+0x53/0xa4 but task is already holding lock: (&per_cpu(cpu_policy_rwsem, cpu)){+.+...}, at: [<c0293bef>] update_policy_cpu+0x3f/0xa4 other info that might help us debug this: Possible unsafe locking scenario: CPU0 ---- lock(&per_cpu(cpu_policy_rwsem, cpu)); lock(&per_cpu(cpu_policy_rwsem, cpu)); *** DEADLOCK *** May be due to missing lock nesting notation 4 locks held by swapper/0/1: #0: (bL_switcher_activation_lock){+.+.+.}, at: [<c0019b03>] bL_switcher_enable+0x17/0x2d8 #1: ((bL_activation_notifier).rwsem){.+.+.+}, at: [<c00370bd>] __blocking_notifier_call_chain+0x1d/0x40 #2: (subsys mutex#6){+.+.+.}, at: [<c023279d>] subsys_interface_unregister+0x1d/0x68 #3: (&per_cpu(cpu_policy_rwsem, cpu)){+.+...}, at: [<c0293bef>] update_policy_cpu+0x3f/0xa4 > This patch fixes this bug by taking both locks directly instead of calling > lock_policy_rwsem_write(). > > Reported-by: Jon Medhurst<tixy@linaro.org> > Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org> > --- > Hi Rafael, > > Probably we can get this patch in for 3.12? The second one can go in 3.13. > > These are compile tested only at my end. Tixy reported these and probably can > give his tested-by once he is done testing them. > > drivers/cpufreq/cpufreq.c | 9 +++++++-- > 1 file changed, 7 insertions(+), 2 deletions(-) > If I take mainline code and just change the line above to: up_write(&per_cpu(cpu_policy_rwsem, (per_cpu(cpufreq_cpu_data, cpu))->last_cpu)); then the big_little cpufreq driver works for me. > diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c > index 43c24aa..c18bf7b 100644 > --- a/drivers/cpufreq/cpufreq.c > +++ b/drivers/cpufreq/cpufreq.c > @@ -952,9 +952,16 @@ static void update_policy_cpu(struct cpufreq_policy *policy, unsigned int cpu) > if (cpu == policy->cpu) > return; > > + /* take direct locks as lock_policy_rwsem_write wouldn't work here */ > + down_write(&per_cpu(cpu_policy_rwsem, policy->cpu)); > + down_write(&per_cpu(cpu_policy_rwsem, cpu)); > + > policy->last_cpu = policy->cpu; > policy->cpu = cpu; > > + up_write(&per_cpu(cpu_policy_rwsem, cpu)); > + up_write(&per_cpu(cpu_policy_rwsem, policy->cpu)); You've just overwritten policy->cpu with cpu. I tried using policy->last_cpu to fix that, but it still doesn't work for me (giving the lockdep warning I mentioned.) If I change the code to just lock the original policy->cpu lock only, then all is fine. Also, this locking is now just happening around policy->cpu update, whereas before this change, it was locked for the whole of update_policy_cpu, i.e. cpufreq_frequency_table_update_policy_cpu and the notifier callbacks. Is that change of lock coverage OK? > + > #ifdef CONFIG_CPU_FREQ_TABLE > cpufreq_frequency_table_update_policy_cpu(policy); > #endif > @@ -1203,9 +1210,7 @@ static int __cpufreq_remove_dev_prepare(struct device *dev, > > new_cpu = cpufreq_nominate_new_policy_cpu(policy, cpu, frozen); > if (new_cpu >= 0) { > - WARN_ON(lock_policy_rwsem_write(cpu)); > update_policy_cpu(policy, new_cpu); > - unlock_policy_rwsem_write(cpu); > > if (!frozen) { > pr_debug("%s: policy Kobject moved to cpu: %d "
On 16 September 2013 21:57, Jon Medhurst (Tixy) <tixy@linaro.org> wrote: > If I take mainline code and just change the line above to: You meant this line by above line? unlock_policy_rwsem_write(cpu); > up_write(&per_cpu(cpu_policy_rwsem, (per_cpu(cpufreq_cpu_data, > cpu))->last_cpu)); > then the big_little cpufreq driver works for me. That would be same as: up_write(&per_cpu(cpu_policy_rwsem, policy->last_cpu)); >> diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c >> index 43c24aa..c18bf7b 100644 >> --- a/drivers/cpufreq/cpufreq.c >> +++ b/drivers/cpufreq/cpufreq.c >> @@ -952,9 +952,16 @@ static void update_policy_cpu(struct cpufreq_policy *policy, unsigned int cpu) >> if (cpu == policy->cpu) >> return; >> >> + /* take direct locks as lock_policy_rwsem_write wouldn't work here */ >> + down_write(&per_cpu(cpu_policy_rwsem, policy->cpu)); >> + down_write(&per_cpu(cpu_policy_rwsem, cpu)); >> + >> policy->last_cpu = policy->cpu; >> policy->cpu = cpu; >> >> + up_write(&per_cpu(cpu_policy_rwsem, cpu)); >> + up_write(&per_cpu(cpu_policy_rwsem, policy->cpu)); > > You've just overwritten policy->cpu with cpu. Stupid enough :) > I tried using > policy->last_cpu to fix that, but it still doesn't work for me (giving > the lockdep warning I mentioned.) If I change the code to just lock the > original policy->cpu lock only, then all is fine. Fixed my patch now.. find attached.. It mentions why lock for last cpu is enough here. Copied that here too.. + /* + * Take direct locks as lock_policy_rwsem_write wouldn't work here. + * Also lock for last cpu is enough here as contention will happen only + * after policy->cpu is changed and after it is changed, other threads + * will try to acquire lock for new cpu. And policy is already updated + * by then. + */ > Also, this locking is now just happening around policy->cpu update, > whereas before this change, it was locked for the whole of > update_policy_cpu, i.e. cpufreq_frequency_table_update_policy_cpu and > the notifier callbacks. Is that change of lock coverage OK? Yeah, the rwsem is only required for updating policy's fields and so that should be okay.
On Monday, September 16, 2013 10:38:05 PM Viresh Kumar wrote: > On 16 September 2013 21:57, Jon Medhurst (Tixy) <tixy@linaro.org> wrote: > > If I take mainline code and just change the line above to: > > You meant this line by above line? > > unlock_policy_rwsem_write(cpu); > > > up_write(&per_cpu(cpu_policy_rwsem, (per_cpu(cpufreq_cpu_data, > > cpu))->last_cpu)); > > then the big_little cpufreq driver works for me. > > That would be same as: up_write(&per_cpu(cpu_policy_rwsem, policy->last_cpu)); > > >> diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c > >> index 43c24aa..c18bf7b 100644 > >> --- a/drivers/cpufreq/cpufreq.c > >> +++ b/drivers/cpufreq/cpufreq.c > >> @@ -952,9 +952,16 @@ static void update_policy_cpu(struct cpufreq_policy *policy, unsigned int cpu) > >> if (cpu == policy->cpu) > >> return; > >> > >> + /* take direct locks as lock_policy_rwsem_write wouldn't work here */ > >> + down_write(&per_cpu(cpu_policy_rwsem, policy->cpu)); > >> + down_write(&per_cpu(cpu_policy_rwsem, cpu)); > >> + > >> policy->last_cpu = policy->cpu; > >> policy->cpu = cpu; > >> > >> + up_write(&per_cpu(cpu_policy_rwsem, cpu)); > >> + up_write(&per_cpu(cpu_policy_rwsem, policy->cpu)); > > > > You've just overwritten policy->cpu with cpu. > > Stupid enough :) > > > I tried using > > policy->last_cpu to fix that, but it still doesn't work for me (giving > > the lockdep warning I mentioned.) If I change the code to just lock the > > original policy->cpu lock only, then all is fine. > > Fixed my patch now.. find attached.. Care to resend with a subject indicating that that's a patch update? Like [PATCH v2] etc. or similar? Rafael
On Mon, 2013-09-16 at 22:38 +0530, Viresh Kumar wrote: > On 16 September 2013 21:57, Jon Medhurst (Tixy) <tixy@linaro.org> wrote: > > If I take mainline code and just change the line above to: > > You meant this line by above line? > > unlock_policy_rwsem_write(cpu); Yes. > > up_write(&per_cpu(cpu_policy_rwsem, (per_cpu(cpufreq_cpu_data, > > cpu))->last_cpu)); > > then the big_little cpufreq driver works for me. > > That would be same as: up_write(&per_cpu(cpu_policy_rwsem, policy->last_cpu)); Yes. (I had cut'n'pasted from the unlock_policy_rwsem_ macro) > >> diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c > >> index 43c24aa..c18bf7b 100644 > >> --- a/drivers/cpufreq/cpufreq.c > >> +++ b/drivers/cpufreq/cpufreq.c > >> @@ -952,9 +952,16 @@ static void update_policy_cpu(struct cpufreq_policy *policy, unsigned int cpu) > >> if (cpu == policy->cpu) > >> return; > >> > >> + /* take direct locks as lock_policy_rwsem_write wouldn't work here */ > >> + down_write(&per_cpu(cpu_policy_rwsem, policy->cpu)); > >> + down_write(&per_cpu(cpu_policy_rwsem, cpu)); > >> + > >> policy->last_cpu = policy->cpu; > >> policy->cpu = cpu; > >> > >> + up_write(&per_cpu(cpu_policy_rwsem, cpu)); > >> + up_write(&per_cpu(cpu_policy_rwsem, policy->cpu)); > > > > You've just overwritten policy->cpu with cpu. > > Stupid enough :) > > > I tried using > > policy->last_cpu to fix that, but it still doesn't work for me (giving > > the lockdep warning I mentioned.) If I change the code to just lock the > > original policy->cpu lock only, then all is fine. > > Fixed my patch now.. find attached.. The commit log to that patch still mentions taking both locks. The code itself fixes the lockdep errors I had, so Tested-by: Jon Medhurst <tixy@linaro.org> however, I still have concerns about the locking (below)... > It mentions why lock for last cpu is > enough here. Copied that here too.. > > + /* > + * Take direct locks as lock_policy_rwsem_write wouldn't work here. > + * Also lock for last cpu is enough here as contention will happen only > + * after policy->cpu is changed and after it is changed, other threads > + * will try to acquire lock for new cpu. And policy is already updated > + * by then. > + */ > > > Also, this locking is now just happening around policy->cpu update, > > whereas before this change, it was locked for the whole of > > update_policy_cpu, i.e. cpufreq_frequency_table_update_policy_cpu and > > the notifier callbacks. Is that change of lock coverage OK? > > Yeah, the rwsem is only required for updating policy's fields and so that > should be okay. But what about reads, like in cpufreq_frequency_table_update_policy_cpu which is called immediately after the unlocking writes? Should that not be covered by a reader lock? And after that call, policy is passed into blocking_notifier_call_chain, do those callbacks not want to look at policy fields? Or are they going to do there own locking? Or is update_policy_cpu itself meant to be called with a read lock held? (It doesn't appear to be as the semaphore 'activiy' field of the lock is zero). Or is there some other non-obvious synchronisation method which means the policy object can't change? This is the first time I've looked at this code, so feel free just to say 'it's complicated, just trust me, I'm the expert, I know what I'm doing'...
On 17 September 2013 00:04, Rafael J. Wysocki <rjw@sisk.pl> wrote: > Care to resend with a subject indicating that that's a patch update? > > Like [PATCH v2] etc. or similar? Yeah.. I was waiting for Tixy to test it once..
On 17 September 2013 00:12, Jon Medhurst (Tixy) <tixy@linaro.org> wrote: > The commit log to that patch still mentions taking both locks. Yeah, it was sent in hurry to just give you a working solution and I forgot to see the log if it is still valid. > The code itself fixes the lockdep errors I had, so > > Tested-by: Jon Medhurst <tixy@linaro.org> Great!! > however, I still have concerns about the locking (below)... :( > But what about reads, like in cpufreq_frequency_table_update_policy_cpu > which is called immediately after the unlocking writes? Should that not > be covered by a reader lock? > > And after that call, policy is passed into blocking_notifier_call_chain, > do those callbacks not want to look at policy fields? Or are they going > to do there own locking? policy->cpu is used at multiple places outside of cpufreq.c and cpufreq core can't really put read locks for those accesses. Things will turn bad only if cpufreq core has got these races where we are trying to access a struct or pointer that belonged to the last policy->cpu, which is updated now.. For example the case of lock you reported.. And so lock is required only for those places.. > Or is update_policy_cpu itself meant to be called with a read lock held? No. > This is the first time I've looked at this code, so feel free just to > say 'it's complicated, just trust me, I'm the expert, I know what I'm > doing'... We aren't that rude :) Now, that you have tested this patch let me resent it...
diff --git a/drivers/cpufreq/cpufreq.c b/drivers/cpufreq/cpufreq.c index 43c24aa..c18bf7b 100644 --- a/drivers/cpufreq/cpufreq.c +++ b/drivers/cpufreq/cpufreq.c @@ -952,9 +952,16 @@ static void update_policy_cpu(struct cpufreq_policy *policy, unsigned int cpu) if (cpu == policy->cpu) return; + /* take direct locks as lock_policy_rwsem_write wouldn't work here */ + down_write(&per_cpu(cpu_policy_rwsem, policy->cpu)); + down_write(&per_cpu(cpu_policy_rwsem, cpu)); + policy->last_cpu = policy->cpu; policy->cpu = cpu; + up_write(&per_cpu(cpu_policy_rwsem, cpu)); + up_write(&per_cpu(cpu_policy_rwsem, policy->cpu)); + #ifdef CONFIG_CPU_FREQ_TABLE cpufreq_frequency_table_update_policy_cpu(policy); #endif @@ -1203,9 +1210,7 @@ static int __cpufreq_remove_dev_prepare(struct device *dev, new_cpu = cpufreq_nominate_new_policy_cpu(policy, cpu, frozen); if (new_cpu >= 0) { - WARN_ON(lock_policy_rwsem_write(cpu)); update_policy_cpu(policy, new_cpu); - unlock_policy_rwsem_write(cpu); if (!frozen) { pr_debug("%s: policy Kobject moved to cpu: %d "
Current code looks like this: WARN_ON(lock_policy_rwsem_write(cpu)); update_policy_cpu(policy, new_cpu); unlock_policy_rwsem_write(cpu); {lock|unlock}_policy_rwsem_write(cpu) takes/releases policy->cpu's rwsem. Because cpu is changing with the call to update_policy_cpu(), the unlock_policy_rwsem_write() will release the incorrect lock. The right solution would be to take rwsem lock/unlock for both old and new cpu. This patch fixes this bug by taking both locks directly instead of calling lock_policy_rwsem_write(). Reported-by: Jon Medhurst<tixy@linaro.org> Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org> --- Hi Rafael, Probably we can get this patch in for 3.12? The second one can go in 3.13. These are compile tested only at my end. Tixy reported these and probably can give his tested-by once he is done testing them. drivers/cpufreq/cpufreq.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-)